Data Science Essentials
Statistics with Python
Performing core statistical calculations using built-in libraries and NumPy.
Interview: Analytical roles. Tested on computing metrics, probability distributions, outlier boundaries, and correlation analysis.
Statistics is critical for analyzing datasets, verifying machine learning assumptions, and interpreting data distributions. Python provides a built-in statistics module for basic metrics, and packages like NumPy and SciPy for heavy mathematical operations.
Core Metrics
- Mean: The arithmetic average of a dataset.
- Median: The middle value of a sorted dataset (resilient to outliers).
- Standard Deviation (Std Dev): A measure of data dispersion/spread around the mean.
- Correlation: A measure of linear relationship strength and direction between two variables (ranging from -1 to +1).
Use Cases
Data Profiling — Finding general metrics (mean, variance) of features before applying ML algorithms.
A/B Testing — Running hypothesis tests to see if changes in a website layout yield statistically significant user clicks.
Outlier Detection — Using standard deviation boundaries (e.g. data points exceeding mean +/- 3 * std_dev) to identify anomalies.
Common Mistakes
Assuming mean represents standard cases — Relying on mean when the dataset has a heavily skewed distribution (median is much more representative in skewed data).
Confusing correlation with causation — Assuming that because two metrics correlate (have high correlation index), one causes the other.
Ignoring standard deviation — Comparing means of two datasets without looking at their spreads (standard deviations), which can lead to incorrect conclusions.