Data Science Essentials
Data Cleaning
Techniques for preprocessing raw, messy data by handling missing values, duplicates, and type mismatches.
Interview: Core data engineering. Crucial for showing how to clean real-world data safely and handle missing values.
Real-world data is messy. It contains missing values, duplicates, formatting errors, and outliers. Data Cleaning (data cleaning/preprocessing) prepares this raw dataset for analysis or machine learning modeling.
Handling Missing Values (NaN)
When a dataset has missing fields, you have two primary options:
- Dropping (Deletion): Removing rows or columns that contain null values using
df.dropna(). Safe if missing values are rare. - Imputation (Fill): Filling null fields with estimates using
df.fillna()(e.g. replacing missing ages with the dataset mean or median, or missing text with placeholders).
Handling Duplicates
Duplicate entries skew statistical models. Identical rows are identified and dropped using df.drop_duplicates().
Use Cases
ML Preprocessing — Formatting raw user feedback datasets into clean arrays for model training.
Database Import — Sanitizing file data (CSV, JSON) before inserting it into database schemas to avoid constraint failures.
Analytics Reporting — Ensuring data metrics like averages are not skewed by null inputs.
Common Mistakes
Imputing with wrong stats — Using mean value instead of median on datasets containing extreme outliers (which heavily skews the mean).
Silently dropping too many rows — Running `dropna()` without parameters, which can delete 90% of a dataset if every row has at least one rare missing field.
Type casting errors — Attempting to clean datasets without casting column data types (e.g., trying to run arithmetic on age values stored as strings).