Data Science Essentials
Pandas Basics
Data analysis and manipulation with Series and DataFrames.
Interview: Industry standard for data engineering and preprocessing. Crucial for understanding dataset slicing, aggregation, and merging.
Pandas is Python's primary library for data analysis and manipulation. It provides easy-to-use data structures modeled after relational database tables and R data frames.
Core Data Structures
- Series: A 1D labeled array capable of holding any data type.
- DataFrame: A 2D labeled data structure with columns of potentially different types (essentially a table of Series).
Selection and Indexing: loc vs iloc
Data alignment is index-based. Selecting rows and columns is performed via two primary properties:
df.loc[...]— Label-based indexing (selects data by index name or column name).df.iloc[...]— Integer-position based indexing (selects data by numerical index values from 0 to N).
Group-By and Aggregations
Similar to SQL's GROUP BY, Pandas uses a Split-Apply-Combine process to group data by column values and apply aggregates (mean, sum, count, etc.).
Use Cases
ETL Pipelines — Reading, transforming, and cleaning messy CSV/Excel datasets for storage in SQL databases.
Feature Engineering — Transforming tabular datasets into feature matrices for machine learning algorithms.
Financial Analysis — Aggregating and joining time-series pricing data.
Common Mistakes
SettingWithCopyWarning — Modifying a slice of a DataFrame (which might be a view) instead of using `loc` or creating an explicit copy.
Iterating with loops — Iterating over DataFrame rows using `for index, row in df.iterrows()` (use vectorized operations or `df.apply()` instead for speed).
Confusing axis parameters — Forgetting that `axis=0` aggregates down the column while `axis=1` aggregates across the row.