What you do with Pandas

• Read data from CSV, JSON, Parquet, Excel, and SQL.
• Handle missing values, outliers, and type casting.
• Feature engineering: groupby, rolling windows, one-hot encoding.
• Merge multiple tables with joins.
• Reshape wide/long data and build pivot tables.
• Work with time-series indices, resampling, and rolling statistics.

Key concepts

Reading and writing data

Prefer Parquet over CSV in production pipelines — it preserves dtypes, compresses well, and loads 5-10x faster. Use usecols to avoid loading columns you do not need.

import pandas as pd

# ── Reading ───────────────────────────────────────────────────
df = pd.read_csv("data.csv")
df = pd.read_csv("data.csv", usecols=["age","salary"], dtype={"age": "Int32"})
df = pd.read_json("data.json")
df = pd.read_parquet("data.parquet")          # fast, typed, compressed
df = pd.read_excel("data.xlsx", sheet_name=0)
df = pd.read_sql("SELECT * FROM users", con=engine)  # SQLAlchemy engine

# ── Writing ───────────────────────────────────────────────────
df.to_csv("out.csv", index=False)
df.to_parquet("out.parquet", index=False)
df.to_json("out.json", orient="records", lines=True)

# ── Quick inspection ──────────────────────────────────────────
df.shape          # (rows, cols)
df.dtypes         # column types
df.head(5)        # first 5 rows
df.tail(5)        # last 5 rows
df.info()         # dtypes + non-null counts
df.describe()     # stats for numeric columns
df.sample(10)     # random sample

Selecting rows and columns

Always use .loc for label-based selection and assignment. Chained indexing like df[...][...] = value may silently write to a copy instead of the original DataFrame.

import pandas as pd

# Column selection
df["age"]                      # Series
df[["age", "salary"]]          # DataFrame (double brackets)

# Row selection by label (use .loc for all assignment)
df.loc[0]                      # single row by index label
df.loc[0:4, "age":"salary"]    # label slice (inclusive on both ends)

# Row selection by integer position
df.iloc[0]                     # first row
df.iloc[0:5, 0:3]             # first 5 rows, first 3 cols

# Boolean filtering
df[df["age"] > 30]
df[(df["age"] > 30) & (df["salary"] < 80_000)]
df[df["city"].isin(["Mumbai", "Delhi"])]
df[df["name"].str.startswith("A")]

# Safe assignment — always use .loc, never chain
df.loc[df["age"] > 30, "senior"] = True

Handling missing values

Always audit missing data before modeling. Whether you drop, fill, or impute depends on the fraction missing and whether missingness is informative.

import pandas as pd

# Detect
df.isnull().sum()                    # count nulls per column
df.isnull().mean()                   # fraction missing per column
df[df["age"].isnull()]               # rows where age is missing

# Drop
df.dropna()                          # drop any row with a null
df.dropna(subset=["age", "salary"])  # drop only if these cols are null
df.dropna(thresh=3)                  # keep rows with at least 3 non-nulls

# Fill
df["age"].fillna(df["age"].median())
df["city"].fillna("Unknown")
df.ffill()                           # forward-fill (time series)
df.bfill()                           # backward-fill

# Interpolate (numeric, time series)
df["price"].interpolate(method="linear")

dtypes and casting

Pandas uses object dtype for mixed or string columns, which is slow and memory-hungry. Cast strings to category for low-cardinality columns and use nullable integer types (Int32, not int32) when nulls are possible.

import pandas as pd

df.dtypes                            # inspect all dtypes

# Cast
df["age"]    = df["age"].astype("Int32")        # nullable integer
df["score"]  = df["score"].astype("float32")
df["label"]  = df["label"].astype("category")   # saves memory for low-cardinality

# Dates
df["date"] = pd.to_datetime(df["date"])
df["date"] = pd.to_datetime(df["date"], format="%Y-%m-%d")

# Numeric coercion (turns bad strings into NaN instead of raising)
df["revenue"] = pd.to_numeric(df["revenue"], errors="coerce")

Cleaning and reshaping

import pandas as pd

# Rename columns
df.rename(columns={"oldname": "newname"}, inplace=True)
df.columns = df.columns.str.lower().str.replace(" ", "_")  # normalize all

# Drop duplicates
df.drop_duplicates()
df.drop_duplicates(subset=["user_id"])

# Drop columns
df.drop(columns=["noise_col", "id"])

# Clip outliers
df["salary"] = df["salary"].clip(lower=0, upper=300_000)

# Reset index after filtering
df = df[df["age"] > 18].reset_index(drop=True)

# Sort
df.sort_values("salary", ascending=False)
df.sort_values(["country", "salary"], ascending=[True, False])

Feature engineering

Prefer vectorized column operations over .apply(lambda ...) — they are 10-100x faster. Reserve .apply() for logic that genuinely cannot be expressed with built-in methods.

import pandas as pd

# Arithmetic features
df["bmi"] = df["weight"] / (df["height"] / 100) ** 2
df["log_salary"] = df["salary"].apply(lambda x: x ** 0.5)  # last resort

# Binning
df["age_group"] = pd.cut(df["age"], bins=[0,18,35,60,100],
                          labels=["teen","young","mid","senior"])
df["age_bucket"] = pd.qcut(df["age"], q=4, labels=False)   # quartile bins

# One-hot encoding
df = pd.get_dummies(df, columns=["city", "gender"], drop_first=True)

# Label encoding
df["label_enc"] = df["category"].astype("category").cat.codes

# String features
df["name_len"]   = df["name"].str.len()
df["first_word"] = df["text"].str.split().str[0]
df["contains_ai"] = df["text"].str.contains("AI", case=False)

# Date features
df["year"]    = df["date"].dt.year
df["month"]   = df["date"].dt.month
df["weekday"] = df["date"].dt.dayofweek
df["is_weekend"] = df["weekday"].isin([5, 6])

GroupBy — split, apply, combine

agg reduces each group to a scalar. transform returns a result the same length as the input — perfect for adding group-level features back to the original DataFrame without a merge.

import pandas as pd

# Basic aggregation
df.groupby("country")["salary"].mean()
df.groupby("country")["salary"].agg(["mean", "median", "std", "count"])

# Multiple columns
df.groupby(["country", "gender"])["salary"].mean()

# Named aggregations (pandas >= 0.25)
df.groupby("country").agg(
    avg_salary=("salary", "mean"),
    max_age=("age", "max"),
    n_users=("user_id", "count"),
)

# Transform — same shape as input (for feature engineering)
df["salary_vs_country_mean"] = df.groupby("country")["salary"].transform("mean")
df["salary_rank"] = df.groupby("country")["salary"].rank(pct=True)

# Filter groups
df.groupby("country").filter(lambda g: len(g) > 100)

Merging and joining tables

pd.merge is the most flexible — it mirrors SQL JOIN syntax. pd.concat stacks DataFrames vertically (more rows) or horizontally (more columns). Always verify row counts after a merge to catch fan-out bugs.

import pandas as pd

# Merge (SQL JOIN equivalent)
result = pd.merge(users, orders, on="user_id", how="left")
result = pd.merge(users, orders, left_on="id", right_on="user_id", how="inner")

# Concat (stack vertically or horizontally)
combined = pd.concat([df_2022, df_2023], ignore_index=True)
wide     = pd.concat([df_features, df_labels], axis=1)

# join (index-based merge)
df1.join(df2, how="left")

# Common how values:
# inner  — keep rows that match in both
# left   — keep all left rows; NaN for missing right
# outer  — keep all rows from both
# right  — keep all right rows; NaN for missing left

Reshaping — pivot, melt, stack

pivot_table converts long format to wide (rows become columns). melt goes the other direction — wide to long — which is often what plotting libraries and tidy-data tools expect.

import pandas as pd

# Pivot table (aggregation + reshape)
pivot = df.pivot_table(
    values="sales",
    index="month",
    columns="product",
    aggfunc="sum",
    fill_value=0,
)

# Melt (wide -> long)
long = pd.melt(df, id_vars=["user_id"], value_vars=["jan","feb","mar"],
               var_name="month", value_name="sales")

# Stack / unstack (multi-index)
stacked   = df.stack()      # columns -> innermost index level
unstacked = stacked.unstack()

# Explode (list column -> rows)
df["tags"] = df["tags"].str.split(",")
df = df.explode("tags").reset_index(drop=True)

Time-series operations

Set a DatetimeIndex to unlock resample, rolling, and shift. These are the building blocks for lag features, moving averages, and seasonality extraction in time-series ML.

import pandas as pd

df = pd.read_csv("prices.csv", parse_dates=["date"], index_col="date")

# Resample (downsample to monthly)
monthly = df["close"].resample("ME").mean()

# Rolling stats
df["ma_7"]  = df["close"].rolling(7).mean()
df["std_30"] = df["close"].rolling(30).std()

# Lag / lead features
df["prev_close"] = df["close"].shift(1)
df["next_close"] = df["close"].shift(-1)

# Percent change
df["returns"] = df["close"].pct_change()

# Date range index
idx = pd.date_range("2024-01-01", periods=365, freq="D")

Method quick-reference

Interoperability

Pandas sits at the centre of the Python ML stack. Use .to_numpy() or .values to hand data to sklearn and PyTorch, and Dataset.from_pandas(df) to move into HuggingFace Datasets.

import pandas as pd
import numpy as np

# To / from NumPy
arr = df.to_numpy()            # loses column names
arr = df["salary"].values      # underlying NumPy array (may share memory)
df2 = pd.DataFrame(arr, columns=df.columns)

# To sklearn
from sklearn.preprocessing import StandardScaler
X = df[["age","salary"]].to_numpy()
X_scaled = StandardScaler().fit_transform(X)

# To PyTorch
import torch
tensor = torch.tensor(df[["age","salary"]].values, dtype=torch.float32)

# To HuggingFace Dataset
from datasets import Dataset
hf_ds = Dataset.from_pandas(df)

Common pitfalls — in depth