df.drop_duplicates(subset=['col1', 'col2'], keep='first', inplace=True)

Use boolean indexing or query() for faster filtering:

df_filtered = df.query('col1 > 100 & col2 == "Yes"')

df['ewma'] = df['col'].ewm(span=3, adjust=False).mean()

df.set_index(['Region', 'Product'], inplace=True)
df.loc['North', 'ProductA']

Use chunksize in read_csv() for batch processing

Use Dask, Vaex, or PySpark for out-of-core computations

Downcast numeric types to reduce memory footprint

def range_func(x):
    return x.max() - x.min()
df.groupby('col1')['col2'].agg(range_func)

Use statsmodels.tsa.seasonal_decompose() to decompose series into trend, seasonality, and residuals

from statsmodels.tsa.seasonal import seasonal_decompose
decompose = seasonal_decompose(df['sales'], model='additive', period=12)
decompose.plot()

Techniques include:

• Resampling: oversampling minority class (SMOTE) or undersampling majority class

• Using class-weight parameter in machine learning models

• Ensemble methods like BalancedRandomForestClassifier

• Use seasonal_decompose() from statsmodels

• Plot rolling mean/median to observe trends

• Use autocorrelation (acf) and partial autocorrelation (pacf) plots

• Wide: multiple variables in columns for each observation

• Long: each observation-variable pair is a separate row

• melt() and pivot() in Pandas can convert between formats

• Calculate correlation matrix: df.corr()

• Drop highly correlated features (threshold > 0.8) to reduce multicollinearity

• Use heatmaps for visualization: seaborn.heatmap()

• concat() → flexible, can concatenate along rows or columns, handles multiple DataFrames

• append() → adds rows to a DataFrame (deprecated in recent Pandas versions)

df['new_col'] = df['col'].apply(lambda x: x**2 if x > 0 else 0)

• Converts categorical values into binary columns

• Preferred when categories are nominal (no ordinal relationship) to avoid introducing artificial order

• Use categorical data types for grouping columns

• Reduce memory usage by downcasting numeric columns

• Consider using dask.dataframe.groupby() for out-of-core computation

• Use histograms: df['col'].hist()

• KDE plots: sns.kdeplot(df['col'])

• Boxplots for outliers: sns.boxplot(x=df['col'])

• Use xs() for cross-section selection: df.xs('index_level_value', level='level_name')

• Use loc with tuples for multi-index slicing: df.loc[('Region1','ProductA')]

Missing data can be handled using:

• • df.dropna() → remove missing values
• • df.fillna(value) → fill missing values with a specific value
• • Imputation using mean, median, or mode

• Series: 1-dimensional labeled array, similar to a single column.
• DataFrame: 2-dimensional labeled data structure, similar to a table with rows and columns.

You can use:

• pd.concat([df1, df2]) → for concatenation

• pd.merge(df1, df2, on='column_name', how='inner/left/right/outer') → for database-like joins

Popular libraries include:

• Matplotlib → basic plotting

• Seaborn → statistical plots with attractive designs

• Plotly → interactive visualizations

Use groupby() to aggregate data:

df.groupby('column_name')['another_column'].sum()

• • Detect duplicates: df.duplicated()
• • Remove duplicates: df.drop_duplicates()

• • loc → label-based indexing (row/column names)
• • iloc → integer-based indexing (row/column positions)

Outliers can be handled using:

• Statistical methods (e.g., Z-score, IQR)

• Python example using IQR:

Q1 = df['col'].quantile(0.25)
Q3 = df['col'].quantile(0.75)
IQR = Q3 - Q1
df = df[~((df['col'] < Q1 - 1.5*IQR) | (df['col'] > Q3 + 1.5*IQR))]

• • Use appropriate data types (category for categorical data)
• • Use df.astype() to convert types
• • Load only required columns with usecols in read_csv()

Vectorization is performing operations on entire arrays rather than element-wise loops. It improves performance and speed in numerical computations, mainly using NumPy arrays.

Pivot is used to reshape data:

df.pivot(index='Date', columns='Category', values='Sales')

Time-series analysis involves data indexed over time. Python libraries like Pandas (pd.to_datetime, resample) and statsmodels can be used for trend analysis, seasonal decomposition, and forecasting.

• • merge() → flexible, SQL-style joins using a key
• • join() → simpler, joins on index by default

• • Use correlation matrix (df.corr())
• • Variance Inflation Factor (VIF) calculation using statsmodels
• • High correlation or VIF > 5-10 indicates multicollinearity

• • Normalization: scale values between 0 and 1 using MinMaxScaler
• • Standardization: scale to zero mean and unit variance using StandardScaler from sklearn

• • Label Encoding → integer representation
• • One-Hot Encoding → binary columns
• • Pandas: pd.get_dummies()

• • Shallow copy: references original data, changes affect original
• • Deep copy: independent copy, changes don’t affect original (df.copy(deep=True))

• • map() → element-wise operation on Series
• • apply() → applies a function along Series or DataFrame axis
• • applymap() → element-wise operation on entire DataFrame

df['rolling_mean'] = df['col'].rolling(window=3).mean()

df.set_index(['col1', 'col2'], inplace=True)