Mastering Pandas DataFrame fillna() for Handling Missing Data

Dealing with missing data is a common challenge in data analysis and manipulation. Pandas, the widely-used Python library for data manipulation, offers a powerful method to handle missing values – the fillna() function. In this blog post, we will delve deep into using fillna() on DataFrames, covering various scenarios and options.

Introduction to fillna()

link to this section

The fillna() function is used to replace NaN or null values in a DataFrame with a specific value, or a method-based imputation. The function signature is as follows:

DataFrame.fillna(value=None, method=None, axis=None, inplace=False, limit=None) 
  • value : Scalar, dict, Series, or DataFrame. The value to use to fill missing values.
  • method : {'backfill', 'bfill', 'pad', 'ffill', None}, default None. Method to use for filling holes in reindexed Series.
  • axis : {0 or 'index', 1 or 'columns'}, default None.
  • inplace : Boolean, default False. If True, fill in-place.
  • limit : Int, default None. If method is specified, this is the maximum number of consecutive NaN values to forward/backward fill.

Filling NaN Values with a Specific Value

link to this section

You can replace all NaN values in a DataFrame with a specific value:

import pandas as pd 
import numpy as np 

# Sample DataFrame with NaN values 
data = {'Name': ['John', 'Anna', np.nan, 'Linda'], 
    'Age': [28, np.nan, 34, 29], 
    'City': ['New York', 'Paris', 'Berlin', np.nan]} 
df = pd.DataFrame(data) 

# Replace all NaN values with a specific value 
df.fillna('Unknown', inplace=True) 

Using a Dictionary to Replace NaN

link to this section

You can use a dictionary to replace NaN values with different values for each column:

# Replace NaN values with different values for each column 
df.fillna({'Name': 'No Name', 'Age': 0, 'City': 'No City'}, inplace=True) 

Forward Filling and Backward Filling

link to this section

You can fill NaN values using the forward fill or backward fill method:

# Forward fill 
df.fillna(method='ffill', inplace=True) 

# Backward fill 
df.fillna(method='bfill', inplace=True) 

Limiting the Number of NaN Values to Fill

link to this section

You can limit the number of consecutive NaN values to fill:

# Limit the number of NaN values to fill 
df.fillna(0, limit=1, inplace=True) 


link to this section

Handling missing data is crucial in ensuring the accuracy and reliability of your analysis. The fillna() function in Pandas provides a versatile and powerful tool to address this issue, offering various methods to replace or impute missing values. Whether you’re dealing with a small dataset or large, mastering the use of fillna() is essential for any data scientist or analyst.