How to use PySpark with Jupyter Notebooks: A Comprehensive Guide


link to this section

Apache Spark is a powerful open-source distributed computing system used for big data processing and analytics. PySpark is the Python library for Spark, providing an easy-to-use API for leveraging the power of Spark with Python. Jupyter Notebooks, on the other hand, is an interactive web-based platform that enables users to create and share live code, equations, visualizations, and narratives. Combining these two tools allows for seamless big data processing, exploration, and visualization in a highly interactive environment.

In this blog post, we will provide a step-by-step guide on how to integrate PySpark with Jupyter Notebooks. We will cover the installation and setup process, and provide examples of how to use PySpark in a Jupyter Notebook for data processing and analysis.

Installation and setup

link to this section

Before we begin, ensure that you have Python and pip installed on your system. If not, download and install Python from the official website ( ).

Step 1: Install PySpark and Jupyter Notebook

Use pip, the Python package manager, to install both PySpark and Jupyter Notebook. Open your terminal or command prompt and run the following commands:

pip install pyspark 
pip install jupyter 

Step 2: Set up environment variables

We need to set up the necessary environment variables for PySpark to work with Jupyter Notebook. Use the following commands in your terminal or command prompt:

For Linux and macOS:


For Windows (using Command Prompt):


For Windows (using PowerShell):

$env:PYSPARK_DRIVER_PYTHON = "jupyter" 

Step 3: Start the Jupyter Notebook

Launch the Jupyter Notebook server with PySpark by running the following command in your terminal or command prompt:

jupyter notebook

This will open the Jupyter Notebook web interface in your default web browser.

Using PySpark in Jupyter Notebooks

link to this section

Now that we have our Jupyter Notebook server up and running with PySpark, let's dive into some examples of how to use PySpark in a Jupyter Notebook.

Example 1: Creating a SparkSession

To start using PySpark, you need to create a SparkSession, which is the entry point for any Spark functionality. In your Jupyter Notebook, enter the following code:

from pyspark.sql import SparkSession 
spark = SparkSession.builder \ 
    .appName("PySpark Jupyter Notebook Integration") \ 

Example 2: Loading data

To load data into a PySpark DataFrame, you can use the read method of the SparkSession object. Here's an example of loading a CSV file:

csv_file_path = "path/to/your/csv_file.csv" 
dataframe =, header=True, inferSchema=True) 

Replace "path/to/your/csv_file.csv" with the path to your CSV file. The header=True option indicates that the first row of the CSV file contains column names, and inferSchema=True tells PySpark to infer the schema (data types) of the columns automatically.

Example 3: Data processing

Once you have your data loaded, you can perform various data processing tasks using PySpark's built-in functions. For example, let's say we want to count the number of occurrences of each unique value in a specific column. We can use the groupBy and count functions to achieve this:

from pyspark.sql.functions import col 
column_name = "your_column_name" 

grouped_data = dataframe.groupBy(col(column_name)).count() 

Replace "your_column_name" with the name of the column you want to analyze.

Example 4: Filtering data

To filter data based on certain conditions, you can use the filter or where functions. Here's an example of filtering rows where the value in a specific column is greater than a given threshold:

threshold = 10 
filtered_data = dataframe.filter(col(column_name) > threshold) 

Example 5: Joining DataFrames

If you need to join two DataFrames based on a common key, you can use the join function. In this example, we will join two DataFrames on the "id" column:

dataframe1 ="path/to/your/csv_file1.csv", header=True, inferSchema=True) 
dataframe2 ="path/to/your/csv_file2.csv", header=True, inferSchema=True) 
joined_data = dataframe1.join(dataframe2, on="id") 

Replace the file paths with the paths to your CSV files.

Example 6: Saving data

After processing your data, you might want to save the results to a file. You can use the write method to save a DataFrame in various formats, such as CSV, JSON, or Parquet. Here's an example of saving a DataFrame as a CSV file:

output_file_path = "path/to/your/output/csv_file.csv" 
dataframe.write.csv(output_file_path, mode="overwrite", header=True) 

Replace "path/to/your/output/csv_file.csv" with the desired output file path. The mode="overwrite" option tells PySpark to overwrite the file if it already exists, and header=True indicates that the output file should include column names.


link to this section

In this blog post, we have shown you how to set up and use PySpark with Jupyter Notebooks for big data processing and analysis. By following this guide, you can now leverage the power of Apache Spark and the convenience of Jupyter Notebooks for your big data projects.

Remember that you need to set up the environment variables every time you open a new terminal or command prompt session to work with PySpark in Jupyter Notebook. Alternatively, you can create a shell script or batch file to automate the process of setting up the environment variables and starting the Jupyter Notebook server with PySpark.

We hope you find this guide helpful and enjoy exploring the capabilities of PySpark in Jupyter Notebooks!