--- title: "How to Install Apache Spark for Windows" description: "Learn how to process and transform large datasets on Windows with Apache Spark in this in-depth ATA Learning tutorial!" canonical: "https://adamtheautomator.com/apache-spark/" --- # How to Install Apache Spark for Windows > Learn how to process and transform large datasets on Windows with Apache Spark in this in-depth ATA Learning tutorial! Source: https://adamtheautomator.com/apache-spark/ --- ATA Learning Tap to hide [ ATA Learning ](/) * [Home](/) * [Tutorials](/tutorials/) * [Instructors](/author/) * [Advertising](/advertising/) * [Recommended Resources](/resources/) * [About Adam](/about-adam/) Search for: * [](https://twitter.com/adbertram) * [](https://github.com/Adam-the-Automator) * [](https://www.linkedin.com/company/adam-the-automator-llc) * [](/feed/) ![How to Install Apache Spark for Windows](https://adamtheautomator.com/wp-content/uploads/2023/11/apache-spark.jpg) # How to Install Apache Spark for Windows [![](https://secure.gravatar.com/avatar/1bc5e1d52466fc3739f550c8d78be310684747bf1466a98d698320627ea243e2?s=192&d=mm&r=g)Michael Nguyen Tu](https://adamtheautomator.com/author/michael-nguyen-tu/)15 November 20238 min. read Categories: [IT Ops](/category/it-ops/) Tags:[Apache](/tag/apache/)[Apache Spark](/tag/apache-spark/) Table of Contents * [Prerequisites](#prerequisites) * [Installing Python for Apache Spark on Windows](#installing-python-for-apache-spark-on-windows) * [Downloading and Installing Apache Spark](#downloading-and-installing-apache-spark) * [Installing the winutils.exe Utility for Required Functionalities](#installing-the-winutilsexe-utility-for-required-functionalities) * [Setting an Environment Variable for Hadoop and Spark Integration](#setting-an-environment-variable-for-hadoop-and-spark-integration) * [Launching Spark Shells for Interactive Spark Programming](#launching-spark-shells-for-interactive-spark-programming) * [Testing Your Spark Setup with Real-world Examples](#testing-your-spark-setup-with-real-world-examples) * [Conclusion](#conclusion) In the ever-expanding realm of massive data, the need for tools that can efficiently handle and process vast datasets is more crucial than ever. If you wish to unlock the transformative power of data analysis, set up Apache Spark on your Windows machine and embark on a data exploration journey. Throughout this tutorial, you’ll discover the ropes of installing Apache Spark on your Windows system, positioning yourself to conquer the fascinating world of data analysis. Transform your Windows environment into a data processing powerhouse! ## Prerequisites Before diving into the installation process, ensure you have all the necessary prerequisites in place. * A Windows 7 system or later – This tutorial uses Windows 10 Pro 22H2. * [Java](https://www.java.com/en/download/help/download_options.html) installed – This tutorial uses Java 1.8.0\_391. Related:[Related: How to Install Java on Mac](https://adamtheautomator.com/install-java-on-mac/) * [Apache Subversion for Windows](https://subversion.apache.org/packages.html#windows) installed – This tutorial uses SlikSvn 1.14.2 (64-bit). ## Installing Python for Apache Spark on **Windows** As you ensure your Windows environment is primed for the upcoming Apache Spark installation, one essential step stands before you — installing Python. Python serves as the glue that enhances the interoperability and extensibility of Spark. From handling data manipulations to running Spark scripts, Python acts as a powerful catalyst. To install Python for Apache Spark on Windows, follow these steps: 1\. Open your favorite web browser, visit the [official Python download page](https://www.python.org/downloads/), and download the latest Python Installer. At this time of writing, the latest version is Python 3.12.0. ![Downloading Python for Apache Spark on Windows](https://adamtheautomator.com/wp-content/uploads/2023/11/image-74.png) Downloading Python for Apache Spark on Windows 2\. Once downloaded, double-click on the installer to begin the installation process. 3\. On the Setup window, tick the **Add python.exe to PATH** option, and click **Install Now**. Enabling this option adds Python to your PATH environment variable, which eases the management of Python packages via package managers like `pip` or `anaconda`. As a result, you can install and manage Python libraries without additional configuration. Related:[How To Install Anaconda on Linux for Data Science Wins!](https://adamtheautomator.com/how-to-install-anaconda/) ![Continuing with the Python installation](https://adamtheautomator.com/wp-content/uploads/2023/11/image-73.png) Continuing with the Python installation 4\. Now, select the Disable path length limit option, and click Close to close the Setup wizard. This option allows Python to use the “Long Paths” feature to access paths longer than the traditional [MAX\_PATH limit](https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry) (260 characters) on Windows. This feature is beneficial as some of the packages used by Apache Spark exceed the MAX\_PATH limit. ![Disabling the path length limit and closing the installer](https://adamtheautomator.com/wp-content/uploads/2023/11/image-72.png) Disabling the path length limit and closing the installer 5\. Lastly, open PowerShell and run the below commands to verify your Python installation. Related:[Run PowerShell as Administrator: Multiple Methods Explored](https://adamtheautomator.com/powershell-run-as-administrator/) ```powershell pip--version python --version ``` ![Checking the installed versions of the pip package manager and Python](https://adamtheautomator.com/wp-content/uploads/2023/11/image-71.png) Checking the installed versions of the pip package manager and Python ## Downloading and Installing Apache Spark With Python installed, you can now lay the foundation for dynamic capabilities to flourish in your Windows environment. Installing Apache Spark opens a world of possibilities where data-driven insights are at your fingertips. To install Apache Spark on Windows, proceed with the following: 1\. Visit the [official Apache Spark download page](https://spark.apache.org/downloads.html) in another web browser tab. 2\. Next, download the Apache Spark installation package as follows: * **Choose a Spark release** – Select the latest release from the dropdown field (i.e., **3.5.0**). * **Choose a package type** – Pick a suitable one depending on whether you want to use Hadoop. Your version of Spark and Hadoop may slightly differ. For this example, choose **Pre-built for Apache Hadoop 3.3 and later** option, which is best if you use Spark with Hadoop. Otherwise, select the option without Hadoop. * **Download Spark** – Click on the hyperlink, which opens a new tab (step three) to select your Apache Spark download location. ![Choosing options for downloading Apache Spark](https://adamtheautomator.com/wp-content/uploads/2023/11/image-80.png) Choosing options for downloading Apache Spark 3\. Click the first hyperlink or choose one from the alternate download locations to download the Apache Spark package. ![Downloading the Apache Spark package](https://adamtheautomator.com/wp-content/uploads/2023/11/image-79.png) Downloading the Apache Spark package 4\. Once downloaded, execute the below command and generate a SHA512 hash of the downloaded file. Ensure you replace `spark-3.5.0-bin-hadoop3.tgz` with the exact file name of the one you downloaded (or the full path). ```powershell certutil -hashfile .\spark-3.5.0-bin-hadoop3.tgz SHA512 ``` Related:[A Guide to Using Certutil for Certificate Management](https://adamtheautomator.com/certutil/) Note down the generated hash, as you will make a comparison later. ![Generating a SHA512 hash of the Apache Spark package](https://adamtheautomator.com/wp-content/uploads/2023/11/image-78.png) Generating a SHA512 hash of the Apache Spark package 5\. Switch back to the download page and click the checksum hyperlink. Doing so opens the hash file containing the expected SHA512 checksum for your chosen download (step six). ![Opening the checksum link](https://adamtheautomator.com/wp-content/uploads/2023/11/image-77.png) Opening the checksum link 6\. Compare the generated hash you noted in step four with the one in the checksum file. Matching both hashes indicates your download was not corrupted or tampered with, and you can proceed with extracting the downloaded archive. ![Comparing the generated hash with the checksum](https://adamtheautomator.com/wp-content/uploads/2023/11/image-76.png) Comparing the generated hash with the checksum 7\. Lastly, unzip/extract the contents of the Apache Spark package using tools like 7-Zip or WinRAR to a directory of your choice. For example, you can extract everything to the _C:\\spark_ folder, which effectively installs Apache Spark in your root directory. Related:[Learn How to Zip and Unzip Files in Linux](https://adamtheautomator.com/unzip-files-in-linux/) After extraction, the Apache Spark installation directory structure should resemble the following: ![Viewing the folder structure](https://adamtheautomator.com/wp-content/uploads/2023/11/image-75.png) Viewing the folder structure ## Installing the winutils.exe Utility for Required Functionalities Now that Apache Spark has found its home on your Windows machine, the next step involves a nuanced touch — installing the winutils.exe utility. Apache Spark requires the winutils.exe utility to run smoothly on Windows. This utility is essential for enabling the required file and directory operations functionality. The winutils.exe utility is crucial because these operations are typically available on Linux-based systems but need additional support on Windows. To install the winutils.exe utility, complete the steps below: 1\. Visit the [GitHub repository](https://github.com/cdarlint/winutils) that provides the _winutils.exe_ utility for Windows and look for the latest Hadoop version. Hadoop is an open-source framework designed for the distributed storage and processing of large data sets. At this time of writing, the latest revision is _hadoop-3.3.5_, as shown below. ![](https://adamtheautomator.com/wp-content/uploads/2023/11/image-83.png) 2\. Next, run the following command to download the contents of the `bin` folder (_hadoop-3.3.5/bin_) to a folder called _`C:\hadoop\bin`._ > _💡 If you see a newer version, change the folder name `hadoop-3.3.5` and the revision number (`r335`) accordingly (i.e., `hadoop-3-3.6` and `r336`)._ This process effectively installs the _winutils.exe_ utility on your Windows system. ```bash svn export https://github.com/cdarlint/winutils/trunk/hadoop-3.3.5/bin@r335 "C:\hadoop\bin" ``` ![Downloading the latest Hadoop version](https://adamtheautomator.com/wp-content/uploads/2023/11/image-82.png) Downloading the latest Hadoop version 3\. Once downloaded, navigate to the _C:\\hadoop\\bin_ folder and confirm you have the following structure. You’ll see files, including the winutils.exe utility, that are necessary to run Hadoop operations on Windows. ![Verifying the winutils.exe utility](https://adamtheautomator.com/wp-content/uploads/2023/11/image-81.png) Verifying the _winutils.exe_ utility ## Setting an Environment Variable for Hadoop and Spark Integration Having laid the groundwork by installing the winutils.exe utility, your journey now takes a turn into integration. You’ll set up the `HADOOP_HOME` environment variable to ensure Spark can locate and use the necessary Hadoop components. To set the `HADOOP_HOME` environment variable, carry out the following: Execute the below [`Set-Item`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/set-item) command to create a new `HADOOP_HOME` environment variable and set its value to Hadoop’s install location (`C:\hadoop`). When successful, this command does not provide output to the console, but you’ll verify the result in the following step. ```powershell Set-Item -Path "Env:\HADOOP_HOME" -Value 'C:\hadoop' ``` Related:[PowerShell Environment Variables: A Deep Dive](https://adamtheautomator.com/powershell-environment-variables/) Next, run the following command to verify the `HADOOP_HOME` environment variable. ```powershell $env:HADOOP_HOME ``` The command returns the value of the `HADOOP_HOME` environment variable, which in this case is Hadoop’s install location (_**C:\\hadoop**_). ![Verifying the HADOOP\_HOME variable](https://adamtheautomator.com/wp-content/uploads/2023/11/image-84.png) Verifying the `HADOOP_HOME` variable ## Launching Spark Shells for Interactive Spark Programming With a fine-tuned environment for seamless Hadoop and Spark integration, prepare yourself for interactive Spark programming. You’ll launch Spark Shells, an interactive command-line interface (CLI) provided by Apache Spark. Spark Shells allow you to interactively work with Spark, where you can execute Spark jobs and explore your data via Spark’s APIs. Below are the two main Spark Shells, depending on your preferred programming language: | Spark Shell | Details | | --- | --- | | Scala Spark Shell | Scala is the default Spark Shell and is primarily used for writing Spark applications in the Scala programming language. | | PySpark Shell | PySpark is the Python API for Apache Spark, and the PySpark Shell is used for writing Spark applications in Python. This Spark Shell is similar to the Scala Spark Shell but for Python developers. | To launch Spark Shells, execute the following: 1\. Run each `Set-Item` command below to set environment variables for Spark Shell and PySpark Shell (i.e., `SPARK_SH` and `PY_SPARK`). These commands have no output on the console but set environment variables to store the full paths of both shells, respectively. Ensure you replace the full paths (`C:\spark\spark-3.5.0-bin-hadoop3\bin\`) with where you unzipped Spark, followed by the shells (`spark-shell` and `pyspark`). This one-time process saves time and is convenient as you can quickly launch either of the shells without typing or specifying the full path each time. ```powershell Set-Item -Path "Env:\SPARK_SH" -Value 'C:\spark\spark-3.5.0-bin-hadoop3\bin\spark-shell' Set-Item -Path "Env:\PY_SPARK" -Value 'C:\spark\spark-3.5.0-bin-hadoop3\bin\pyspark' ``` 2\. Next, execute the below `Start-Process` command to launch the Scala Spark Shell (`$env:SPARK_SH`). You only need to call the environment variable that holds the Scala Spark Shell’s full path instead of manually typing everything. ```powershell Start-Process $env:SPARK_SH ``` Related:[Start-Process in PowerShell: Intuitive Process Launching](https://adamtheautomator.com/start-process/) As you launch the Scala Spark Shell, a new console opens where you’ll see the Scala Read-Eval-Print Loop (REPL) prompt (**scala>**). This prompt lets you interactively enter and execute Scala code with access to Spark APIs. ![Launching the Scala Spark Shell](https://adamtheautomator.com/wp-content/uploads/2023/11/image-87.png) Launching the Scala Spark Shell 3\. When prompted, click **Allow access** Java through Windows firewall, as shown below. Otherwise, you may encounter issues launching the Spark Shell. This prompt appears since Apache Spark uses a Java-based web server, Jetty, to display its Web UIs. ![Allowing Java through Windows firewall](https://adamtheautomator.com/wp-content/uploads/2023/11/image-86.png) Allowing Java through Windows firewall 4\. Now, switch back to the previous PowerShell console and execute the following command to launch the PySpark Shell (`$env:PY_SPARK`). Like with the Scala Spark Shell, you only need to call the environment variable that holds the full path of the PySpark Shell. ```powershell Start-Process $env:PY_SPARK ``` The PySpark Shell opens in another console, where you’ll see the Python REPL prompt (**\>>>**). This prompt lets you interactively enter and execute Python code with access to Spark’s PySpark API. ![Launching the PySpark Shell](https://adamtheautomator.com/wp-content/uploads/2023/11/image-85.png) Launching the PySpark Shell 5\. Press Ctrl+D, regardless of which shell you launched, to exit the shell, which also closes the console. ## **Testing Your Spark Setup with Real-world Examples** After venturing into the dynamic realm of launching Spark Shells for interactive programming, the time has come to put your Spark setup to the test. In this example, you’ll perform a basic analysis of a CSV file containing information about movie ratings via PySpark’s DataFrame API. To test your Spark setup, perform these steps: 1\. Download the [sample CSV file](https://www.mediafire.com/file/tfr9ypfbu371r3t/movie_ratings.csv/file) containing movie ratings to a directory of your choice (i.e., _C:\\logs\\movie\_ratings.csv_). Remember the full path, as you will need it for the following step. 2\. Next, launch the PySpark shell and run the following command, which has no output but loads the CSV data (`movie_ratings.csv`) into a DataFrame. Replace the path (`C:\\logs\\movie_ratings.csv`) with the one where you saved the sample CSV file in step one. ```python df = spark.read.csv("C:\\logs\\movie_ratings.csv", header=True, inferSchema=True) ``` 3\. Afterward, run the below commands to perform basic data exploration tasks. ```python # Show the first few rows of the DataFrame df.show() # Display the schema of the DataFrame df.printSchema() # Compute basic statistics of the "rating" column df.describe("rating").show() ``` ![Performing basic data exploration tasks](https://adamtheautomator.com/wp-content/uploads/2023/11/image-89.png) Performing basic data exploration tasks 4\. Lastly, run each command below to perform a basic data analysis task that calculates the average rating for each movie. ```python # Import necessary functions from pyspark.sql.functions import avg # Calculate the average rating for each movie average_ratings = df.groupBy("movie_id").agg(avg("rating").alias("avg_rating")) average_ratings.show() ``` ![Performing a basic data analysis task](https://adamtheautomator.com/wp-content/uploads/2023/11/image-88.png) Performing a basic data analysis task ## Conclusion You’ve successfully navigated the twists and turns of setting up Apache Spark on your Windows machine. Moreover, you seamlessly integrated Spark with Hadoop, launched Spark Shells, and tested your Spark setup with real-world examples. Throughout this journey, here you stand, having conquered each step with resilience and determination. Armed with a Windows environment primed for data analysis, consider this stage as just the beginning. Continue experimenting with diverse datasets to refine your skills. Now, why not consider exploring advanced Spark features, such as creating [Resilient Distributed Datasets](https://spark.apache.org/docs/latest/rdd-programming-guide.html) (RDDs), distributed processing, and parallel computing? Spark has a rich set of APIs for various programming languages like Scala, [Python](https://sparkbyexamples.com/pyspark-tutorial/), Java, R, and [SQL](https://spark.apache.org/sql/). Use these APIs to develop robust data applications to process massive datasets in real-time! Related:[Monitor SQL Server Performance with PowerShell](https://adamtheautomator.com/sql-server-performance-monitor/) Share this article [Share on X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fadamtheautomator.com%2Fapache-spark%2F&text=How%20to%20Install%20Apache%20Spark%20for%20Windows)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fadamtheautomator.com%2Fapache-spark%2F)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fadamtheautomator.com%2Fapache-spark%2F) ## Related Posts ![](https://adamtheautomator.com/wp-content/uploads/2022/04/How-to-Install-Apache-on-Ubuntu.jpg) ### [Install Apache on Ubuntu and Master Website Traffic Management](/install-apache-on-ubuntu/) Ready to optimize your website for high traffic? Install Apache on Ubuntu with this step-by-step guide and ensure smooth performance under heavy loads. ![](https://adamtheautomator.com/wp-content/uploads/2024/02/apache-hadoop.jpg) ### [How to Install Apache Hadoop for Linux](/apache-hadoop/) Unlock Apache Hadoop on Linux: Transform data handling with efficient, scalable processing and analysis in this ATA tutorial! ![](https://adamtheautomator.com/wp-content/uploads/2023/10/pagespeed.jpg) ### [Speed Up Your Apache Web Server with the Pagespeed Module](/pagespeed/) Learn how to make your Apache Web Server even quicker with the Pagespeed Module in this ATA Learning tutorial! ## Categories * [IT Ops](/category/it-ops/) * [Cloud](/category/cloud/) * [DevOps](/category/devops/) * [Home Ops](/category/home-ops/) * [Information Security](/category/infosec/) * [Software Development](/category/software-development/) ## Site * [Home](/) * [Tutorials](/tutorials/) * [Instructors](/author/) * [Advertising](/advertising/) * [Recommended Resources](/resources/) * [About Adam](/about-adam/) Copyright 2026© ATA Learning | [Privacy Policy](/privacy/)