top of page
BlogPageTop

Installing Spark on Windows (pyspark)

Updated: Nov 18, 2022

Prerequisite: Follow these steps to install Apache Spark on windows machine.


Now-a-days Python is used by many applications. So it is quite possible that Python is already available on your machine. To check, just run this command on your command prompt.


C:\Users\rajar> python --version

'python' is not recognized as an internal or external command,

operable program or batch file.


If Python is present on your computer, command will output the Python version like this.


Python x.x.x


Check if Java is properly installed, just run java -version and you should be able to see Java version running on your computer.


 

Download & Install Python


Go to Python download page and download the latest version (don't download Python 2).



Download 64 bit or 32 bit installer depending upon your system configuration.



Double click on the downloaded executable file.


Don't forget to check the box - Add Python 3.7 to PATH , then click Install now. Thats all, it will take couple of minutes to complete the installation.


Now test it, run previous command again and you should be able to see Python version this time.


C:\Users\rajar> python --version


Python 3.7.4

 

Run pyspark


Now, run the command pyspark and you should be able to see the Spark version.


If you have any question please mention in comments section below and I will help you out with installation process. Thank you.



Next: Just enough Scala for Spark


Navigation menu

​

1. Apache Spark and Scala Installation

1.1 Spark installation on Windows​

1.2 Spark installation on Mac

2. Getting Familiar with Scala IDE

2.1 Hello World with Scala IDE​

3. Spark data structure basics

3.1 Spark RDD Transformations and Actions example

4. Spark Shell

4.1 Starting Spark shell with SparkContext example​

5. Reading data files in Spark

5.1 SparkContext Parallelize and read textFile method

5.2 Loading JSON file using Spark Scala

5.3 Loading TEXT file using Spark Scala

5.4 How to convert RDD to dataframe?

6. Writing data files in Spark

​6.1 How to write single CSV file in Spark

7. Spark streaming

7.1 Word count example Scala

7.2 Analyzing Twitter texts

8. Sample Big Data Architecture with Apache Spark

9. What's Artificial Intelligence, Machine Learning, Deep Learning, Predictive Analytics, Data Science?

10. Spark Interview Questions and Answers

Interested in sharing your story?

Disclaimer: Please note that the information provided on this website is for general informational purposes only and should not be taken as legal advice. Dataneb is a platform for individuals to share their personal experiences with visa and immigration processes, and their views and opinions may not necessarily reflect those of the website owners or administrators. While we strive to keep the information up-to-date and accurate, we make no representations or warranties of any kind, express or implied, about the completeness, accuracy, reliability, suitability, or availability with respect to the website or the information, products, services, or related graphics contained on the website for any purpose. Any reliance you place on such information is therefore strictly at your own risk. We strongly advise that you consult with a qualified immigration attorney or official government agencies for any specific questions or concerns related to your individual situation. We are not responsible for any losses, damages, or legal disputes arising from the use of information provided on this website. By using this website, you acknowledge and agree to the above disclaimer and Google's Terms of Use (https://policies.google.com/terms) and Privacy Policy (https://policies.google.com/privacy).

RECOMMENDED FROM DATANEB

bottom of page