What Are Spark, Scala, and PySpark?

Twenty-two million taxi trips. A laptop with 8 GB of memory. Twenty-three seconds.

That is what these tools are for. Here is what each one is, in plain language.

Everything below is reproducible. The companion notebook downloads the data, runs every function, and prints every number in this post. It takes about a minute on a normal laptop.

The problem

Normal data tools load your whole file into memory. Open a 200 MB file and 200 MB goes into RAM. Fine.

Now try a 50 GB file on an 8 GB laptop. It does not get slow. It becomes impossible.

You can buy a bigger computer, which has a limit and a price. Or you can split the work across many computers. Spark is the second option.

Spark

Picture counting a deck of cards. Alone, you count all 52. With four friends, everyone counts 13 and you add up the totals.

Spark does that with data. It cuts your file into chunks, gives each chunk to a different worker, and combines the answers. You never manage any of it. You say what you want, Spark figures out how to divide it.

Scala

Spark is written in a language called Scala, which runs on the Java Virtual Machine.

You will never write Scala. But it leaks into your life in exactly one way: Spark needs Java installed, even though you never touch it.

When I set this up, everything installed correctly and still failed instantly. No Java. That error catches a lot of people.

PySpark

PySpark lets you write Spark in Python. It is not a Python version of Spark. It is a remote control for the Scala one.

Your Python does not crunch numbers. It describes the work, sends that description to Spark, and gets answers back.

This is why Java is required, and why Spark errors are full of Java names you never wrote. It also means Python is not the slow choice. Python and Scala send identical instructions to the same engine.

The idea that confuses everyone

This function converts seven columns to proper types:

def clean_data(df):
    df2 = df.withColumn("passenger_count",df.passenger_count.cast('int'))\
    .withColumn("total_amount",df.total_amount.cast('float'))\
    .withColumn("tip_amount",df.tip_amount.cast('float'))\
    .withColumn("trip_distance",df.trip_distance.cast('float'))\
    .withColumn("fare_amount",df.fare_amount.cast('float'))\
    .withColumn("tpep_pickup_datetime",df.tpep_pickup_datetime.cast('timestamp'))\
    .withColumn("tpep_dropoff_datetime",df.tpep_dropoff_datetime.cast('timestamp'))
    return df2

Run it on 22 million rows and it finishes instantly. Because it converts nothing.

The notebook times this side by side. clean_data returns in 0.1 seconds; the .count() right after it is what actually does the work.

Spark has two kinds of operations. Ones that describe work, like withColumn and groupBy, return immediately. Ones that demand answers, like show() and count(), make everything actually run.

It is a shopping list versus going to the store. You can add fifty items in seconds because you have not left the house.

Two things follow. Spark sees your whole plan before running it, so it can optimize. And your mistake in cell 3 will not appear until cell 9, wherever the first real answer is requested.

What it found

Average taxi speed, across 22.6 million trips:

Thursday is the worst day to be in a cab. Sunday is the best. The whole spread is under 2 mph, which says something about Manhattan.

Nights cost less per mile than days, $5.66 against$6.41, because less traffic means fewer metered minutes sitting still.

Where Spark actually earns its keep

Twenty-two million rows is a demo. Here is what the real cases look like.

The data does not fit anywhere. A payment processor holds years of card transactions. A telecom logs every call and connection. A factory floor streams sensor readings from a thousand machines. These are billions of rows. There is no laptop, and no single server, that opens them.

The job has a deadline. Plenty of overnight pipelines have to be finished before the business opens. When a job that takes nine hours needs to take one, you cannot optimize your way there on one machine. You add machines.

Two big things have to be joined. Matching a year of transactions against a customer table is easy to describe and brutal to execute, because every matching row has to physically meet its partner. Spark shuffles that across a cluster. One machine just runs out of room.

The data lives in cloud storage. Most companies keep their data as files in S3 or Google Cloud Storage rather than in a database. Spark reads those directly, in parallel, in the format they are already in. That is the daily reality of most data engineering work.

The same code has to run at both sizes. This is the one people underrate. In the notebook for this post, Spark runs with local[4], meaning four threads on a laptop. Pointing it at a 200-machine cluster changes that one line. Nothing else. You prototype on a sample at your desk and run the identical code on the full thing in production.

When to skip it

If your data fits comfortably in memory, do not use Spark. You will pay for starting a JVM, planning a distributed job, and coordinating workers, and get nothing back. On a few thousand rows it is slower than doing nothing special at all.

Spark is not a faster way to handle small data. It is the thing you reach for when the data stopped being small, and the honest answer to "should I use Spark" is usually no.

Run it yourself

The notebook is here, or download it directly. It downloads the taxi data itself, so there is nothing to find or configure.

python3 -m venv ~/sparkenv
~/sparkenv/bin/python -m pip install pyspark==3.5.3 jdk4py ipykernel
~/sparkenv/bin/python -m ipykernel install --user --name pyspark-env --display-name "Python (PySpark)"

jdk4py is the useful trick there. It installs a Java runtime as a Python package, so you skip the system-wide Java install that normally wants an admin password.

Open the notebook, pick the Python (PySpark) kernel, and run all. Then try the experiments at the bottom: change local[4] to local[1] and watch the timings move, or add df.cache() and watch Spark stop re-reading from disk.