PySpark Interview Questions

You can write working PySpark and still stumble here, because these questions ask what the engine does rather than what your code says. Interviewers open with them precisely because they separate people who have tuned a job from people who have only run one. This section covers the model questions and what a full answer includes.

• Transformation or action, what is the difference? Transformations build a plan and return a new DataFrame lazily; actions such as count, collect or a write trigger execution. Nothing computes until an action asks for a result.
• Why is laziness useful? The optimizer sees the whole chain before running it, so it can reorder filters, prune columns and combine steps.
• Narrow or wide transformation? Narrow operations such as filter and select keep each output partition dependent on one input partition. Wide operations such as groupBy, join and distinct redistribute data across partitions, which is a shuffle.
• What is a shuffle and why does it matter? Data moves across the network and hits disk, forming a stage boundary. It is usually the most expensive thing a job does.
• Explain job, stage and task. An action starts a job, shuffle boundaries split it into stages, and each stage runs one task per partition.
• RDD, DataFrame or Dataset? Prefer DataFrames, because the Catalyst optimizer and columnar execution apply. RDDs remain for low-level control. Typed Datasets are a JVM concept, so in Python the honest answer is that they do not apply.

Say the words shuffle and stage when they belong in the answer. Interviewers use them as a shortcut for whether you have read a Spark UI.

Most senior PySpark interviews are performance interviews. The questions arrive as scenarios rather than definitions.

• A join is slow. What do you check? The size of each side first. If one fits in executor memory, broadcast it and skip the shuffle entirely. Otherwise look at partitioning and skew before touching cluster size.
• What is data skew and how do you fix it? A few keys hold most of the rows, so one task runs long after the rest finish. Remedies include salting the hot key, broadcasting the small side, or filtering nulls that all hash together. The diagnostic signal is the task duration spread in the Spark UI.
• When do you cache or persist? When a DataFrame is reused across multiple actions and recomputation would be expensive. Caching something used once wastes memory, and unpersisting matters in long jobs.
• Repartition or coalesce? Repartition shuffles and can increase or decrease partitions evenly; coalesce merges without a full shuffle, which is the cheaper way to reduce output files.
• Why avoid a Python UDF? Rows serialize between the JVM and a Python process, and the optimizer cannot see inside the function. Prefer built-in functions, and reach for a vectorized UDF only when no built-in exists.
• Why is collect dangerous? It pulls the full result to the driver and can exhaust its memory.
• A job fails with out of memory. What is your order of investigation? Whether it is driver or executor, then skew, then partition sizing, then the memory configuration. Raising memory first is the answer that signals inexperience.

A data engineer interviewing for a platform team was asked why a nightly job that had run for a year suddenly took four hours. The answer that landed was not a configuration change, it was that one upstream partner started sending nulls in the join key, so every null row hashed to one partition. Interviewers reward that order: look at the data before the cluster.

Related banks by role sit under interview questions by role.

The remaining questions check whether you have shipped a pipeline rather than finished a tutorial.

• How do you read data efficiently? Columnar formats such as Parquet, partition pruning on the filter column, and predicate pushdown. Explain why reading fewer bytes beats optimizing what happens afterwards.
• Why define a schema instead of inferring one? Inference costs an extra pass over the data and can guess types inconsistently between runs.
• How do you handle nulls and duplicates? The relevant functions, plus the point that null-heavy join keys create skew.
• What are window functions used for? Ranking, running totals and deduplicating to the latest record per key, which is a very common pipeline task.
• How do you write output without producing thousands of small files? Coalesce or repartition before writing, and partition the output by a column with sensible cardinality.
• How do you test PySpark code? Small local sessions with fixture DataFrames, and business logic factored into functions that take and return DataFrames.
• How do you submit and configure a job? Executor count, cores and memory, and the reasoning that too many small executors and too few large ones both waste capacity.

PySpark answers fail out loud in a recognizable way. The candidate knows that a shuffle is expensive but cannot say which operations cause one, so the answer becomes a list of adjectives. Reading a question bank produces recognition, and recognition is not the same as an explanation delivered while someone waits.

Take one pipeline you have built and narrate it end to end: the read, every transformation, where the stage boundaries fall, and what you would check first if it slowed down. Do it aloud until you stop restarting. Running these prompts against an AI interviewer that asks the follow-up is closer to a real round than rereading notes, and that is what mock interview mode is built for.

Aaron Cao, founder of SubcueAI, built practice around that speaking gap rather than around supplying more questions. In a live interview the desktop app and the browser extension Side Panel can surface structure as the interviewer speaks, which helps most on material you have already rehearsed. Setup takes a few minutes and is covered on the tutorial page.