Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Databricks Certified Associate Developer for Apache Spark 3.5 – Python Question and Answers

In Spark, when a CSV row does not match the provided schema, Spark does not raise an error by default. Instead, it returns null for fields that cannot be parsed correctly.

In the first row, "hello" cannot be cast to Integer for the age field → Spark sets age=None

In the second row, "20" is a valid integer → age=20

So the output will be:

[Row(name='bambi', age=None), Row(name='alladin', age=20)]

Final Answer: C

The partitionBy() method in Spark organizes output into subdirectories based on unique combinations of the specified columns:

e.g.

/path/to/output/color=red/fruit=apple/part-0000.parquet

/path/to/output/color=green/fruit=banana/part-0001.parquet

This improves query performance via partition pruning.

It does not consolidate into a single file.

Null values are allowed in partitions.

It does not "append" unless .mode("append") is used.

broadcast(df1) tells Spark to send the small DataFrame (df1) to all worker nodes.

This eliminates the need for shuffling df1 during the join.

Broadcast joins are optimized for scenarios with one large and one small table.

Spark's best practice is to estimate partition count based on data volume and a reasonable partition size — typically 128 MB to 256 MB per partition.

With 1 TB of data: 1 TB / 128 MB ≈ ~8000 partitions

This ensures that tasks are distributed across available CPUs for parallelism and that each task processes an optimal volume of data.

Option A (equal to cores) may result in partitions that are too large.

Option B (fixed 200) is arbitrary and may underutilize the cluster.

Option C (nodes) gives too few partitions (10), limiting parallelism.

In Structured Streaming, only transformation operations are allowed on streaming DataFrames. These include select(), filter(), where(), groupBy(), withColumn(), etc.

Example of supported transformation:

filtered_df = streaming_df.filter("count < 30")

However, actions such as count(), show(), and collect() are not supported directly on streaming DataFrames because streaming queries are unbounded and never finish until stopped.

To perform aggregations, the query must be executed through writeStream and an output sink.

Why the other options are incorrect:

A: count() is an action, not allowed directly on streaming DataFrames.

C: countDistinct() is a stateful aggregation, not supported outside of a proper streaming query.

D: show() is also an action, unsupported on streaming queries.

Spark Connect introduces a decoupled client-server architecture. Its key feature is enabling Spark job submission and execution from remote clients — in Python, Java, etc.

From Databricks documentation:

“Spark Connect allows remote clients to connect to a Spark cluster and execute Spark jobs without being co-located with the Spark driver.”

A is close, but "any language" is overstated (currently supports Python, Java, etc., not literally all).

B refers to REST, which is not Spark Connect's mechanism.

D is incorrect; Spark Connect isn’t focused on ingestion.

Final Answer: C

The question requires creating a dictionary where keys are region values and values are the corresponding region_id integers. Furthermore, it asks to retrieve only the smallest 3 region_id values.

Key observations:

select('region', 'region_id') puts the column order as expected by dict() — where the first column becomes the key and the second the value.

sort('region_id') ensures sorting in ascending order so the smallest IDs are first.

take(3) retrieves exactly 3 rows.

Wrapping the result in dict(...) correctly builds the required Python dictionary: { 'AFRICA': 0, 'AMERICA': 1, 'ASIA': 2 }.

Incorrect options:

Option B flips the order to region_id first, resulting in a dictionary with integer keys — not what's asked.

Option C uses .limit(3) without sorting, which leads to non-deterministic rows based on partition layout.

Option D sorts in descending order, giving the largest rather than smallest region_ids.

Hence, Option A meets all the requirements precisely.

To remove specific columns from a PySpark DataFrame, the drop() method is used. This method returns a new DataFrame without the specified columns. The correct syntax for dropping multiple columns is to pass each column name as a separate argument to the drop() method.

Correct Usage:

df_user_non_pii = df_user.drop("first_name", "last_name", "email", "birthdate")

This line of code will return a new DataFrame df_user_non_pii that excludes the specified PII columns.

Explanation of Options:

A. Correct. Uses the drop() method with multiple column names passed as separate arguments, which is the standard and correct usage in PySpark.

B. Although it appears similar to Option A, if the column names are not enclosed in quotes or if there's a syntax error (e.g., missing quotes or incorrect variable names), it would result in an error. However, as written, it's identical to Option A and thus also correct.

C. Incorrect. The dropfields() method is not a method of the DataFrame class in PySpark. It's used with StructType columns to drop fields from nested structures, not top-level DataFrame columns.

D. Incorrect. Passing a single string with comma-separated column names to dropfields() is not valid syntax in PySpark.

A broadcast join is a type of join where the smaller DataFrame is replicated (broadcast) to all worker nodes in the cluster. This avoids shuffling the large DataFrame across the network.

Benefits:

Eliminates shuffle for the smaller dataset.

Greatly improves performance when one side of the join is small enough to fit in memory.

Correct usage example:

df_result = df2.join(broadcast(df1), "id")

This is a map-side join, where each executor joins its local partition of the large dataset with the broadcasted copy of the small one.

Why the other options are incorrect:

A: Broadcasting does not change partition sizes.

B: Joins always match on key equality; this is not specific to broadcast joins.

D: Broadcasting does not filter; it distributes data for faster joins.

Adaptive Query Execution (AQE) is a Spark 3.x feature that dynamically optimizes query plans at runtime. One of its core features is:

Dynamically switching join strategies (e.g., from sort-merge to broadcast) based on runtime statistics.

Other AQE capabilities include:

Coalescing shuffle partitions

Skew join handling

Option A is correct.

Option B refers to statistics collection, which is not AQE's primary function.

Option C is too broad and not AQE-specific.

Option D refers to Delta Lake optimizations, unrelated to AQE.

Final Answer: A

To sort a Spark DataFrame by multiple columns, use .orderBy() (or .sort()) with column expressions.

Correct syntax for descending and ascending mix:

from pyspark.sql.functions import col

df1.orderBy(col("count").desc(), col("Name").asc())

This sorts primarily by count in descending order and secondarily by Name in ascending order (alphabetically).

Why the other options are incorrect:

B/C: Default sort order is ascending; won’t place highest counts first.

D: Reverses sorting logic — sorts Name descending, not required.

Spark Connect enables decoupling of the client and Spark driver processes, allowing remote access. Spark supports configuring the remote Spark Connect server in multiple ways:

From Databricks and Spark documentation:

Option B (--remote "sc://localhost") is a valid command-line argument for the pyspark shell to connect using Spark Connect.

Option C (setting SPARK_REMOTE environment variable) is also a supported method to configure the remote endpoint.

Option A is incorrect because Spark Connect uses the sc:// protocol, not https:// .

Option D requires modifying the code, which the question explicitly avoids.

Option E configures the port on the server side but doesn’t start a client connection.

Final Answers: B and C

To enable a Structured Streaming query to recover from failures or intentional shutdowns, it is essential to specify the checkpointLocation option during the writeStream operation. This checkpoint location stores the progress information of the streaming query, allowing it to resume from where it left off.

According to the Databricks documentation:

"You must specify the checkpointLocation option before you run a streaming query, as in the following example:

option("checkpointLocation", "/path/to/checkpoint/dir")

toTable("catalog.schema.table")

— Databricks Documentation: Structured Streaming checkpoints

By setting the checkpointLocation during writeStream, Spark can maintain state information and ensure exactly-once processing semantics, which are crucial for reliable streaming applications.

Apache Spark was designed for big data and high-velocity workloads. Its core strength lies in its in-memory computation and parallel distributed processing model.

These features allow Spark to:

Process large-scale datasets quickly across many nodes.

Support real-time and near–real-time analytics for tasks like fraud detection and recommendations.

Minimize disk I/O through caching and memory persistence.

Thus, the key advantage in this use case is Spark’s ability to handle large data volumes efficiently using distributed, in-memory computation.

Why the other options are incorrect:

A: Spark is optimized for large, not small, datasets.

C: SQL support is useful but doesn’t solve the scalability issue.

D: MLlib supports machine learning but relies on Spark’s parallel computation for speed.

The correct output mode for streaming aggregations that need to output the full updated results at each trigger is "complete".

From the official documentation:

“complete: The entire updated result table will be output to the sink every time there is a trigger.”

This is ideal for aggregations, such as counts or averages grouped by a key, where the result table changes incrementally over time.

append: only outputs newly added rows

replace and aggregate: invalid values for output mode

According to the PySpark API documentation:

"getOrCreate(): Gets an existing SparkSession or, if there is no existing one, creates a new one based on the options set in this builder."

This means Spark maintains a global singleton session within a JVM process. Repeated calls to getOrCreate() return the same session, unless explicitly stopped.

Option A is incorrect: the method does not destroy any session.

Option B incorrectly ties uniqueness to appName, which does not influence session reusability.

Option D is incorrect: it contradicts the fundamental behavior of getOrCreate().

(Source: PySpark SparkSession API Docs)

When queries frequently filter on certain columns, partitioning by those columns ensures partition pruning, allowing Spark to scan only relevant directories instead of the entire dataset.

Correct code:

final.write.partitionBy("event_year", "event_month").parquet("events.liveLatest")

This improves read performance dramatically for filters like:

SELECT * FROM events.liveLatest WHERE event_year = 2024 AND event_month = 5;

bucketBy() helps in clustering and joins, not in partition pruning for file-based tables.

Why the other options are incorrect:

B: Bucket count changes parallelism, not query pruning.

C: sortBy organizes data within files, not across partitions.

D: Partitioning by only one column limits pruning benefits.

Which operation is supported with streaming_df?

A. streaming_df.select(countDistinct("Name"))

B. streaming_df.groupby("Id").count()

C. streaming_df.orderBy("timestamp").limit(4)

D. streaming_df.filter(col("count") < 30).show()

Answer: B

In Structured Streaming, only a limited subset of operations is supported due to the nature of unbounded data. Operations like sorting (orderBy) and global aggregation (countDistinct) require a full view of the dataset, which is not possible with streaming data unless specific watermarks or windows are defined.

Review of Each Option:

A. select(countDistinct("Name"))

Not allowed — Global aggregation like countDistinct() requires the full dataset and is not supported directly in streaming without watermark and windowing logic.

In Spark’s distributed architecture:

The Driver Node coordinates the execution of a Spark application. It converts the logical plan into a physical plan of stages and tasks.

The Executors, running on Worker Nodes, are responsible for executing tasks assigned by the driver and storing data (in memory or disk) during execution.

Key point:

Executors are the active agents that perform the actual computations on data partitions. Each executor runs multiple tasks in parallel using available CPU cores.

Why the other options are incorrect:

A (Driver Nodes): The driver schedules tasks; it doesn’t execute them.

C (CPU Cores): CPU cores execute within executors, but they are hardware, not Spark architectural components.

D (Worker Nodes): Worker nodes host executors but do not directly execute tasks; executors do.

References (Databricks Apache Spark 3.5 – Python / Study Guide):

Spark Architecture Components — Driver, Executors, Cluster Manager, Worker Nodes.

Databricks Exam Guide (June 2025): Section “Apache Spark Architecture and Components” — describes the roles of driver and executor nodes in distributed processing.

Adaptive Query Execution (AQE) is a powerful optimization framework introduced in Apache Spark 3.0 and enabled by default since Spark 3.2. It dynamically adjusts query execution plans based on runtime statistics, leading to significant performance improvements. The key benefits of AQE include:

Dynamic Join Strategy Selection: AQE can switch join strategies at runtime. For instance, it can convert a sort-merge join to a broadcast hash join if it detects that one side of the join is small enough to be broadcasted, thus optimizing the join operation .

Handling Skewed Data: AQE detects skewed partitions during join operations and splits them into smaller partitions. This approach balances the workload across tasks, preventing scenarios where certain tasks take significantly longer due to data skew .

Coalescing Post-Shuffle Partitions: AQE dynamically coalesces small shuffle partitions into larger ones based on the actual data size, reducing the overhead of managing numerous small tasks and improving overall query performance .

These runtime optimizations allow Spark to adapt to the actual data characteristics during query execution, leading to more efficient resource utilization and faster query processing times.

When reading CSV files using Spark SQL or the DataFrame API, Spark by default assumes that the first line of the file is data, not headers. To interpret the first line as column names, the header option must be set to true.

Correct syntax:

CREATE TABLE locations

USING csv

OPTIONS (

path '/data/locations.csv',

header 'true'

);

This tells Spark to read the first row as column headers and correctly map columns like city, lat, and long.

Why the other options are incorrect:

B (header 'false'): Default behavior; would keep reading header as data.

C / D (sep): Used to specify the delimiter; not relevant unless the file uses a different separator (e.g., |).

References (Databricks Apache Spark 3.5 – Python / Study Guide):

PySpark SQL Data Sources — CSV options (header, inferSchema, sep).

Databricks Exam Guide (June 2025): Section “Using Spark SQL” — Reading data from files with different formats using Spark SQL and DataFrame APIs.

===========

Spark Connect enables remote execution of Spark jobs by decoupling the client from the driver using the Spark Connect protocol (gRPC).

It allows users to run Spark code from different environments (like notebooks, IDEs, or remote clients) while executing jobs on the cluster.

Key Features:

Enables remote interaction between client and Spark driver.

Supports interactive development and lightweight client sessions.

Improves developer productivity without needing driver resources locally.

Why the other options are incorrect:

A: Spark Connect is not limited to ingestion tasks.

B: It allows multi-language clients (Python, Scala, etc.) but runs via Spark Connect API, not arbitrary remote code.

C: Uses gRPC protocol, not REST.

Comprehensive Explanation:

To cover structured data processing, SQL querying, and machine learning in Apache Spark, the correct combination of components is:

Spark DataFrames: for structured data processing

Spark SQL: to execute SQL queries over structured data

MLlib: Spark's scalable machine learning library

This trio is designed for exactly this type of use case.

Why other options are incorrect:

A: GraphX is for graph processing — not needed here.

B: Pandas API on Spark is useful, but MLlib is essential for ML, which this option omits.

C: Spark Streaming is legacy; GraphX is irrelevant here.

The approx_percentile function in Spark is a performance-optimized alternative to percentile. It takes an optional accuracy parameter:

approx_percentile(column, percentage, accuracy)

Higher accuracy values → more precise results, but increased memory/computation.

Lower values → faster but less accurate.

From the documentation:

“Increasing the accuracy improves precision but increases memory usage.”

Final Answer: D

In Spark, the default join type is an inner join, which returns only the rows with matching keys in both DataFrames. To retain all records from the left DataFrame (purch_df) and include matching records from the right DataFrame (cust_df), a left outer join should be used.

By specifying the join type as 'left', the modified code ensures that all records from purch_df are preserved, and matching records from cust_df are included. Records in purch_df without a corresponding match in cust_df will have null values for the columns from cust_df.

This approach is consistent with standard SQL join operations and is supported in PySpark's DataFrame API.

When writing a DataFrame using .partitionBy() in Spark, the data is physically organized into directory structures corresponding to unique combinations of the partition columns.

Example:

/path/to/output/color=Red/taste=Sweet/part-0001.parquet

/path/to/output/color=Green/taste=Sour/part-0002.parquet

This structure improves query performance by pruning partitions when filtering on these columns.

Why the other options are incorrect:

A: Appending requires .mode("append"), which isn’t used here.

B: Null values in partition columns are handled; they don’t raise errors.

D: Partitioning prevents storing all data in a single file.

To create a Python dictionary from a Spark DataFrame, you can first collect the data to the driver node and then convert it into a Python dictionary using dict().

Steps:

Select only relevant columns.

Order by region_id to get the smallest ones.

Limit to 3 rows.

Map each row into key–value pairs.

Collect results to the driver and convert to a dictionary.

Correct code:

regions_dict = dict(

regions.orderBy("region_id")

limit(3)

rdd.map(lambda x: (x.region_id, x.region_name))

collect()

)

This produces a dictionary like:

{10: 'North', 12: 'East', 14: 'West'}

Why the other options are incorrect:

A/B: take(3) returns a list of Row objects, not key–value pairs.

C: Doesn’t order or limit by smallest IDs, so the result may not be correct.

coalesce(n) reduces the number of partitions without triggering a full shuffle, unlike repartition().

This is ideal when reducing partition count, especially during write operations.

When running in local mode (e.g., local[4]), the number inside the brackets defines how many threads Spark will use.

Using local[*] ensures Spark uses all available CPU cores for parallelism.

Example:

spark-submit --master local[*]

Dynamic allocation and executor memory apply to cluster-based deployments, not local mode.

Pandas API on Spark provides a distributed implementation of the Pandas DataFrame API on top of Apache Spark.

Advantages:

Executes transformations in parallel across all nodes and cores in the cluster.

Maintains Pandas-like syntax, making it easy for Python users to transition.

Enables scaling of existing Pandas code to handle large datasets without memory limits.

Therefore, it combines Pandas usability with Spark’s distributed power, offering both speed and scalability.

Why the other options are incorrect:

B: While it uses Python, that’s not its main advantage.

C: It runs distributed across the cluster, not on a single node.

D: Pandas API on Spark uses lazy evaluation, not eager computation.

The correct approach to perform a parallelized groupBy operation across Spark worker nodes using Pandas API is via applyInPandas. This function enables grouped map operations using Pandas logic in a distributed Spark environment. It applies a user-defined function to each group of data represented as a Pandas DataFrame.

As per the Databricks documentation:

"applyInPandas() allows for vectorized operations on grouped data in Spark. It applies a user-defined function to each group of a DataFrame and outputs a new DataFrame. This is the recommended approach for using Pandas logic across grouped data with parallel execution."

Option A is correct and achieves this parallel execution.

Option B (mapInPandas) applies to the entire DataFrame, not grouped operations.

Option C uses built-in aggregation functions, which are efficient but not customizable with Pandas logic.

Option D creates a scalar Pandas UDF which does not perform a group-wise transformation.

Therefore, to run a groupBy with parallel Pandas logic on Spark workers, Option A using applyInPandas is the only correct answer.

To use a Python function as a UDF (User Defined Function) in Spark SQL, it must first be registered using spark.udf.register().

Correct usage:

spark.udf.register("cube_func", cube_func)

num_df.selectExpr("cube_func(num)").show()

This registers cube_func as a callable SQL function available in expressions or queries.

Why the other options are incorrect:

B: You must wrap with udf() or selectExpr; calling plain Python functions won’t work.

C: createDataFrame is for building DataFrames, not calling UDFs.

D: DataFrames cannot directly register UDFs.

Temp views like createOrReplaceTempView are session-scoped.

They disappear once the Spark session ends.

To retain data across sessions, it must be persisted:

df.write.saveAsTable("users_vw")

Thus, the view needs to be persisted as a table to survive session termination.

To handle JSON files with evolving or differing schemas, Spark SQL supports the option mergeSchema 'true', which merges all fields across files into a unified schema.

Correct syntax:

CREATE EXTERNAL TABLE users

USING json

OPTIONS (path '/data/input.json', mergeSchema 'true');

This creates an external table directly on the JSON data, inferring schema automatically and merging variations.

Why the other options are incorrect:

B: Missing schema merge configuration — fails with inconsistent files.

C: inferSchema applies to CSV/other file types, not JSON.

D: mergeAll is not a valid Spark SQL option.

In Apache Spark’s execution hierarchy, the relationships are structured as follows:

Job: Created when an action (e.g., count(), collect(), save()) is triggered on an RDD or DataFrame.

Stage: Each job is divided into one or more stages, separated by shuffle boundaries (e.g., after a reduceByKey or join).

Task: Each stage consists of multiple tasks, one per partition, executed in parallel on executors.

Execution Hierarchy:

Job → Stage(s) → Task(s)

So, a job contains multiple stages, and each stage contains multiple tasks.

Why the other options are incorrect:

A: A job does not directly contain tasks without stages.

B: A stage cannot contain multiple jobs; it belongs to a single job.

C: Tasks do not contain jobs.

References (Databricks Apache Spark 3.5 – Python / Study Guide):

Spark Architecture Overview — Execution Hierarchy: Jobs, Stages, and Tasks.

Databricks Exam Guide (June 2025): Section “Apache Spark Architecture and Components” — describes execution hierarchy and lazy evaluation.

===========

The .coalesce(numPartitions) function is used to reduce the number of partitions in a DataFrame. It does not increase the number of partitions. If the specified number of partitions is greater than the current number, it will not have any effect.

From the official Spark documentation:

“coalesce() results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of the 100 new partitions will claim one or more of the current partitions.”

However, if you try to increase partitions using coalesce (e.g., from 10 to 20), the number of partitions remains unchanged.

Hence, df.coalesce(20) will still return a DataFrame with 10 partitions.

The groupBy() operation causes a shuffle because it requires all values for a specific key to be brought together, which may involve moving data across partitions.

In contrast:

filter() and select() are narrow transformations and do not cause shuffles.

coalesce() tries to reduce the number of partitions and avoids shuffling by moving data to fewer partitions without a full shuffle (unlike repartition()).