Which Spark DataFrame writer option controls how data is distributed into folders when saving?

Which Spark DataFrame writer option controls how data is distributed into folders when saving?

• A. splitBy
• B. distributeBy
• C. partitionBy
• D. shuffleBy

Answer: (C)

Partitioning the output by specific columns is what determines how data ends up organized into folders on disk. When you write a DataFrame and specify partitionBy("col1", "col2"), Spark writes the data into a directory structure where each folder corresponds to a unique combination of the partition column values (for example, col1=valueA/col2=valueB). This layout makes queries that filter on those columns much more efficient because Spark can skip entire folders that don’t match the filter. Other terms listed don’t apply to controlling the folder layout of the saved data in Spark’s DataFrameWriter. They aren’t used to define how the output is partitioned on disk, and some refer to different concepts like reshuffling data during processing rather than how the final files are organized.

Partitioning the output by specific columns is what determines how data ends up organized into folders on disk. When you write a DataFrame and specify partitionBy("col1", "col2"), Spark writes the data into a directory structure where each folder corresponds to a unique combination of the partition column values (for example, col1=valueA/col2=valueB). This layout makes queries that filter on those columns much more efficient because Spark can skip entire folders that don’t match the filter.

Other terms listed don’t apply to controlling the folder layout of the saved data in Spark’s DataFrameWriter. They aren’t used to define how the output is partitioned on disk, and some refer to different concepts like reshuffling data during processing rather than how the final files are organized.