1. What is PySpark?
PySpark is the Python API for Apache Spark, a powerful open-source distributed computing system. It allows Python developers to utilize Spark's capabilities for big data processing, analytics, and machine learning.
2. What are the advantages of using PySpark over traditional Python for big data processing?
PySpark leverages Spark's distributed computing capabilities, enabling efficient processing of large-scale data, while providing high-level APIs in Python for easier integration with existing Python libraries and workflows.
3. Explain the concept of RDD (Resilient Distributed Dataset) in PySpark.
RDD (Resilient Distributed Dataset) in PySpark represents an immutable, fault-tolerant distributed collection of elements that can be operated on in parallel.
4. What are the different ways to create RDDs in PySpark?
RDDs in PySpark can be created from existing Python collections, by parallelizing existing data, by loading data from external storage systems, or by transforming existing RDDs through various operations.
5. What is the difference between transformation and action in PySpark?
Transformations in PySpark are lazy operations that define the structure of a new RDD without computing it immediately, while actions trigger the execution of transformations and return results to the driver program.
6. How does PySpark handle fault tolerance?
PySpark achieves fault tolerance through RDD lineage, which records the transformations applied to the base RDD to recreate lost data in case of failure.
7. Explain the concept of DataFrame in PySpark.
DataFrame in PySpark is a distributed collection of data organized into named columns, akin to a table in a relational database, providing a structured and efficient way to work with data.
8. What are the different ways to create DataFrames in PySpark?
DataFrames in PySpark can be created from existing RDDs, structured data formats (e.g., CSV, JSON, Parquet), relational databases using JDBC, or by applying schema to RDDs or Python dictionaries.
9. What is the significance of SparkSession in PySpark?
SparkSession in PySpark serves as the entry point to Spark functionality, providing a unified interface for working with various Spark features such as creating DataFrames, executing SQL queries, and managing resources.
10. How can you optimize PySpark jobs for performance?
You can optimize PySpark jobs for performance by tuning Spark configuration settings, caching intermediate results, minimizing shuffling operations, using appropriate partitioning strategies, and leveraging data locality.
