Category: Spark

  • Spark data abstractions

    In Spark, RDD, DataFrames, and Datasets are three different abstractions for working with distributed data. RDD is the fundamental data structure in Spark and stands for Resilient Distributed Dataset. RDDs are fault-tolerant, immutable distributed collections of objects. They can be created from data stored in HDFS, local file systems, or other data sources. RDDs provide…