DATA 603 – Big Data Platforms Homework #9 – Spark SQL _ Explain how SQL is applied to a typical RDD?

downloaded from

Scholarfriends

https://scholarfriends.com Downloaded by: writerforacademic@gmail.com | https://scholarfriends.com/singlePaper/313996/data-603-big-data-platforms-homework-9spark-sql-explain-how-sql-is-applied-to-a-typical Distribution of this document is illegal.

UMBC – CSEE Department Data Science Program Fall 2021

Dr. Waleed Youssef youssef1@umbc.edu

DATA 603 – Big Data Platforms Homework #9 – Spark SQL

(1) [10 Points] Explain how SQL is applied to a typical RDD? What components are needed to perform this task?

Ans) Spark SQL integrates a processing like the processing of relational databases with Spark’s functional programming. It provides support for various data sources and makes it possible to weave SQL queries with code transformations thus resulting in a very powerful tool and hence blurs the gap between RDD and relational table. It also provides higher optimization.

Spark SQL transforms RDDs into SQL using a special type of RDD called SchemaRDD. It is essentially a RDD with schema. As it contains schema, run relation queries can be run on the data along with basic RDD functions. The SchemaRDD can be registered as a table so that SQL queries can be executed on it using Spark SQL.A schemaRDD is made up of Object data which refers to the data stored in RDD and schema which describes the data types of the objects.

Spark SQL supports two different methods for converting existing RDDs into SchemaRDDs. The first method uses reflection to infer the schema of an RDD that contains specific types of objects. This reflection-based approach leads to more concise code and works well when the schema is known at the time of creating the Spark application. For example, in case of the application being written in Python environment Spark SQL can convert an RDD of Row objects to a SchemaRDD, inferring the datatypes. Rows are constructed by passing a list of key/value pairs as kwargs to the Row class. The keys of this list define the column names of the table, and the types are inferred by looking at the first row.

The second method for creating SchemaRDDs is through a programmatic interface that allows you to construct a schema and then apply it to an existing RDD. While this method is more verbose, it allows you to construct SchemaRDDs when the columns and their types are not known until runtime. In case of Python environment, a SchemaRDD can be created programmatically with three steps.

Create an RDD of tuples or lists from the original RDD.

Create the schema represented by a Struct Type matching the structure of tuples or lists in the RDD created in the step 1.

Apply the schema to the RDD via apply Schema method provided by SQLContext.

SchemaRDD can also be created using the already existing RDD and once SchemaRDD is created it can be registered as a table SQL query can be run against it.

When Spark is run in combination with RDD then the disadvantage is that functional transformations on partitioned collections are opaque or are not clearly expressed objects. But if we use SQL in combination with SchemaRDDs Declarative transformations can be done on partitioned collections of tuples or Rows.

References: 1)Blackboard slides 2) (2) [10 Points] Describe how Spark SQL supports Hive and Mongo DB? How do you connect to these data sources?

Hive and Spark SQL: Spark SQL supports Apache Hive using HiveContext, which in turn uses Spark SQL execution engine to work(read and write operations) with data stored in Hive. However, since Hive has a large number of dependencies, these dependencies are not included in the default Spark distribution. If Hive dependencies can be found on the class path, Spark will load them automatically. It is important to keep in mind that the Hive dependencies must also be present on all of the worker nodes, as they will need access to the Hive serialization and deserialization libraries (SerDes) in order to access data stored in Hive.

When working with Hive the SparkSession needs to be instantiated with Hive support, including connectivity to a persistent Hive metastore, support for Hive Serdes, and Hive user-defined functions. If there is no existing Hive deployment end users can still enable Hive support. When not configured by the hive-site.xml, the context automatically creates metastore_db in the current directory and creates a directory configured by spark.sql.warehouse.dir, which defaults to the directory spark-warehouse in the current directory that the Spark application is started.

Configuration of Hive is done by placing your hive-site.xml, core-site.xml (for security configuration), and hdfs-site.xml (for HDFS configuration) file in conf/.

One of the most important pieces of Spark SQL’s Hive support is interaction with Hive metastore, which enables Spark SQL to access metadata of Hive tables.Spark SQL can be used to query different versions of Hive metastores, using the configurations like spark.sql.hive.metastore.version, spark.sql.hive.metastore.sharedPrefixes etc .Internally Spark SQL will compile against built-in Hive and use those classes for internal execution independent of the version of Hive that is being used to talk to the metastore.

Spark and MongoDB: Spark data frames can easily be created from MongoDb collections through spark connector with mangodb. While creating a spark session spark.mongodb.input.uri and spark.mongodb.output.uri configuration options is specified, the created spark session object will use the pointed mongodb. In case of PySpark, when Pyspark is started, a SparkSession object called Spark is created by default. For MongoDB, if the spark.mongodb.input.uri and spark.mongodb.output.uri configuration options is specified when Pyspark is started, then the default SparkSession object uses them.

The most common architectural patterns for using MongoDB with Spark SQL:  



MongoDB as application DB - the data resides in MongoDB as an operational system and is processed by Spark SQL on-demand via a DataFrame MongoDB as storage in data lake - for data that you want to analyze with Spark and especially over which you want to query for slices of your data (where indexes are enormously helpful), you would store datasets of your data lake in MongoDB and run Spark on those slices. You still might store other datasets on HDFS in which scanning data is sufficient.

Results operationalized in MongoDB - after you run analytics in Spark, you often want the results to be distributed to end users via reporting or operational applications. With ad hoc indexing, dynamic schema, and easy scaling, MongoDB is the best location to operationalize your analysis from Spark.

Example code:

package com.mongodb object GettingStarted { def main(args: Array[String]): Unit = { /* Create the SparkSession. * If config arguments are passed from the command line using --conf, * parse args for the values to set. */ import org.apache.spark.sql.SparkSession val spark = SparkSession.builder() .master("local") .appName("MongoSparkConnectorIntro") .config("spark.mongodb.input.uri", "mongodb://127.0.0.1/test.myCollection") .config("spark.mongodb.output.uri", "mongodb://127.0.0.1/test.myCollection") .getOrCreate() } }

(source: https://docs.mongodb.com/spark-connector/current/scala-api/) References: 1)Blackboard slides 2) https://docs.mongodb.com/spark-connector/current/scala-api/ 3) https://medium.com/olivers-tech-blog/running-spark-sql-applications-with-a-remote-hivecluster-fac753feed81 (3) [10 Points] Why do we need file formats like Parquet? How is data stored in Parquet?

Ans) Parquet is an open source file format built to handle flat columnar storage data formats.

Parquet operates well with complex data in large volumes. It is known for its both performant data compression and its ability to handle a wide variety of encoding types. Parquet is widely used in the Hadoop world for analytics workloads by many query engines like Hive,Impala and Spark SQL etc.

The first feature is the columnar storage nature of the format. This simply means that data is encoded and stored by columns instead of by rows. This pattern allows for analytical queries to select a subset of columns for all rows. Parquet stores columns as chunks and can further split files within each chunk too. This allows restricting the disk input and output operations to a minimum.

The second feature to mention is data schema and types. Parquet is a binary format and allows encoded data types. Unlike some formats, it is possible to store data with a specific type of boolean, numeric( int32, int64, int96, float, double) and byte array. This allows clients to easily and efficiently serialize and reserialize the data when reading and writing to parquet format.

In addition to the data types, Parquet specification also stores metadata which records the schema at three levels: file, chunk(column) and page header. The footer for each file contains the file metadata.

The data is written first, then the metadata can be accurately written knowing all the locations, size and encoding of the written data. Many formats write their metadata in the header.

However, this requires multiple passes as data is written after the header. Parquet makes this efficient to read metadata and the data itself. Another advantage is that the file is splitable in any desirable size.

While CSV is simple and the most widely used data format (Excel, Google Sheets), there are several distinct advantages for Parquet, including: One of the unique features of Parquet is that it can store data with nested structures in a columnar fashion too. This means that in a Parquet file format, even the nested fields can be read individually without reading all the fields in the nested structure. Parquet format uses the record shredding and assembly algorithm for storing nested structures in a columnar fashion.

Parquet is column oriented, and CSV is row oriented. Row-oriented formats are optimized for OLTP workloads while column-oriented formats are better suited for analytical workloads.

Column-oriented databases such as AWS Redshift Spectrum bill by the amount data scanned per query Therefore, converting CSV to Parquet with partitioning and compression lowers overall costs and improves performance Parquet has helped its users reduce storage requirements by at least one-third on large datasets, in addition, it greatly improves scan and deserialization time, hence the overall costs.

References: References: 1)Blackboard

2) https://www.snowflake.com/guides/what-parquet 3) https://bigdataanalyticsnews.com/5-reasons-to-choose-parquet-for-spark-sql/ (4) [10 Points] Describe DataFrames? How it is used in Spark? Explain its benefits in ML? DataFrame supports reading data from the most popular formats, including JSON files, Parquet files, Hive tables. It can read from local file systems, distributed file systems (HDFS), cloud storage (S3), and external relational database systems via JDBC. In addition, through Spark SQL’s external data sources API, DataFrames can be extended to support any third-party data formats or sources. Existing third-party extensions already include Avro, CSV, ElasticSearch, and Cassandra.

Like RDDs, DataFrames are evaluated lazily. That is to say, computation only happens when an action is required. This allows their executions to be optimized, by applying techniques such as predicate pushdowns and bytecode generation. All DataFrame operations are also automatically parallelized and distributed on clusters. DataFrames’ support for data sources enables applications to easily combine data from disparate sources. But,unlike RDDs, DataFrames usually keep track of their schema and support various relational operations that lead to a more optimized execution.

RDDs cannot modify the system in such a way that it runs more efficiently. RDDs do not allow us to debug errors during the runtime. They store the data as a collection of Java objects.RDDs use serialization and garbage collection techniques. This increases the overhead on the memory of the system as they are very lengthy. This was when DataFrames were introduced to overcome the limitations Spark RDDs had.

Features of DataFrames Some of the unique features of DataFrames are: Input Optimization Engine: DataFrames make use of the input optimization engines, e.g., Catalyst Optimizer, to process data efficiently. We can use the same engine for all Python, Java, Scala, and R DataFrame APIs. Handling of Structured Data: DataFrames provide a schematic view of data. Here, the data has some meaning to it when it is being stored.

Custom Memory Management: In RDDs, the data is stored in memory, whereas DataFrames store data off-heap (outside the main Java Heap space, but still inside RAM), which in turn reduces the garbage collection overload.

Flexibility: DataFrames, like RDDs, can support various formats of data, such as CSV, Cassandra, etc.

Scalability: DataFrames can be integrated with various other Big Data tools, and they allow processing megabytes to petabytes of data at once.

Spark DataFrames are used in Advanced Analytics and Machine Learning.DataFrames could be used directly in MLlib’s ML pipeline API. Added to that, several different programs can run complex user functions on DataFrames. These advanced analytics tasks could be specified using ML Pipeline API in MLlib.

Most common advanced analytics tasks can be specified using the new pipeline API in MLlib.

For example, below is the code to create a simple text classification pipeline consisting of a tokenizer, a hashing term frequency feature extractor, and logistic regression. tokenizer = Tokenizer(inputCol="text", outputCol="words") hashingTF = HashingTF(inputCol="words", outputCol="features") lr = LogisticRegression(maxIter=10, regParam=0.01) pipeline = Pipeline(stages=[tokenizer, hashingTF, lr]) Once the pipeline is setup, we can use it to train on a DataFrame directly: df = context.load("/path/to/data") model = pipeline.fit(df) For more complicated tasks beyond what the machine learning pipeline API provides, applications can also apply arbitrarily complex functions on a DataFrame, which can also be manipulated using Spark’s existing RDD API. Unlike the eagerly evaluated data frames in R and Python, DataFrames in Spark have their execution automatically optimized by a query optimizer. Before any computation on a DataFrame starts, the Catalyst optimizer compiles the operations that were used to build the DataFrame into a physical plan for execution. Because the optimizer understands the semantics of operations and structure of the data, it can make intelligent decisions to speed up computation.

References: 1)Blackboard slides 2) https://phoenixnap.com/kb/spark-dataframe 3) https://www.datacamp.com/community/tutorials/apache-spark-tutorial-machine-learning (5) [20 Points] Spark SQL: Write a Spark program to load students’ data into a DataFrame (Student ID, Student Name, Student Phone Number, GPA) Data file: 543, John Smith, 301-304-4044,3.3,325, Nancy Thomas, 240-340-3444, 3.95, etc.

Query the data to categorize students by grade (A, B, C, D, …). Output results should be something like this:

Student ID 543 325 …..

Student Name John Smith Nancy Thomas

Phone Number 301-304-4044 240-340-3444

Grade B A

Hints    

Grade A is for GPA 3.6 or higher, Grade B for GPA 3.2 to less than 3.6, Grade C for GPA 2.8 to less than 3.2.

Enter at least 10 records in your data file, so your technique can be illustrated correctly.

Focus on visualization and presentation of your results.

You may use <<DataFrame>>.withColumn if needed

1)connecting to spark environment.

2)Initializing the data.

3)converting into a data frame

Convering the GPA to letter grades.

Visualization:

Data603_Assignme nt_9.ipynb

References: 1)https://sparkbyexamples.com/spark/spark-case-when-otherwise-example/ 2) https://www.analyticsvidhya.com/blog/2020/11/a-must-read-guide-on-howto-work-with-pyspark-on-google-colab-for-data-scientists/ 3) https://stackoverflow.com/questions/37707305/pyspark-multiple-conditions-in-when-clause 4) https://stackoverflow.com/questions/65732120/pyspark-py4j-py4jexception-method-andclass-javalang-integer-does-not-exist