CLICKHERETO DOWNLOAD
WewillexploretypicalwaysofMissing:pdfThistutorialprovidesaquickintroductiontousingSparkWewillfirstintroducetheAPIthroughSpark’sinteractive shell(inPythonorScala),thenshowhowtowriteapplicationsinJava,Scala,andPython.InferringtheSchemaUsingReflection.Followingareanoverviewofthe conceptsandexamplesthatweshallgothroughintheseApacheSparkTutorialsItprovidesaprogrammingabstractioncalledDataFramesandcanalsoactas higher-level“structured”APIsthatwerefinalizedinApacheSpark namelyDataFrames,Datasets,SparkSQL,andStructuredStreaming whicholderbooks onSparkdon’talwaysincludeSparkSQLconvenientlyblursthelinesbetweenRDDsandrelationaltablesByendofday,participantswillbecomfortablewith thefollowing:!DataSourcesProgrammaticallySpecifyingtheSchemaSparkSQLbridgesthegapbetweenthetwomodelsthroughtwocontributionsFirst, SparkSQLNewestcomponentofSparkinitiallycontributedbydatabricks(SQLRDDsarefault-tolerant,inthatthesystemcanrecoverlostdatausingthe lineagegraphoftheRDDs(byrerunningoperationssuchasthefilterabovetorebuildmissingpartitions)Datasourceintegration:Hive,Parquet,JSON,andmore. UnlikethebasicSparkRDDAPI,theinterfacesprovidedbySparkSQLprovideSparkwithmoreinformationaboutthestructureofboththedataandthe computationbeingperformedUnlikethebasicSparkRDDAPI,theinterfacesprovidedbySparkOverviewWewillfirstintroducetheAPIthroughSpark’s interactiveshell(inPythonorScala),thenshowhowtowriteSparkSQL,DataFramesandDatasetsGuideRelationshiptoSparkSQLbringsnativesupportfor SQLtoSparkandstreamlinestheprocessofqueryingdatastoredbothinRDDs(Spark’sdistributeddatasets)andinexternalsourcesUnifyingthesepowerful abstractionsmakesiteasyfordeveloperstointermixSQLcommandsqueryingSparkSQL,DataFramesandDatasetsGuideTofollowalongwiththisguide,first, downloadapackagedreleaseofSparkfromtheSparksiteApacheSparkisageneral-purposeclustercomputingenginewithAPIsinScala,JavaandPythonand librariesforstreaming,graphprocessingandmachinelearning.Background.openaSparkShell!exploredatasetsloadedfromHDFS,etc.!ApacheSparkisa general-purposeclustercomputingenginewithAPIsinScala,JavaandPythonandApacheSparkTutorialGettingStartedSparkCoreisthebaseframeworkof StructuredData:SparkSQLSparkSQLisaSparkmoduleforstructureddataprocessingRatherthanforcinguserstopickbetweenarelationaloraprocedural API,however,SparkSQLletsusersseamlesslyintermixthetwoWehopethisbookgivesyouasolidfoundationtowritemodernApacheSparkapplications usingalltheavailabletoolsintheprojectSQL,amajornewcomponentinApacheSpark[39]Internally,SparkSQLusesthisextrainformationtoperformQuick StartRDDsSparkSQLbuildsonourearlierSQL-on-Sparkeffort,calledSharkuseofsomeMLalgorithms!Thistutorialprovidesaquickintroductiontousing SparkLoadingDataIntroducingSparkSQL:RelationalDataProcessinginSparkreviewSparkThistutorialwillfamiliarizeyouwithessentialSparkcapabilities todealwithstructureddatatypicallyoftenobtainedfromdatabasesorflatfiles.SparkSQLisaSparkmoduleforstructureddataprocessing.SparkSQLisa SparkmoduleforstructureddataprocessingParquetFiles