EssentialGuidetoPandas
IbonMartínez-Arranz
EssentialGuidetoPandas:HarnessingthePower
ofDataAnalysis
Welcometoourin-depthmanualonPandas,acornerstonePythonlibrarythatisindispensableinthe realmsofdatascienceandanalysis.Pandasprovidesarichsetoftoolsandfunctionsthatmakedata analysis,manipulation,andvisualizationbothaccessibleandpowerful.
Pandas,shortfor“PanelData”,isanopen-sourcelibrarythatoffershigh-leveldatastructuresandavast arrayoftoolsforpracticaldataanalysisinPython.Ithasbecomesynonymouswithdatawrangling,offeringtheDataFrameasitscentraldatastructure,whichiseffectivelyatableoratwo-dimensional,sizemutable,andpotentiallyheterogeneoustabulardatastructurewithlabeledaxes(rowsandcolumns).
TobeginusingPandas,it’stypicallyimportedalongsideNumPy,anotherkeylibraryfornumerical computations.TheconventionalwaytoimportPandasisasfollows:
1 import pandas as pd
2 import numpy as np
Inthismanual,wewillexplorethemultifacetedfeaturesofPandas,coveringawiderangeoffunctionalitiesthatcatertotheneedsofdataanalystsandscientists.Ourguidewillwalkyouthroughthe followingkeyareas:
1. DataLoading: LearnhowtoefficientlyimportdataintoPandasfromdifferentsourcessuchas CSVfiles,Excelsheets,anddatabases.
2. BasicDataInspection: Understandthestructureandcontentofyourdatathroughsimpleyet powerfulinspectiontechniques.
3. DataCleaning: Learntoidentifyandrectifyinconsistencies,missingvalues,andanomaliesin yourdataset,ensuringdataqualityandreliability.
4. DataTransformation: Discovermethodstoreshape,aggregate,andmodifydatatosuityour analyticalneeds.
5. DataVisualization: IntegratePandaswithvisualizationtoolstocreateinsightfulandcompelling graphicalrepresentationsofyourdata.
6. StatisticalAnalysis: UtilizePandasfordescriptiveandinferentialstatistics,makingdata-driven decisionseasierandmoreaccurate.
7. IndexingandSelection: Mastertheartofaccessingandselectingdatasubsetsefficientlyfor analysis.
8. DataFormattingandConversion: Adaptyourdataintothedesiredformat,enhancingits usabilityandcompatibilitywithdifferentanalysistools.
9. AdvancedDataTransformation: Delvedeeperintosophisticateddatatransformationtechniquesforcomplexdatamanipulationtasks.
10. HandlingTimeSeriesData: Explorethehandlingoftime-stampeddata,crucialfortimeseries analysisandforecasting.
11. FileImport/Export: Learnhowtoeffortlesslyreadfromandwritetovariousfileformats,making datainterchangeseamless.
12. AdvancedQueries: Employadvancedqueryingtechniquestoextractspecificinsightsfromlarge datasets.
13. Multi-IndexOperations: Understandthemulti-levelindexingtoworkwithhigh-dimensional datamoreeffectively.
14. DataMergingTechniques: Explorevariousstrategiestocombinedatasets,enhancingyour analyticalpossibilities.
15. DealingwithDuplicates: Detectandhandleduplicaterecordstomaintaintheintegrityofyour analysis.
16. CustomOperationswithApply: HarnessthepowerofcustomfunctionstoextendPandas’ capabilities.
17. IntegrationwithMatplotlibforCustomPlots: CreatebespokeplotsbyintegratingPandaswith Matplotlib,aleadingplottinglibrary.
18. AdvancedGroupingandAggregation: Performcomplexgroupingandaggregationoperations forsophisticateddatasummaries.
19. TextDataSpecificOperations: ManipulateandanalyzetextualdataeffectivelyusingPandas’ stringfunctions.
20. WorkingwithJSONandXML: HandlemoderndataformatslikeJSONandXMLwithease.
21. AdvancedFileHandling: LearnadvancedtechniquesformanagingfileI/Ooperations.
22. DealingwithMissingData: Developstrategiestoaddressandimputemissingvaluesinyour datasets.
23. DataReshaping: Transformthestructureofyourdatatofacilitatedifferenttypesofanalysis.
24. CategoricalDataOperations: Efficientlymanageandanalyzecategoricaldata.
25. AdvancedIndexing: Leverageadvancedindexingtechniquesformorepowerfuldatamanipulation.
26. EfficientComputations: Optimizeperformanceforlarge-scaledataoperations.
27. AdvancedDataMerging: Exploresophisticateddatamergingandjoiningtechniquesforcomplex datasets.
28. DataQualityChecks: Implementstrategiestoensureandmaintainthequalityofyourdata throughouttheanalysisprocess.
29. Real-WorldCaseStudies:Applytheconceptsandtechniqueslearnedthroughoutthemanual toreal-worldscenariosusingtheTitanicdataset.Thischapterdemonstratespracticaldata analysisworkflows,includingdatacleaning,exploratoryanalysis,andsurvivalanalysis,providing insightsintohowtoutilizePandasinpracticalapplicationstoderivemeaningfulconclusions fromcomplexdatasets.
Thismanualisdesignedtoempoweryouwiththeknowledgeandskillstoeffectivelymanipulateand analyzedatausingPandas,turningrawdataintovaluableinsights.Let’sbeginourjourneyintothe worldofdataanalysiswithPandas.
Pandas,beingacornerstoneinthePythondataanalysislandscape,hasawealthofresourcesandreferencesavailableforthoselookingtodelvedeeperintoitscapabilities.Belowaresomekeyreferences andresourceswhereyoucanfindadditionalinformation,documentation,andsupportforworking withPandas:
1. OfficialPandasWebsiteandDocumentation:
• TheofficialwebsiteforPandasis pandas.pydata.org.Here,youcanfindcomprehensive documentation,includingadetaileduserguide,APIreference,andnumeroustutorials. Thedocumentationisaninvaluableresourceforbothbeginnersandexperiencedusers, offeringdetailedexplanationsofPandas’functionalitiesalongwithexamples.
2. PandasGitHubRepository:
• ThePandasGitHubrepository, github.com/pandas-dev/pandas,istheprimarysourceof thelatestsourcecode.It’salsoahubforthedevelopmentcommunitywhereyoucanreport issues,contributetothecodebase,andreviewupcomingfeatures.
3. PandasCommunityandSupport:
• StackOverflow: Alargenumberofquestionsandanswerscanbefoundunderthe‘pandas’tagonStackOverflow.It’sagreatplacetoseekhelpandcontributetocommunity discussions.
• MailingList: Pandashasanactivemailinglistfordiscussionandaskingquestionsabout usageanddevelopment.
• SocialMedia: FollowPandasonplatformslikeTwitterforupdates,tips,andcommunity interactions.
4. ScientificPythonEcosystem:
• PandasisapartofthelargerecosystemofscientificcomputinginPython,whichincludes librarieslikeNumPy,SciPy,Matplotlib,andIPython.Understandingtheselibrariesin conjunctionwithPandascanbehighlybeneficial.
5. BooksandOnlineCourses:
• TherearenumerousbooksandonlinecoursesavailablethatcoverPandas,oftenwithinthe broadercontextofPythondataanalysisanddatascience.Thesecanbeexcellentresources forstructuredlearningandin-depthunderstanding.
6. CommunityConferencesandMeetups:
• PythonanddatascienceconferencesoftenfeaturetalksandworkshopsonPandas.Local Pythonmeetupscanalsobeagoodplacetolearnfromandnetworkwithotherusers.
7. JupyterNotebooks:
• ManyonlinerepositoriesandplatformshostJupyterNotebooksshowcasingPandasuse cases.Theseinteractivenotebooksareexcellentforlearningbyexampleandexperimenting withcode.
Byexploringtheseresources,youcandeepenyourunderstandingofPandas,stayupdatedwiththe latestdevelopments,andconnectwithavibrantcommunityofusersandcontributors.
DataLoading
Efficientdataloadingisfundamentaltoanydataanalysisprocess.Pandasoffersseveralfunctionsto readdatafromdifferentformats,makingiteasiertomanipulateandanalyzethedata.Inthischapter, wewillexplorehowtoreaddatafromCSVfiles,Excelfiles,andSQLdatabasesusingPandas.
ReadCSVFile
The read_csv functionisusedtoloaddatafromCSVfilesintoaDataFrame.Thisfunctionishighly customizablewithnumerousparameterstohandledifferentformatsanddatatypes.Hereisabasic example:
1 import pandas as pd
2
3 # Load data from a CSV file into a DataFrame
4 df = pd.read_csv('filename.csv')
Thiscommandreadsdatafrom‘filename.csv’andstoresitintheDataFrame df.Thefilepathcanbea URLoralocalfilepath.
ReadExcelFile
ToreaddatafromanExcelfile,usethe read_excel function.Thisfunctionsupportsreadingfrom bothxlsandxlsxfileformatsandallowsyoutospecifythesheettobeloaded.
1 # Load data from an Excel file into a DataFrame 2 df = pd.read_excel('filename.xlsx')
ThisreadsthefirstsheetintheExcelworkbook‘filename.xlsx’bydefault.Youcanspecifyadifferent sheetbyusingthe sheet_name parameter.
ReadfromSQLDatabase
PandascanalsoloaddatadirectlyfromaSQLdatabaseusingthe read_sql function.Thisfunction requiresaSQLqueryandaconnectionobjecttothedatabase.
1 import sqlalchemy
2
3 # Create a connection to a SQL database
4 engine = sqlalchemy.create_engine('sqlite:///example.db')
5 query = "SELECT * FROM my_table"
6
7 # Load data from a SQL database into a DataFrame
8 df = pd.read_sql(query, engine)
ThisexampledemonstrateshowtoconnecttoaSQLitedatabaseandreaddatafrom‘my_table’intoa DataFrame.
BasicDataInspection
DisplayTopRows(df.head())
Thiscommand,df.head(),displaysthefirstfiverowsoftheDataFrame,providingaquickglimpse ofthedata,includingcolumnnamesandsomeofthevalues.
1 A B C D E
2 0810.692744 Yes 2023-01-01-1.082325
3 1540.316586 Yes 2023-01-020.031455
4 2570.860911 Yes 2023-01-03-2.599667
5 360.182256 No 2023-01-04-0.603517
6 4820.210502 No 2023-01-05-0.484947
DisplayBottomRows(df.tail())
Thiscommand,df.tail(),showsthelastfiverowsoftheDataFrame,usefulforcheckingtheendof yourdataset.
1 A B C D E
2 5730.463415 No 2023-01-06-0.442890
3 6130.513276 No 2023-01-07-0.289926
4 7230.528147 Yes 2023-01-081.521620
5 8870.138674 Yes 2023-01-09-0.026802
6 9390.005347 No 2023-01-10-0.159331
DisplayDataTypes(df.dtypes)
Thiscommand, df.types(),returnsthedatatypesofeachcolumnintheDataFrame.It’shelpfulto understandthekindofdata(integers,floats,strings,etc.)eachcolumnholds.
1 A int64
2 B float64
3 C object
4 D datetime64[ns]
5 E float64
SummaryStatistics(df.describe())
Thiscommand, df.describe(),providesdescriptivestatisticsthatsummarizethecentraltendency, dispersion,andshapeofadataset’sdistribution,excluding NaN values.It’susefulforaquickstatistical overview.
1 A B E
2 count 10.00000010.00000010.000000
3 mean 51.5000000.391186-0.413633
4 std 29.9638670.2676981.024197
5 min 6.0000000.005347-2.599667
6 25%27.0000000.189317-0.573874
7 50%55.5000000.390001-0.366408
8 75%79.0000000.524429-0.059934
9 max 87.0000000.8609111.521620
DisplayIndex,Columns,andData(df.info())
Thiscommand, df.info(),providesaconcisesummaryoftheDataFrame,includingthenumberof non-nullvaluesineachcolumnandthememoryusage.It’sessentialforinitialdataassessment.
1 <class 'pandas.core.frame.DataFrame'>
2 RangeIndex:10 entries,0 to 9
3 Data columns (total 5 columns):
4 # Column Non-Null Count Dtype
5
6 0 A 10 non-null int64
7 1 B 10 non-null float64
8 2 C 10 non-null object
9 3 D 10 non-null datetime64[ns]
10 4 E 10 non-null float64
11 dtypes: datetime64[ns](1), float64(2), int64(1), object(1)
12 memory usage:528.0 bytes
DataCleaning
Let’sgothroughthedatacleaningprocessinamoredetailedmanner,stepbystep.Wewillstartby creatingaDataFramethatincludesmissing(NA or null)values,thenapplyvariousdatacleaning operations,showingboththecommandsusedandtheresultingoutputs.
First,wecreateasampleDataFramethatincludessomemissingvalues:
1 import pandas as pd
2
3 # Sample DataFrame with missing values
4 data ={
5 'old_name':[1,2, None,4,5],
6 'B':[10, None,12, None,14],
7 'C':['A', 'B', 'C', 'D', 'E'],
8 'D': pd.date_range(start = '2023-01-01', periods =5, freq = 'D'),
9 'E':[20,21,22,23,24]
10 }
11 df = pd.DataFrame(data)
ThisDataFramecontainsmissingvaluesincolumns‘old_name’and‘B’.
CheckingforMissingValues
Tofindoutwherethemissingvaluesarelocated,weuse:
1 missing_values = df.isnull().sum()
Result:
1 old_name 1
2 B 2
3 C 0
4 D 0
5 E 0
6 dtype: int64
FillingMissingValues
Wecanfillmissingvalueswithaspecificvalueoracomputedvalue(likethemeanofthecolumn):
1 filled_df = df.fillna({'old_name':0, 'B': df['B'].mean()})
Result:
1 old_name B C D E
2 01.010.0 A 2023-01-0120
3 12.012.0 B 2023-01-0221
4 20.012.0 C 2023-01-0322
5 34.012.0 D 2023-01-0423
6 45.014.0 E 2023-01-0524
DroppingMissingValues
Alternatively,wecandroprowswithmissingvalues:
1 dropped_df = df.dropna(axis = 'index')
Result:
1 old_name B C D E
2 01.010.0 A 2023-01-0120
3 45.014.0 E 2023-01-0524
Wecanalsodropcolumnswithmissingvalues:
1 dropped_df = df.dropna(axis = 'columns')
Result:
1 C D E
2 0 A 2023-01-0120
3 1 B 2023-01-0221
4 2 C 2023-01-0322
5 3 D 2023-01-0423
6 4 E 2023-01-0524
RenamingColumns
Torenamecolumnsforclarityorstandardization:
EssentialGuidetoPandas
1 renamed_df = df.rename(columns ={'old_name': 'A'})
Result:
1 A B C D E
2 01.010.0 A 2023-01-0120
3 12.0 NaN B 2023-01-0221
4 2 NaN 12.0 C 2023-01-0322
5 34.0 NaN D 2023-01-0423
6 45.014.0 E 2023-01-0524
DroppingColumns
Toremoveunnecessarycolumns:
1 dropped_columns_df = df.drop(columns =['E'])
Result:
1 old_name B C D
2 01.010.0 A 2023-01-01
3 12.0 NaN B 2023-01-02
4 2 NaN 12.0 C 2023-01-03
5 34.0 NaN D 2023-01-04
6 45.014.0 E 2023-01-05
EachofthesestepsdemonstratesafundamentalaspectofdatacleaninginPandas,crucialforpreparing yourdatasetforfurtheranalysis.
DataTransformation
Datatransformationisacrucialstepinpreparingyourdatasetforanalysis.Pandasprovidespowerful toolstotransform,summarize,andcombinedataefficiently.Thischaptercoverskeytechniques suchasapplyingfunctions,groupingandaggregatingdata,creatingpivottables,andmergingor concatenatingDataFrames.
ApplyFunction
The apply functionallowsyoutoapplyacustomfunctiontotheDataFrameelements.Thismethodis extremelyflexibleandcanbeappliedtoasinglecolumnortheentireDataFrame.Here’sanexample using apply onasinglecolumntocalculatethesquareofeachvalue:
1 import pandas as pd
2
3 # Sample DataFrame
4 data ={'number':[1,2,3,4,5]}
5 df = pd.DataFrame(data)
6
7 # Applying a lambda function to square each value
8 df['squared']= df['number'].apply(lambda x: x**2)
Result:
1 number squared 2 011
3 124 4 239 5 3416 6 4525
GroupByandAggregate
Groupingandaggregatingdataareessentialforsummarizingdata.Here’showyoucangroupbyone columnandaggregateanothercolumnusing sum:
1 # Sample DataFrame
2 data ={'group':['A', 'A', 'B', 'B', 'C'], 3 'value':[10,15,10,20,30]}
4 df = pd.DataFrame(data)
5
6 # Group by the 'group' column and sum the 'value' column
7 grouped_df = df.groupby('group').agg({'value': 'sum'})
Result:
1 value
2 group
3 A 25
4 B 30
5 C 30
PivotTables
PivottablesareusedtosummarizeandreorganizedatainaDataFrame.Here’sanexampleofcreating apivottabletofindthemeanvalues:
1 # Sample DataFrame
2 data ={'category':['A', 'A', 'B', 'B', 'A'],
3 'value':[100,200,300,400,150]}
4 df = pd.DataFrame(data)
5
6 # Creating a pivot table
7 pivot_table = df.pivot_table(index = 'category', values = 'value', aggfunc = 'mean')
Result:
1 value
2 category
3 A 150.0
4 B 350.0
MergeDataFrames
MergingDataFramesisakintoperformingSQLjoins.Here’sanexampleofmergingtwoDataFrameson acommoncolumn:
1 # Sample DataFrames
2 data1 ={'id':[1,2,3],
EssentialGuidetoPandas
3 'name':['Alice', 'Bob', 'Charlie']}
4 df1 = pd.DataFrame(data1)
5 data2 ={'id':[1,2,4],
6 'age':[25,30,35]}
7 df2 = pd.DataFrame(data2)
8
9 # Merging df1 and df2 on the 'id' column
10 merged_df = pd.merge(df1, df2, on = 'id')
Result:
1 id name age
2 01 Alice 25
3 12 Bob 30
ConcatenateDataFrames
ConcatenatingDataFramesisusefulwhenyouneedtocombinesimilardatafromdifferentsources. Here’showtoconcatenatetwoDataFrames:
1 # Sample DataFrames
2 data3 ={'name':['David', 'Ella'],
3 'age':[28,22]}
4 df3 = pd.DataFrame(data3)
5
6 # Concatenating df2 and df3
7 concatenated_df = pd.concat([df2, df3])
Result:
1 id age name
2 01.025 NaN
3 12.030 NaN
4 24.035 NaN
5 0 NaN 28 David
6 1 NaN 22 Ella
Thesetechniquesprovidearobustframeworkfortransformingdata,allowingyoutoprepareand analyzeyourdatasetsmoreeffectively.
DataVisualizationIntegration
Visualizingdataisapowerfulwaytounderstandandcommunicatetheunderlyingpatternsandrelationshipswithinyourdataset.PandasintegratesseamlesslywithMatplotlib,acomprehensivelibrary forcreatingstatic,animated,andinteractivevisualizationsinPython.Thischapterdemonstrateshow tousePandasforcommondatavisualizations.
Histogram
Histogramsareusedtoplotthedistributionofadataset.Here’showtocreateahistogramfroma DataFramecolumn:
1 import pandas as pd
2 import matplotlib.pyplot as plt
3
4 # Sample DataFrame
5 data ={'scores':[88,76,90,84,65,79,93,80]}
6 df = pd.DataFrame(data)
7
8 # Creating a histogram
9 df['scores'].hist()
10 plt.title('Distribution of Scores')
11 plt.xlabel('Scores')
12 plt.ylabel('Frequency')
13 plt.show()
Boxplot
Boxplotsareusefulforvisualizingthedistributionofdatathroughtheirquartilesanddetectingoutliers. Here’showtocreateboxplotsformultiplecolumns:
1 # Sample DataFrame
2 data ={'math_scores':[88,76,90,84,65],
3 'eng_scores':[78,82,88,91,73]}
4 df = pd.DataFrame(data)
5
6 # Creating a boxplot
7 df.boxplot(column =['math_scores', 'eng_scores'])
8 plt.title('Score Distribution')
9 plt.ylabel('Scores')
10 plt.show()
ScatterPlot
Scatterplotsareidealforexaminingtherelationshipbetweentwonumericvariables.Here’showto createascatterplot:
1 # Sample DataFrame
2 data ={'hours_studied':[10,15,8,12,6],
3 'test_score':[95,80,88,90,70]}
4 df = pd.DataFrame(data)
5
6 # Creating a scatter plot
7 df.plot.scatter(x = 'hours_studied', y = 'test_score', c = 'DarkBlue')
8 plt.title('Test Score vs Hours Studied')
9 plt.xlabel('Hours Studied')
10 plt.ylabel('Test Score')
11 plt.show()
LinePlot
Lineplotsareusedtovisualizedatapointsconnectedbystraightlinesegments.Thisisparticularly usefulintimeseriesanalysis:
1 # Sample DataFrame
2 data ={'year':[2010,2011,2012,2013,2014],
3 'sales':[200,220,250,270,300]}
4 df = pd.DataFrame(data)
5
6 # Creating a line plot
7 df.plot.line(x = 'year', y = 'sales', color = 'red')
8 plt.title('Yearly Sales')
9 plt.xlabel('Year')
10 plt.ylabel('Sales')
11 plt.show()
EssentialGuidetoPandas
BarChart
Barchartsareusedtocomparedifferentgroups.Here’sanexampleofabarchartvisualizingthecount ofvaluesinacolumn:
1 # Sample DataFrame
2 data ={'product':['Apples', 'Oranges', 'Bananas', 'Apples', 'Oranges' , 'Apples']}
3 df = pd.DataFrame(data)
4
5 # Creating a bar chart
6 df['product'].value_counts().plot.bar(color = 'green')
7 plt.title('Product Frequency')
8 plt.xlabel('Product')
9 plt.ylabel('Frequency')
10 plt.show()
Eachofthesevisualizationtechniquesprovidesinsightsintodifferentaspectsofyourdata,makingit easiertoperformcomprehensivedataanalysisandinterpretation.
StatisticalAnalysis
Statisticalanalysisisakeycomponentofdataanalysis,helpingtounderstandtrends,relationships, anddistributionsindata.Pandasoffersarangeoffunctionsforperformingstatisticalanalyses,which canbeincrediblyinsightfulwhenexploringyourdata.Thischapterwillcoverthebasics,including correlation,covariance,andvariouswaysofsummarizingdatadistributions.
CorrelationMatrix
Acorrelationmatrixdisplaysthecorrelationcoefficientsbetweenvariables.Eachcellinthetable showsthecorrelationbetweentwovariables.Here’showtogenerateacorrelationmatrix:
1 import pandas as pd
2
3 # Sample DataFrame
4 data ={'age':[25,30,35,40,45],
5 'salary':[50000,44000,58000,62000,66000]}
6 df = pd.DataFrame(data)
7
8 # Creating a correlation matrix
9 corr_matrix = df corr()
10 print(corr_matrix)
Result:
1 age salary
2 age 1.0000000.883883
3 salary 0.8838831.000000
CovarianceMatrix
Thecovariancematrixissimilartoacorrelationmatrixbutshowsthecovariancebetweenvariables. Here’showtogenerateacovariancematrix:
1 # Creating a covariance matrix
2 cov_matrix = df.cov()
3 print(cov_matrix)
Result:
1 age salary
2 age 62.56250.0
3 salary 6250.080000000.0
ValueCounts
Thisfunctionisusedtocountthenumberofuniqueentriesinacolumn,whichcanbeparticularly usefulforcategoricaldata:
1 # Sample DataFrame
2 data ={'department':['HR', 'Finance', 'IT', 'HR', 'Finance']}
3 df = pd.DataFrame(data)
4
5 # Using value counts
6 value_counts = df['department'].value_counts()
7 print(value_counts)
Result:
1 Finance 2
2 HR 2
3 IT 1
UniqueValuesinColumn
Tofinduniquevaluesinacolumn,usethe unique function.Thiscanhelpidentifythediversityof entriesinacolumn:
1 # Getting unique values from the column
2 unique_values = df['department'].unique()
3 print(unique_values)
Result:
1 ['HR' 'Finance' 'IT']
NumberofUniqueValues
Ifyouneedtoknowhowmanyuniquevaluesareinacolumn,use nunique:
1 # Counting unique values
2 num_unique_values = df['department'].nunique()
3 print(num_unique_values)
Result:
1 3
Thesetoolsprovideafundamentalinsightintothestatisticalcharacteristicsofyourdata,essentialfor bothpreliminarydataexplorationandadvancedanalyses.
IndexingandSelection
EffectivedatamanipulationinPandasofteninvolvespreciseindexingandselectiontoisolatespecific datasegments.ThischapterdemonstratesseveralmethodstoselectcolumnsandrowsinaDataFrame, enablingrefineddataanalysis.
SelectColumn
ToselectasinglecolumnfromaDataFrameandreturnitasaSeries:
1 import pandas as pd
2
3 # Sample DataFrame
4 data ={'name':['Alice', 'Bob', 'Charlie'],
5 'age':[25,30,35]}
6 df = pd.DataFrame(data)
7
8 # Selecting a single column
9 selected_column = df['name']
10 print(selected_column)
Result:
1 0 Alice
2 1 Bob
3 2 Charlie
4 Name: name, dtype: object
SelectMultipleColumns
Toselectmultiplecolumns,usealistofcolumnnames.TheresultisanewDataFrame:
1 # Selecting multiple columns
2 selected_columns = df[['name', 'age']]
3 print(selected_columns)
Result:
1 name age
2 0 Alice 25
3 1 Bob 30
4 2 Charlie 35
SelectRowsbyPosition
Youcanselectrowsbasedontheirpositionusing iloc,whichisprimarilyintegerpositionbased:
1 # Selecting rows by position
2 selected_rows = df.iloc[0:2]
3 print(selected_rows)
Result:
1 name age
2 0 Alice 25
3 1 Bob 30
SelectRowsbyLabel
Toselectrowsbylabelindex,use loc,whichuseslabelsintheindex:
1 # Selecting rows by label
2 selected_rows_by_label = df.loc[0:1]
3 print(selected_rows_by_label)
Result:
1 name age
2 0 Alice 25
3 1 Bob 30
ConditionalSelection
Forconditionalselection,useaconditionwithinbracketstofilterdatabasedoncolumnvalues:
1 # Conditional selection
2 condition_selected = df[df['age']>30]
3 print(condition_selected)
Result:
1 name age
2 2 Charlie 35
ThisselectionandindexingfunctionalityinPandasallowsforflexibleandefficientdatamanipulations, formingthebasisofmanydataoperationsyou’llperform.
DataFormattingandConversion
Dataoftenneedstobeformattedorconvertedtodifferenttypestomeettherequirementsofvarious analysistasks.Pandasprovidesversatilecapabilitiesfordataformattingandtypeconversion,allowing foreffectivemanipulationandpreparationofdata.Thischaptercoverssomeessentialoperationsfor dataformattingandconversion.
ConvertDataTypes
ChangingthedatatypeofacolumninaDataFrameisoftennecessaryduringdatacleaningand preparation.Use astype toconvertthedatatypeofacolumn:
1 import pandas as pd
2
3 # Sample DataFrame
4 data ={'age':['25', '30', '35']}
5 df = pd.DataFrame(data)
6
7 # Converting the data type of the 'age' column to integer
8 df['age']= df['age'].astype(int)
9 print(df['age'].dtypes)
Result:
1 int64
StringOperations
PandascanperformvectorizedstringoperationsonSeriesusing .str.Thisisusefulforcleaningand transformingtextdata:
1 # Sample DataFrame
2 data ={'name':['Alice', 'Bob', 'Charlie']}
3 df = pd.DataFrame(data)
4
5 # Converting all names to lowercase
6 df['name']= df['name'].str.lower()
7 print(df)
Result:
1 name
2 0 alice
3 1 bob
4 2 charlie
DatetimeConversion
Convertingstringsorotherdatetimeformatsintoastandardized datetime64 typeisessentialfor timeseriesanalysis.Use pd.to_datetime toconvertacolumn:
1 # Sample DataFrame
2 data ={'date':['2023-01-01', '2023-01-02', '2023-01-03']}
3 df = pd.DataFrame(data)
4
5 # Converting 'date' column to datetime
6 df['date']= pd.to_datetime(df['date'])
7 print(df['date'].dtypes)
Result:
1 datetime64[ns]
SettingIndex
SettingaspecificcolumnastheindexofaDataFramecanfacilitatefastersearches,betteralignment, andeasieraccesstorows:
1 # Sample DataFrame
2 data ={'name':['Alice', 'Bob', 'Charlie'],
3 'age':[25,30,35]}
4 df = pd.DataFrame(data)
5
6 # Setting 'name' as the index
7 df.set_index('name', inplace=True)
8 print(df)
Result:
Theseformattingandconversiontechniquesarecrucialforpreparingyourdatasetfordetailedanalysis andensuringcompatibilityacrossdifferentanalysisandvisualizationtools.
AdvancedDataTransformation
Advanceddatatransformationinvolvessophisticatedtechniquesthathelpinreshaping,restructuring, andsummarizingcomplexdatasets.Thischapterdelvesintosomeofthemoreadvancedfunctions availableinPandasthatenabledetailedmanipulationandtransformationofdata.
LambdaFunctions
LambdafunctionsprovideaquickandefficientwayofapplyinganoperationacrossaDataFrame. Here’showyoucanuse apply withalambdafunctiontoincrementeveryelementintheDataFrame:
1 import pandas as pd
2
3 # Sample DataFrame
4 data ={'A':[1,2,3],
5 'B':[4,5,6]}
6 df = pd.DataFrame(data)
7
8 # Applying a lambda function to add 1 to each element
9 df = df.apply(lambda x: x +1)
10 print(df)
Result:
1 A B
2 025
3 136 4 247
PivotLonger/WiderFormat
The melt functionisusedtotransformdatafromwideformattolongformat,whichcanbemore suitableforanalysis:
1 # Example of melting a DataFrame
2 data ={'Name':['Alice', 'Bob'],
3 'Age':[25,30],
4 'Salary':[50000,60000]}
5 df = pd.DataFrame(data)
6
7 # Pivoting from wider to longer format
8 df_long = df.melt(id_vars =['Name'])
9 print(df_long)
Result:
1 Name variable value
2 0 Alice Age 25
3 1 Bob Age 30
4 2 Alice Salary 50000
5 3 Bob Salary 60000
Stack/Unstack
StackingandunstackingarepowerfulforreshapingaDataFramebypivotingthecolumnsorthe index:
1 # Stacking and unstacking example
2 df = pd.DataFrame(data)
3
4 # Stacking
5 stacked = df.stack()
6 print(stacked)
7
8 # Unstacking
9 unstacked = stacked.unstack()
10 print(unstacked)
Resultforstack:
1 0 Name Alice
2 Age 25
3 Salary 50000
4 1 Name Bob
5 Age 30
6 Salary 60000
7 dtype: object
Resultforunstack:
1 Name Age Salary
2 0 Alice 2550000
3 1 Bob 3060000
CrossTabulations
Crosstabulationsareusedtocomputeasimplecross-tabulationoftwo(ormore)factors.Thiscanbe veryusefulinstatisticsandprobabilityanalysis:
1 # Cross-tabulation example
2 data ={'Gender':['Female', 'Male', 'Female', 'Male'],
3 'Handedness':['Right', 'Left', 'Right', 'Right']}
4 df = pd.DataFrame(data)
5
6 # Creating a cross tabulation
7 crosstab = pd.crosstab(df['Gender'], df['Handedness'])
8 print(crosstab)
Result:
1 Handedness Left Right
2 Gender
3 Female 02
4 Male 11
Theseadvancedtransformationsenablesophisticatedhandlingofdatastructures,enhancingthe abilitytoanalyzecomplexdatasetseffectively.
HandlingTimeSeriesData
Timeseriesdataanalysisisacrucialaspectofmanyfieldssuchasfinance,economics,andmeteorology. Pandasprovidesrobusttoolsforworkingwithtimeseriesdata,allowingfordetailedanalysisoftimestampedinformation.Thischapterwillexplorehowtomanipulatetimeseriesdataeffectivelyusing Pandas.
SetDatetimeIndex
Settingadatetimeindexisfoundationalintimeseriesanalysisasitfacilitateseasierslicing,aggregation, andresamplingofdata:
1 import pandas as pd
2
3 # Sample DataFrame with date information
4 data ={'date':['2023-01-01', '2023-01-02', '2023-01-03', '2023-01-04' ],
5 'value':[100,110,120,130]}
6 df = pd.DataFrame(data)
7
8 # Converting 'date' column to datetime and setting it as index
9 df['date']= pd.to_datetime(df['date'])
10 df = df.set_index('date')
11 print(df)
Result:
1 value
2 date
3 2023-01-01100
4 2023-01-02110
5 2023-01-03120
6 2023-01-04130
ResamplingData
Resamplingisapowerfulmethodfortimeseriesdataaggregationordownsampling,whichchanges thefrequencyofyourdata:
1 # Resampling the data monthly and calculating the mean
2 monthly_mean = df.resample('M').mean()
3 print(monthly_mean)
Result:
1 value
2 date
3 2023-01-31115.0
RollingWindowOperations
Rollingwindowoperationsareusefulforsmoothingorcalculatingmovingaverages,whichcanhelpin identifyingtrendsintimeseriesdata:
1 # Adding more data points for a better rolling example
2 additional_data ={'date': pd.date_range('2023-01-05', periods =5, freq = 'D'),
3 'value':[140,150,160,170,180]}
4 additional_df = pd.DataFrame(additional_data)
5 df = pd.concat([df, additional_df.set_index('date')])
6
7 # Calculating rolling mean with a window of 5 days
8 rolling_mean = df.rolling(window =5).mean()
9 print(rolling_mean)
Result:
1 value
2 date
3 2023-01-01 NaN
4 2023-01-02 NaN
5 2023-01-03 NaN
6 2023-01-04 NaN
7 2023-01-05120.0
8 2023-01-06130.0
9 2023-01-07140.0
10 2023-01-08150.0
11 2023-01-09160.0
Thesetechniquesareessentialforanalyzingtimeseriesdataefficiently,providingthetoolsneededto
EssentialGuidetoPandas
handletrends,seasonality,andothertemporalstructuresindata.
IbonMartínez-ArranzPage39
FileExport
Oncedataanalysisiscomplete,itisoftennecessarytoexportdataintovariousformatsforreporting, furtheranalysis,orsharing.Pandasprovidesversatiletoolstoexportdatatodifferentfileformats, includingCSV,Excel,andSQLdatabases.ThischapterwillcoverhowtoexportDataFramestothese commonformats.
WritetoCSV
ExportingaDataFrametoaCSVfileisstraightforwardandoneofthemostcommonmethodsfordata sharing:
1 import pandas as pd
2
3 # Sample DataFrame
4 data ={'name':['Alice', 'Bob', 'Charlie'],
5 'age':[25,30,35]}
6 df = pd.DataFrame(data)
7
8 # Writing the DataFrame to a CSV file
9 df.to_csv('filename.csv', index = False) # index=False to avoid writing row indices
ThisfunctionwillcreateaCSVfilenamed filename.csv inthecurrentdirectorywithouttheindex column.
WritetoExcel
ExportingdatatoanExcelfilecanbedoneusingthe to_excel method,whichallowsforthestorage ofdataalongwithformattingthatcanbeusefulforreports:
1 # Writing the DataFrame to an Excel file
2 df.to_excel('filename.xlsx', index = False) # index=False to avoid writing row indices
ThiswillcreateanExcelfile filename.xlsx inthecurrentdirectory.
WritetoSQLDatabase
PandascanalsoexportaDataFramedirectlytoaSQLdatabase,whichisusefulforintegratinganalysis resultsintoapplicationsorstoringdatainacentralizeddatabase:
1 import sqlalchemy
2
3 # Creating a SQL connection engine
4 engine = sqlalchemy.create_engine('sqlite:///example.db') # Example using SQLite
5
6 # Writing the DataFrame to a SQL database
7 df.to_sql('table_name', 8 con = engine, 9 index = False, 10 if_exists = 'replace')
The to_sql functionwillcreateanewtablenamed table_name inthespecifiedSQLdatabaseand writetheDataFrametothistable.The if_exists='replace' parameterwillreplacethetableifit alreadyexists;use if_exists='append' toadddatatoanexistingtableinstead.
TheseexportfunctionalitiesenhancetheversatilityofPandas,allowingforseamlesstransitionsbetweendifferentstagesofdataprocessingandsharing.
AdvancedDataQueries
PerformingadvancedqueriesonaDataFrameallowsforprecisedatafilteringandextraction,which isessentialfordetailedanalysis.Thischapterexplorestheuseofthe query functionandthe isin methodforsophisticateddataqueryinginPandas.
QueryFunction
The query functionallowsyoutofilterrowsbasedonaqueryexpression.It’sapowerfulwaytoselect datadynamically:
1 import pandas as pd
2
3 # Sample DataFrame
4 data ={'name':['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
5 'age':[25,30,35,40,45]}
6 df = pd.DataFrame(data)
7
8 # Using query to filter data
9 filtered_df = df.query('age > 30')
10 print(filtered_df)
Result:
1 name age
2 2 Charlie 35
3 3 David 40
4 4 Eve 45
Thisqueryreturnsallrowswherethe age isgreaterthan30.
Filteringwithisin
The isin methodisusefulforfilteringdatarowswherethecolumnvalueisinapredefinedlistof values.It’sespeciallyusefulforcategoricaldata:
1 # Sample DataFrame
2 data ={'name':['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
3 'department':['HR', 'Finance', 'IT', 'HR', 'IT']}
4 df = pd.DataFrame(data)
5
6 # Filtering using isin
7 filtered_df = df[df['department'].isin(['HR', 'IT'])]
8 print(filtered_df)
Result:
1 name department
2 0 Alice HR
3 2 Charlie IT
4 3 David HR
5 4 Eve IT
Thisexamplefiltersrowswherethe department columncontainseither‘HR’or‘IT’.
Theseadvancedqueryingtechniquesenhancetheabilitytoperformtargeteddataanalysis,allowing fortheextractionofspecificsegmentsofdatabasedoncomplexcriteria.
Multi-IndexOperations
Handlinghigh-dimensionaldataoftenrequirestheuseofmulti-levelindexing,orMultiIndex,which allowsyoutostoreandmanipulatedatawithanarbitrarynumberofdimensionsinlower-dimensional datastructureslikeDataFrames.ThischaptercoverscreatingaMultiIndexandperformingslicing operationsonsuchstructures.
CreatingMultiIndex
MultiIndexingenhancesdataaggregationandgroupingcapabilities.Itallowsformorecomplexdata manipulationsandmoresophisticatedanalysis:
1 import pandas as pd
2
3 # Sample DataFrame
4 data ={
5 'state':['CA', 'CA', 'NY', 'NY', 'TX', 'TX'],
6 'year':[2001,2002,2001,2002,2001,2002],
7 'population':[34.5,35.2,18.9,19.7,20.1,20.9]
8 }
9 df = pd.DataFrame(data)
10
11 # Creating a MultiIndex DataFrame
12 df.set_index(['state', 'year'], inplace = True)
13 print(df)
Result:
1 population
2 state year
3 CA 200134.5
4 200235.2
5 NY 200118.9
6 200219.7
7 TX 200120.1
8 200220.9
SlicingonMultiIndex
SlicingaDataFramewithaMultiIndexinvolvesspecifyingtherangesforeachleveloftheindex,which canbedoneusingthe slice functionorbyspecifyingindexvaluesdirectly:
1 # Slicing MultiIndex DataFrame
2 sliced_df = df.loc[(slice('CA', 'NY'),)]
3 print(sliced_df)
Result:
1 population
2 state year
3 CA 200134.5
4 200235.2
5 NY 200118.9
6 200219.7
ThisexampledemonstratesslicingtheDataFrametoincludedatafromstates‘CA’to‘NY’fortheyears 2001and2002.
TheseMultiIndexoperationsareessentialforworkingwithcomplexdatastructureseffectively,enabling morenuanceddataretrievalandmanipulation.
DataMergingTechniques
Mergingdataisafundamentalaspectofmanydataanalysistasks,especiallywhencombininginformationfrommultiplesources.PandasprovidespowerfulfunctionstomergeDataFramesinamanner similartoSQLjoins.Thischapterwillcoverfourprimarytypesofmerges:outer,inner,left,andright joins.
OuterJoin
AnouterjoinreturnsallrecordswhenthereisamatchineithertheleftorrightDataFrame.Ifthereis nomatch,themissingsidewillcontain NaN.
1 import pandas as pd
2
3 # Sample DataFrames
4 data1 ={'column':['A', 'B', 'C'],
5 'values1':[1,2,3]}
6 df1 = pd.DataFrame(data1)
7 data2 ={'column':['B', 'C', 'D'],
8 'values2':[4,5,6]}
9 df2 = pd.DataFrame(data2)
10
11 # Performing an outer join
12 outer_joined = pd.merge(df1, df2, on = 'column', how = 'outer')
13 print(outer_joined)
Result:
1 column values1 values2
2 0 A 1.0 NaN
3 1 B 2.04.0
4 2 C 3.05.0
5 3 D
InnerJoin
AninnerjoinreturnsrecordsthathavematchingvaluesinbothDataFrames.
1 # Performing an inner join
2 inner_joined = pd.merge(df1, df2, on = 'column', how = 'inner')
3 print(inner_joined)
Result:
1 column values1 values2
2 0 B 24
3 1 C 35
LeftJoin
AleftjoinreturnsallrecordsfromtheleftDataFrame,andthematchedrecordsfromtherightDataFrame. Theresultis NaN intherightsidewherethereisnomatch.
1 # Performing a left join
2 left_joined = pd.merge(df1, df2, on = 'column', how = 'left')
3 print(left_joined)
Result:
1 column values1 values2
2 0 A 1 NaN
3 1 B 24.0
4 2 C 35.0
RightJoin
ArightjoinreturnsallrecordsfromtherightDataFrame,andthematchedrecordsfromtheleft DataFrame.Theresultis NaN intheleftsidewherethereisnomatch.
1 # Performing a right join
2 right_joined = pd.merge(df1, df2, on = 'column', how = 'right')
3 print(right_joined)
Result:
1 column values1 values2
2 0 B 24.0
Thesedatamergingtechniquesarecrucialforcombiningdatafromdifferentsources,allowingfor morecomprehensiveanalysesbycreatingaunifieddatasetfrommultipledisparatesources.
DealingwithDuplicates
Duplicatedatacanskewanalysisandleadtoincorrectconclusions,makingitessentialtoidentifyand handleduplicateseffectively.Pandasprovidesstraightforwardtoolstofindandremoveduplicatesin yourdatasets.Thischapterwillguideyouthroughtheseprocesses.
FindingDuplicates
The duplicated() functionreturnsabooleanseriesindicatingwhethereachrowisaduplicateofa rowthatappearedearlierintheDataFrame.Here’showtouseit:
1 import pandas as pd
2
3 # Sample DataFrame
4 data ={'name':['Alice', 'Bob', 'Charlie', 'Bob', 'Charlie'],
5 'age':[25,30,35,30,35]}
6 df = pd.DataFrame(data)
7
8 # Finding duplicates
9 duplicates = df.duplicated()
10 print(duplicates)
Result:
1 0 False
2 1 False
3 2 False
4 3 True
5 4 True
6 dtype: bool
Inthisoutput, True indicatesthattherowisaduplicateofanearlierrowintheDataFrame.
RemovingDuplicates
ToremovetheduplicaterowsfromtheDataFrame,usethe drop_duplicates() function.By default,thisfunctionkeepsthefirstoccurrenceandremovessubsequentduplicates.
1 # Removing duplicates
2 df_unique = df.drop_duplicates()
3 print(df_unique)
Result:
1 name age
2 0 Alice 25
3 1 Bob 30
4 2 Charlie 35
Thismethodhasremovedrows3and4,whichwereduplicatesofearlierrows.Youcanalsocustomize thisbehaviorwiththe keep parameter,whichcanbesetto 'last' tokeepthelastoccurrence insteadofthefirst,or False toremoveallduplicatesentirely.
Thesetechniquesareessentialforensuringdataquality,enablingaccurateandreliabledataanalysis bymaintainingonlyuniquedataentriesinyourDataFrame.
CustomOperationswithApply
The apply functioninPandasishighlyversatile,allowingyoutoexecutecustomfunctionsacross anentireDataFrameoralongaspecifiedaxis.Thisflexibilitymakesitindispensableforperforming complexoperationsthatarenotdirectlysupportedbybuilt-inmethods.Thischapterwilldemonstrate howtouse apply forcustomoperations.
CustomApplyFunctions
Using apply withalambdafunctionallowsyoutodefineinlinefunctionstoapplytoeachrowor columnofaDataFrame.Hereishowyoucanuseacustomfunctiontoprocessdatarow-wise:
1 import pandas as pd
2
3 # Define a custom function
4 def custom_func(x, y):
5 return x *2+ y
6
7 # Sample DataFrame
8 data ={'col1':[1,2,3],
9 'col2':[4,5,6]}
10 df = pd.DataFrame(data)
11
12 # Applying a custom function row-wise
13 df['result']= df.apply(lambda row: custom_func(row['col1'], row['col2' ]), axis =1)
14 print(df)
Result:
1 col1 col2 result
2 0146
3 1259 4 23612
Inthisexample,the custom_func isappliedtoeachrowoftheDataFrameusing apply.Thefunction calculatesanewvaluebasedoncolumns‘col1’and‘col2’foreachrow,andtheresultsarestoredina newcolumn‘result’.
Thismethodofapplyingcustomfunctionsispowerfulfordatamanipulationandtransformation, allowingforoperationsthatgobeyondsimplearithmeticoraggregation.It’sparticularlyusefulwhen youneedtoperformoperationsthatarespecifictoyourdataandnotprovidedbyPandas’built-in methods.
IntegrationwithMatplotlibforCustomPlots
Visualizingdataisakeystepindataanalysis,providinginsightsthatarenotapparentfromrawdata alone.PandasintegratessmoothlywithMatplotlib,apopularplottinglibraryinPython,tooffer versatileoptionsfordatavisualization.ThischapterwillshowhowtocreatecustomplotsusingPandas andMatplotlib.
CustomPlotting
Pandas’plottingcapabilitiesarebuiltonMatplotlib,allowingforstraightforwardgenerationofvarious typesofplotsdirectlyfromDataFrameandSeriesobjects.
LinePlot
Here’showtocreateasimplelineplotdisplayingtrendsoveraseriesofvalues:
1 import pandas as pd
2 import matplotlib.pyplot as plt
3
4 # Sample data
5 data ={'Year':[2010,2011,2012,2013,2014],
6 'Sales':[100,150,200,250,300]}
7 df = pd.DataFrame(data)
8
9 # Plotting
10 df.plot(x = 'Year', y = 'Sales', kind = 'line')
11 plt.title('Yearly Sales')
12 plt.ylabel('Sales')
13 plt.show()
Histogram
Histogramsaregreatforvisualizingthedistributionofnumericaldata:
1 # Sample data
2 data ={'Grades':[88,92,80,89,90,78,84,76,95,92]}
3 df = pd.DataFrame(data)
4
5 # Plotting a histogram
6 df['Grades']\
7 .plot(kind = 'hist',
8 bins =5, 9 alpha =0.7)
10 plt.title('Distribution of Grades')
11 plt.xlabel('Grades')
12 plt.show()
ScatterPlot
Scatterplotsareusedtoobserverelationshipsbetweenvariables:
1 # Sample data
2 data ={'Hours':[1,2,3,4,5],
3 'Scores':[77,78,85,93,89]}
4 df = pd.DataFrame(data)
5
6 # Creating a scatter plot
7 df.plot(kind = 'scatter', x = 'Hours', y = 'Scores')
8 plt.title('Test Scores by Hours Studied')
9 plt.xlabel('Hours Studied')
10 plt.ylabel('Test Scores')
11 plt.show()
BarChart
Barchartsareusefulforcomparingquantitiescorrespondingtodifferentgroups:
1 # Sample data
2 data ={'Bars':['A', 'B', 'C', 'D'],
3 'Values':[10,15,7,10]}
4 df = pd.DataFrame(data)
5
6 # Creating a bar chart
7 df.plot(kind = 'bar', 8 x = 'Bars', 9 y = 'Values', 10 color = 'blue', 11 legend = None)
12 plt.title('Bar Chart Example')
13 plt.ylabel('Values')
EssentialGuidetoPandas
14 plt.show()
TheseexamplesillustratehowtointegratePandaswithMatplotlibtocreateinformativeandvisually appealingplots.Thisintegrationisvitalforanalyzingtrends,distributions,relationships,andpatterns indataeffectively.
AdvancedGroupingandAggregation
Groupingandaggregatingdataarefundamentaloperationsindataanalysis,especiallywhendealing withlargeorcomplexdatasets.Pandasoffersadvancedcapabilitiesthatallowforsophisticatedgroupingandaggregationstrategies.Thischapterexploressomeoftheseadvancedtechniques,including groupingbymultiplecolumns,usingmultipleaggregationfunctions,andapplyingtransformation functions.
GroupbyMultipleColumns
Groupingbymultiplecolumnsallowsyoutoperformmoredetailedanalysis.Here’showtocompute themeanofgroupsdefinedbymultiplecolumns:
1 import pandas as pd
2
3 # Sample DataFrame
4 data ={
5 'Department':['Sales', 'Sales', 'HR', 'HR', 'IT', 'IT'],
6 'Team':['A', 'B', 'A', 'B', 'A', 'B'],
7 'Revenue':[200,210,150,160,220,230]
8 }
9 df = pd.DataFrame(data)
10
11 # Grouping by multiple columns and calculating mean
12 grouped_mean = df.groupby(['Department', 'Team']).mean()
13 print(grouped_mean)
Result:
1 Revenue 2 Department Team
7
8
AggregatewithMultipleFunctions
Youcanapplymultipleaggregationfunctionsatoncetogetabroaderstatisticalsummary:
1 # Applying multiple aggregation functions
2 grouped_agg = df.groupby('Department')['Revenue'].agg(['mean', 'sum'])
3 print(grouped_agg)
Result:
1 Revenue
2 mean sum
3 Department
4
5
6 Sales
TransformFunction
The transform functionisusefulforperformingoperationsthatreturnaDataFramewiththesame indexastheoriginal.Itisparticularlyhandyforstandardizingdatawithingroups:
1 # Using transform to standardize data within groups
2 df['Revenue_normalized']=\
3 df\
4 .groupby('Department')['Revenue']\
5 .transform(lambda x:(x - x.mean())/ x.std())
6 print(df)
Result:
1 Department
2 0
3 1
4 2
5 3
6 4
7 5
Thisexampledemonstrateshowtonormalizethe‘Revenue’withineach‘Department’,showingdeviationsfromthedepartmentmeanintermsofstandarddeviations.
Theseadvancedgroupingandaggregationtechniquesprovidepowerfultoolsforbreakingdown complexdataintomeaningfulsummaries,enablingmorenuancedanalysisandinsights.
TextDataSpecificOperations
Textdataoftenrequiresspecificprocessingtechniquestoextractmeaningfulinformationortoreformat itforfurtheranalysis.Pandasprovidesarobustsetofstringoperationsthatcanbeappliedefficiently toSeriesandDataFrames.Thischapterexploressomeessentialoperationsforhandlingtextdata, includingsearchingforsubstrings,splittingstrings,andusingregularexpressions.
StringContains
The contains methodallowsyoutofilterrowsbasedonwhetheracolumn’stextcontainsaspecified substring.Thisisusefulforsubsettingdatabasedontextualcontent:
1 import pandas as pd
2
3 # Sample DataFrame
4 data ={'Description':['Apple is sweet', 'Banana is yellow', 'Cherry is red']}
5 df = pd.DataFrame(data)
6
7 # Filtering rows where the Description column contains 'sweet'
8 contains_sweet = df[df['Description'].str.contains('sweet')]
9 print(contains_sweet)
Result:
1 Description
2 0 Apple is sweet
StringSplit
Splittingstringsintoseparatecomponentscanbeessentialfordatacleaningandpreparation.The split methodsplitseachstringintheSeries/Indexbythegivendelimiterandoptionallyexpandsto separatecolumns:
1 # Splitting the Description column into words
2 split_description = df['Description'].str.split(' ' , expand = True)
3 print(split_description)
Result:
1 012
2 0 Apple is sweet
3 1 Banana is yellow
4 2 Cherry is red
Thissplitsthe‘Description’columnintoseparatecolumnsforeachword.
RegularExpressionExtraction
Regularexpressionsareapowerfultoolforextractingpatternsfromtext.The extract methodapplies aregularexpressionpatternandextractsgroupsfromthefirstmatch:
1 # Extracting the first word where it starts with a capital letter followed by lower case letters
2 extracted_words = df['Description'].str.extract(r'([A-Z][a-z]+)')
3 print(extracted_words)
Result:
1 0
2 0 Apple
3 1 Banana 4 2 Cherry
Thisregularexpressionextractsthefirstwordfromeachdescription,whichstartswithacapitalletter andisfollowedbylowercaseletters.
Thesetext-specificoperationsinPandassimplifytheprocessofworkingwithtextualdata,allowingfor efficientandpowerfulstringmanipulationandanalysis.
WorkingwithJSONandXML
Intoday’sdata-drivenworld,JSON(JavaScriptObjectNotation)andXML(eXtensibleMarkupLanguage) aretwoofthemostcommonformatsusedforstoringandtransferringdataontheweb.Pandasprovides built-infunctionstoeasilyreadtheseformatsintoDataFrames,facilitatingtheanalysisofstructured data.ThischapterexplainshowtoreadJSONandXMLfilesusingPandas.
ReadingJSON
JSONisalightweightformatthatiseasyforhumanstoreadandwrite,andeasyformachinestoparse andgenerate.PandascandirectlyreadJSONdataintoaDataFrame:
1 import pandas as pd
2
3 # Reading JSON data
4 df = pd.read_json('filename.json')
5 print(df)
ThismethodwillconvertaJSONfileintoaDataFrame.ThekeysoftheJSONobjectwillcorrespondto columnnames,andthevalueswillformthedataentriesfortherows.
ReadingXML
XMLisusedforrepresentingdocumentswithastructuredmarkup.ItismoreverbosethanJSONbut allowsforamorestructuredhierarchy.PandascanreadXMLdataintoaDataFrame,similartohowit readsJSON:
1 # Reading XML data
2 df = pd.read_xml('filename.xml')
3 print(df)
ThiswillparseanXMLfileandcreateaDataFrame.ThetagsoftheXMLfilewilltypicallydefinethe columns,andtheirrespectivecontentwillbethedatafortherows.
Thesefunctionalitiesallowforseamlessintegrationofdatafromwebsourcesandothersystemsthat utilizeJSONorXMLfordatainterchange.ByleveragingPandas’abilitytoworkwiththeseformats, analystscanfocusmoreonanalyzingthedataratherthanspendingtimeondatapreparation.
AdvancedFileHandling
Handlingfileswithvariousconfigurationsandformatsisacommonnecessityindataanalysis.Pandas providesextensivecapabilitiesforreadingfromandwritingtodifferentfiletypeswithvaryingdelimiters andformats.ThischapterwillexplorereadingCSVfileswithspecificdelimitersandwritingDataFrames toJSONfiles.
ReadCSVwithSpecificDelimiter
CSVfilescancomewithdifferentdelimiterslikecommas(,),semicolons(;),ortabs(\t).Pandas allowsyoutospecifythedelimiterwhenreadingthesefiles,whichiscrucialforcorrectlyparsingthe data.
ReadingCSVwithSemicolonDelimiter
SupposeyouhaveaCSVfile filename.csv withthefollowingcontent:
1 Name;Age;City
2 Alice;30;New York
3 Bob;25;Los Angeles
4 Charlie;35;Chicago
ToreadthisCSVfileintoaDataFrameusingPandas,specifythesemicolonasthedelimiter:
1 import pandas as pd
2
3 # Reading a CSV file with semicolon delimiter
4 df = pd.read_csv('filename.csv', delimiter = ';')
5 print(df)
Result:
1 Name Age City
2 0 Alice 30 New York
3 1 Bob 25 Los Angeles
4 2 Charlie 35 Chicago
ReadingCSVwithTabDelimiter
IftheCSVfileusestabsasdelimiters,here’showyoumightseethefileandreadit:
Filecontent(filename_tab.csv):
1 Name Age City
2 Alice 30 New York
3 Bob 25 Los Angeles
4 Charlie 35 Chicago
Toreadthisfile:
1 # Reading a CSV file with tab delimiter
2 df_tab = pd.read_csv('filename_tab.csv', delimiter = '\t')
3 print(df_tab)
Result:
1 Name Age City
2 0 Alice 30 New York
3 1 Bob 25 Los Angeles
4 2 Charlie 35 Chicago
WritingtoJSON
WritingdatatoJSONformatcanbeusefulforwebapplicationsandAPIs.Here’showtowritea DataFrametoaJSONfile:
1 # DataFrame to write to JSON 2 df.to_json('filename.json')
Assuming df containsthepreviousdata,theJSONfile filename.json wouldlooklikethis:
1 {"Name":{"0":"Alice","1":"Bob","2":"Charlie"},"Age":{"0":30,"1":25,"2" :35},"City":{"0":"New York","1":"Los Angeles","2":"Chicago"}}
Thisformatisknownas‘column-oriented’JSON.PandasalsosupportsotherJSONorientationswhich canbespecifiedusingthe orient parameter.
Theseadvancedfilehandlingtechniquesensurethatyoucanworkwithawiderangeoffileformatsand configurations,facilitatingdatasharingandintegrationacrossdifferentsystemsandapplications.
DealingwithMissingData
Missingdatacansignificantlyimpacttheresultsofyourdataanalysisifnotproperlyhandled.Pandas providesseveralmethodstodealwithmissingvalues,allowingyoutoeitherfillthesegapsormake interpolationsbasedontheexistingdata.Thischapterexploresmethodslikeinterpolation,forward filling,andbackwardfilling.
InterpolateMissingValues
Interpolationisamethodofestimatingmissingvaluesbyusingotheravailabledatapoints.Itis particularlyusefulintimeseriesdatawherethiscanestimatethetrendsaccurately:
1 import pandas as pd
2 import numpy as np
3
4 # Sample DataFrame with missing values
5 data ={'value':[1, np.nan, np.nan,4,5]}
6 df = pd DataFrame(data)
7
8 # Interpolating missing values
9 df['value']= df['value'].interpolate()
10 print(df)
Result:
1 value 2 01.0
3 12.0
4 23.0 5 34.0 6 45.0
Here, interpolate() linearlyestimatesthemissingvaluesbetweentheexistingnumbers.
ForwardFillMissingValues
Forwardfill(ffill)propagatesthelastobservednon-nullvalueforwarduntilanothernon-nullvalue isencountered:
1 # Sample DataFrame with missing values
2 data ={'value':[1, np.nan, np.nan,4,5]}
3 df = pd.DataFrame(data)
4
5 # Applying forward fill
6 df['value'].ffill(inplace = True)
7 print(df)
Result:
1 value
2 01.0
3 11.0
4 21.0
5 34.0
6 45.0
BackwardFillMissingValues
Backwardfill(bfill)propagatesthenextobservednon-nullvaluebackwardsuntilanothernon-null valueismet:
1 # Sample DataFrame with missing values
2 data ={'value':[1, np.nan, np.nan,4,5]}
3 df = pd.DataFrame(data)
4
5 # Applying backward fill
6 df['value'].bfill(inplace = True)
7 print(df)
Result:
1 value
2 01.0
3 14.0
4 24.0
5 34.0
6 45.0
Thesemethodsprovideyouwithflexibleoptionsforhandlingmissingdatabasedonthenatureofyour datasetandthespecificrequirementsofyouranalysis.Correctlyaddressingmissingdataiscrucialfor
maintainingtheaccuracyandreliabilityofyouranalyticalresults. IbonMartínez-ArranzPage69
DataReshaping
Datareshapingisacrucialaspectofdatapreparationthatinvolvestransformingdatabetweenwide format(withmorecolumns)andlongformat(withmorerows),dependingontheneedsofyour analysis.Thischapterdemonstrateshowtoreshapedatafromwidetolongformatsandviceversa usingPandas.
WidetoLongFormat
The wide_to_long functioninPandasisapowerfultoolfortransformingdatafromwideformatto longformat,whichisoftenmoreamenabletoanalysisinPandas:
1 import pandas as pd
2
3 # Sample DataFrame in wide format
4 data ={
5 'id':[1,2],
6 'A_2020':[100,200],
7 'A_2021':[150,250],
8 'B_2020':[300,400],
9 'B_2021':[350,450]
10 }
11 df = pd.DataFrame(data)
12
13 # Transforming from wide to long format
14 long_df = pd.wide_to_long(df, stubnames =['A', 'B'], sep = ' ' , i = ' id', j = 'year')
15 print(long_df)
Result:
1 A B
2 id year
3 12020100300
4 2021150350
5 22020200400
6 2021250450
ThisoutputrepresentsaDataFrameinlongformatwhereeachrowcorrespondstoasingleyearfor eachvariable(AandB)andeachid.
LongtoWideFormat
Convertingdatafromlongtowideformatinvolvescreatingapivottable,whichcansimplifycertain typesofdataanalysisbydisplayingdatawithonevariablepercolumnandcombinationsofother variablesperrow:
1 # Assuming long_df is the DataFrame in long format from the previous example
2 # We will use a slight modification for clarity
3 long_data ={
4 'id':[1,1,2,2],
5 'year':[2020,2021,2020,2021],
6 'A':[100,150,200,250],
7 'B':[300,350,400,450]
8 }
9 long_df = pd.DataFrame(long_data)
10
11 # Transforming from long to wide format
12 wide_df = long_df.pivot(index = 'id', columns = 'year')
13 print(wide_df)
Result:
1 A B
2 year 2020202120202021
3 id
4 1100150300350
5 2200250400450
ThisresultdemonstratesaDataFrameinwideformatwhereeach id hasassociatedvaluesofAandB foreachyearspreadacrossmultiplecolumns.
Reshapingdataeffectivelyallowsforeasieranalysis,particularlywhendealingwithpaneldataortime seriesthatrequireoperationsacrossdifferentdimensions.
CategoricalDataOperations
Categoricaldataiscommoninmanydatasetsinvolvingcategoriesorlabels,suchassurveyresponses, producttypes,oruserroles.Efficienthandlingofsuchdatacanleadtosignificantperformance improvementsandeaseofuseindatamanipulationandanalysis.Pandasprovidesrobustsupportfor categoricaldata,includingconvertingdatatypestocategoricalandspecifyingtheorderofcategories.
ConvertColumntoCategorical
Convertingacolumntoacategoricaltypecanoptimizememoryusageandimproveperformance, especiallyforlargedatasets.Here’showtoconvertacolumntocategorical:
1 import pandas as pd
2
3 # Sample DataFrame
4 data ={'product':['apple', 'banana', 'apple', 'orange', 'banana', ' apple']}
5 df = pd.DataFrame(data)
6
7 # Converting 'product' column to categorical
8 df['product']= df['product'].astype('category')
9 print(df['product'])
Result:
1 0 apple
2 1 banana
3 2 apple
4 3 orange
5 4 banana
6 5 apple
7 Name: product, dtype: category
8 Categories (3, object):['apple', 'banana', 'orange']
Thisshowsthatthe‘product’columnisnowoftype category withthreecategories.
OrderCategories
Sometimes,thenaturalorderofcategoriesmatters(e.g.,inordinaldatasuchas‘low’,‘medium’,‘high’).
Pandasallowsyoutosetandordercategories:
1 # Sample DataFrame with unordered categorical data
2 data ={'size':['medium', 'small', 'large', 'small', 'large', 'medium' ]}
3 df = pd.DataFrame(data)
4 df['size']= df['size'].astype('category')
5
6 # Setting and ordering categories
7 df['size']= df['size'].cat.set_categories(['small', 'medium', 'large' ], ordered = True,)
8 print(df['size'])
Result:
1 0 medium
2 1 small
3 2 large
4 3 small
5 4 large
6 5 medium
7 Name: size, dtype: category
8 Categories (3, object):['small' < 'medium' < 'large']
Thisconversionandorderingprocessensuresthatthe‘size’columnisnotonlycategoricalbutalso correctlyorderedfrom‘small’to‘large’.
ThesecategoricaldataoperationsinPandasfacilitatetheeffectivehandlingofnominalandordinal data,enhancingbothperformanceandthecapacityformeaningfuldataanalysis.
AdvancedIndexing
AdvancedindexingtechniquesinPandasenhancedatamanipulationcapabilities,allowingformore sophisticateddataretrievalandmodificationoperations.Thischapterwillfocusonresettingindexes, settingmultipleindexes,andslicingthroughMultiIndexes,whicharecrucialforhandlingcomplex datasetseffectively.
ResetIndex
ResettingtheindexofaDataFramecanbeusefulwhentheindexneedstobetreatedasaregular column,orwhenyouwanttoreverttheindexbacktothedefaultintegerindex:
1 import pandas as pd
2
3 # Sample DataFrame
4 data ={'state':['CA', 'NY', 'FL'],
5 'population':[39500000,19500000,21400000]}
6 df = pd.DataFrame(data)
7 df.set_index('state', inplace = True)
8
9 # Resetting the index
10 reset_df = df.reset_index(drop = True)
11 print(reset_df)
Result:
1 population
2 039500000
3 119500000
4 221400000
Using drop=True removestheoriginalindexandjustkeepsthedatacolumns.
SetMultipleIndexes
Settingmultiplecolumnsasanindexcanprovidepowerfulwaystoorganizeandselectdata,especially usefulinpaneldataorhierarchicaldatasets:
1 # Re-using previous DataFrame without resetting 2 df = pd.DataFrame(data)
3
4 # Setting multiple columns as an index
5 df.set_index(['state', 'population'], inplace = True)
6 print(df)
Result:
1 Empty DataFrame
2 Columns:[]
3 Index:[(CA,39500000),(NY,19500000),(FL,21400000)]
TheDataFramenowusesacompositeindexmadeupof‘state’and‘population’.
MultiIndexSlicing
SlicingdatawithaMultiIndexcanbecomplexbutpowerful.The xs method(cross-section)isoneof themostconvenientwaystoslicemulti-levelindexes:
1 # Assuming the DataFrame with a MultiIndex from the previous example
2 # Adding some values to demonstrate slicing
3 df['data']=[10,20,30]
4
5 # Slicing with xs
6 slice_df = df.xs(key = 'CA', level = 'state')
7 print(slice_df)
Result:
1 data 2 population
3 3950000010
Thisoperationretrievesallrowsassociatedwith‘CA’fromthe‘state’leveloftheindex,showingonly thedataforthepopulationofCalifornia.
AdvancedindexingtechniquesprovidenuancedcontroloverdataaccesspatternsinPandas,enhancing dataanalysisandmanipulationcapabilitiesinawiderangeofapplications.
EfficientComputations
Efficientcomputationiskeyinhandlinglargedatasetsorperformingcomplexoperationsrapidly. Pandasincludesfeaturesthatleverageoptimizedcodepathstospeedupoperationsandreduce memoryusage.Thischapterdiscussesusing eval() forarithmeticoperationsandthe query() methodforfiltering,whicharebothdesignedtoenhanceperformance.
Useofeval()forEfficientOperations
The eval() functioninPandasallowsfortheevaluationofstringexpressionsusingDataFrame columns,whichcanbesignificantlyfaster,especiallyforlargeDataFrames,asitavoidsintermediate datacopies:
1 import pandas as pd
2
3 # Sample DataFrame
4 data ={'col1':[1,2,3],
5 'col2':[4,5,6]}
6 df = pd.DataFrame(data)
7
8 # Using eval() to perform efficient operations
9 df['col3']= df.eval('col1 + col2')
10 print(df)
Result:
1 col1 col2 col3
2 0145
3 1257
4 2369
Thisexampledemonstrateshowtoaddtwocolumnsusing eval(),whichcanbefasterthantraditionalmethodsforlargedatasetsduetooptimizedcomputation.
QueryMethodforFiltering
The query() methodallowsyoutofilterDataFramerowsusinganintuitivequerystring,whichcan bemorereadableandperformantcomparedtotraditionalBooleanindexing:
1 # Sample DataFrame
2 data ={'col1':[10,20,30],
3 'col2':[20,15,25]}
4 df = pd.DataFrame(data)
5
6 # Using query() to filter data
7 filtered_df = df.query('col1 < col2')
8 print(filtered_df)
Result:
1 col1 col2
2 01020
Inthisexample, query() filterstheDataFrameforrowswhere‘col1’islessthan‘col2’.Thismethodcan beespeciallyefficientwhenworkingwithlargeDataFrames,asitutilizesnumexprforfastevaluation ofarrayexpressions.
ThesemethodsenhancePandas’performance,makingitapowerfultoolfordataanalysis,particularlywhenworkingwithlargeorcomplexdatasets.Efficientcomputationsensurethatresourcesare optimallyused,speedingupdataprocessingandanalysis.
AdvancedDataMerging
Combiningdatasetsisacommonrequirementindataanalysis.Beyondbasicmerges,Pandasoffers advancedtechniquessimilartoSQLoperationsandallowsconcatenationalongdifferentaxes.This chapterexploresSQL-likejoinsandvariousconcatenationmethodstoeffectivelycombinemultiple datasets.
SQL-likeJoins
SQL-likejoinsinPandasareachievedusingthe merge function.Thismethodisextremelyversatile, allowingforinner,outer,left,andrightjoins.Here’showtoperformaleftjoin,whichincludesall recordsfromtheleftDataFrameandthematchedrecordsfromtherightDataFrame.Ifthereisno match,theresultis NaN onthesideoftherightDataFrame.
1 import pandas as pd
2
3 # Sample DataFrames
4 data1 ={'col':['A', 'B', 'C'],
5 'col1':[1,2,3]}
6 df1 = pd.DataFrame(data1)
7 data2 ={'col':['B', 'C', 'D'],
8 'col2':[4,5,6]}
9 df2 = pd.DataFrame(data2)
10
11 # Performing a left join
12 left_joined_df = pd.merge(df1, df2, how = 'left', on = 'col')
13 print(left_joined_df)
Result:
1 col col1 col2
2
3
4
Thisresultshowsthatallentriesfrom df1 areincluded,andwheretherearematching‘col’valuesin df2,the‘col2’valuesarealsoincluded.
ConcatenatingAlongaDifferentAxis
Concatenationcanbeperformednotjustvertically(defaultaxis=0),butalsohorizontally(axis=1).This isusefulwhenyouwanttoaddnewcolumnstoanexistingDataFrame:
1 # Concatenating df1 and df2 along axis 1
2 concatenated_df = pd.concat([df1, df2], axis =1)
3 print(concatenated_df)
Result:
1 col col1 col col2
2 0 A 1 B 4
3 1 B 2 C 5
4 2 C 3 D 6
ThisresultdemonstratesthattheDataFramesareconcatenatedside-by-side,aligningbyindex.Note thatbecausethe‘col’valuesdonotmatchbetween df1 and df2,theyappeardisjointed,illustrating theimportanceofindexalignmentinsuchoperations.
Theseadvanceddatamergingtechniquesprovidepowerfultoolsfordataintegration,allowingfor complexmanipulationsandcombinationsofdatasets,muchlikeyouwouldaccomplishusingSQLina databaseenvironment.
DataQualityChecks
Ensuringdataqualityisacriticalstepinanydataanalysisprocess.Dataoftencomeswithissueslike missingvalues,incorrectformats,oroutliers,whichcansignificantlyimpactanalysisresults.Pandas providestoolstoperformthesechecksefficiently.Thischapterfocusesonusingassertionstovalidate dataquality.
AssertStatementforDataValidation
The assert statementinPythonisaneffectivewaytoensurethatcertainconditionsaremetinyour data.Itisusedtoperformsanitychecksandcanhalttheprogramiftheassertionfails,whichishelpful inidentifyingdataqualityissuesearlyinthedataprocessingpipeline.
CheckingforMissingValues
OnecommoncheckistoensurethattherearenomissingvaluesinyourDataFrame.Here’showyou canusean assert statementtoverifythattherearenomissingvaluesacrosstheentireDataFrame:
1 import pandas as pd
2 import numpy as np
3
4 # Sample DataFrame with possible missing values
5 data ={'col1':[1,2, np.nan], 'col2':[4, np.nan,6]}
6 df = pd.DataFrame(data)
7
8 # Assertion to check for missing values
9 try:
10 assert df.notnull().all().all(), "There are missing values in the dataframe"
11 except AssertionError as e:
12 print(e)
IftheDataFramecontainsmissingvalues,theassertionfails,andtheerrormessage“Therearemissing valuesinthedataframe”isprinted.Ifnomissingvaluesarepresent,thescriptcontinueswithout interruption.
Thismethodofdatavalidationhelpsinenforcingthatdatameetstheexpectedqualitystandards beforeproceedingwithfurtheranalysis,thussafeguardingagainstanalysisbasedonfaultydata.
Real-WorldCaseStudies:TitanicDataset
DescriptionoftheData
ThiscodeloadstheTitanicdatasetdirectlyfromapubliclyaccessibleURLintoaPandasDataFrame andprintsthefirstfewentriestogetapreliminaryviewofthedataanditsstructure.The info() functionisthenusedtoprovideaconcisesummaryoftheDataFrame,detailingthenon-nullcountand datatypeforeachcolumn.Thissummaryisinvaluableforquicklyidentifyinganymissingdataand understandingthedatatypespresentineachcolumn,settingthestageforfurtherdatamanipulation andanalysis.
1 import pandas as pd
2
3 # URL of the Titanic dataset CSV from the Seaborn GitHub repository 4 url = "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/ titanic.csv"
5
6 # Load the dataset from the URL directly into a Pandas DataFrame 7 titanic = pd.read_csv(url)
8
9 # Display the first few rows of the dataframe 10 print(titanic.head())
11
12 # Show a summary of the dataframe 13 print(titanic.info())
ExploratoryDataAnalysis(EDA)
Thissectiongeneratesstatisticalsummariesfornumericalcolumnsusingdescribe(),whichprovidesa quickoverviewofcentraltendencies,dispersion,andshapeofthedataset’sdistribution.Histograms andboxplotsareplottedtovisualizethedistributionofanddetectoutliersinnumericaldata.The value_counts() methodgivesacountofuniquevaluesforcategoricalvariables,whichhelpsin understandingthedistributionofcategoricaldata.The pairplot() functionfromSeabornshows pairwiserelationshipsinthedataset,coloredbythe‘Survived’columntoseehowvariablescorrelate withsurvival.
1 import matplotlib.pyplot as plt
2 import seaborn as sns
3
4 # Summary statistics for numeric columns
5 print(titanic.describe())
6
7 # Distribution of key categorical features
8 print(titanic['Survived'].value_counts())
9 print(titanic['Pclass'].value_counts())
10 print(titanic['Sex'].value_counts())
11
12 # Histograms for numerical columns
13 titanic.hist(bins=10, figsize=(10,7))
14 plt.show()
15
16 # Box plots to check for outliers
17 titanic.boxplot(column=['Age', 'Fare'])
18 plt.show()
19
20 # Pairplot to visualize the relationships between numerical variables
21 sns.pairplot(titanic.dropna(), hue = 'Survived')
22 plt.show()
DataCleaningandPreparation
Thiscodechecksformissingvaluesandhandlesthembyfillingwithmedianvaluesfor Age andthe modefor Embarked.Itconvertscategoricaldata(Sex)intoanumericalformatsuitableformodeling. Columnsthatarenotnecessaryfortheanalysisaredroppedtosimplifythedataset.
1 # Checking for missing values
2 print(titanic.isnull().sum())
3
4 # Filling missing values
5 titanic['Age'].fillna(titanic['Age'].median(), inplace = True)
6 titanic['Embarked'].fillna(titanic['Embarked'].mode()[0], inplace = True)
7
8 # Converting categorical columns to numeric
9 titanic['Sex']= titanic['Sex'].map({'male':0, 'female':1})
10
11 # Dropping unnecessary columns
12 titanic.drop(['Cabin', 'Ticket', 'Name'], axis =1, inplace = True)
SurvivalAnalysis
Thissegmentexaminessurvivalratesby class and sex.Ituses groupby() tosegmentdatafollowed bymeancalculationstoanalyzesurvivalrates.Resultsarevisualizedusingbarplotstoprovideaclear visualcomparisonofsurvivalratesacrossdifferentgroups.
1 # Group data by survival and class
2 survival_rate = titanic.groupby('Pclass')['Survived'].mean()
3 print(survival_rate)
4
5 # Survival rate by sex
6 survival_sex = titanic.groupby('Sex')['Survived'].mean()
7 print(survival_sex)
8
9 # Visualization of survival rates
10 sns.barplot(x = 'Pclass', y = 'Survived', data=titanic)
11 plt.title('Survival Rates by Class')
12 plt.show()
13
14 sns.barplot(x = 'Sex', y = 'Survived', data=titanic)
15 plt.title('Survival Rates by Sex')
16 plt.show()
ConclusionsandApplications
Thefinalsectionsummarizesthekeyfindingsfromtheanalysis,highlightingtheinfluenceoffactors like sex and class onsurvivalrates.Italsodiscusseshowthetechniquesappliedcanbeusedwithother datasetstoderiveinsightsandsupportdecision-makingprocesses.
1 # Summary of findings
2 print("Key Findings from the Titanic Dataset:")
3 print("1. Higher survival rates were observed among females and upperclass passengers.")
4 print("2. Age and fare prices also appeared to influence survival chances.")
5
6 # Discussion on applications
7 print("These analysis techniques can be applied to other datasets to uncover underlying patterns and improve decision-making.")
AdditionalResources
Providesadditionalresourcesforreaderstoexploremoreabout Pandas anddataanalysis.Thisincludes linkstoofficialdocumentationandthe Kaggle competitionpagefortheTitanicdataset,whichoffersa platformforpracticingandimprovingdataanalysisskills.
Thiscomprehensivechapteroutlineandcodeexplanationsgivereadersathoroughunderstandingof dataanalysisworkflowsusing Pandas,fromdataloadingtocleaning,analysis,anddrawingconclusions.
1 # This section would list URLs or references to further reading
2 print("For more detailed tutorials on Pandas and data analysis, visit:" )
3 print("- The official Pandas documentation: https://pandas.pydata.org/ pandas-docs/stable/")
4 print("- Kaggle's Titanic Competition for more explorations: https:// www.kaggle.com/c/titanic")
Thischapterprovidesathoroughwalk-throughusingtheTitanicdatasettodemonstratevariousdata handlingandanalysistechniqueswith Pandas,offeringpracticalinsightsandmethodsthatcanbe appliedtoawiderangeofdataanalysisscenarios.