Search

Text Color

Margin Size

Font Type

Enable Dyslexic Font

15.3: Pandas

Last updated

Mar 30, 2025
Save as PDF
- 15.2: NumPy
- 15.4: Exploratory Data Analysis

$\newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} }$

$\newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}}$

$\newcommand{\id}{\mathrm{id}}$ $\newcommand{\Span}{\mathrm{span}}$

( \newcommand{\kernel}{\mathrm{null}\,}\) $\newcommand{\range}{\mathrm{range}\,}$

$\newcommand{\RealPart}{\mathrm{Re}}$ $\newcommand{\ImaginaryPart}{\mathrm{Im}}$

$\newcommand{\Argument}{\mathrm{Arg}}$ $\newcommand{\norm}[1]{\| #1 \|}$

$\newcommand{\inner}[2]{\langle #1, #2 \rangle}$

$\newcommand{\Span}{\mathrm{span}}$

$\newcommand{\id}{\mathrm{id}}$

$\newcommand{\Span}{\mathrm{span}}$

$\newcommand{\kernel}{\mathrm{null}\,}$

$\newcommand{\range}{\mathrm{range}\,}$

$\newcommand{\RealPart}{\mathrm{Re}}$

$\newcommand{\ImaginaryPart}{\mathrm{Im}}$

$\newcommand{\Argument}{\mathrm{Arg}}$

$\newcommand{\norm}[1]{\| #1 \|}$

$\newcommand{\inner}[2]{\langle #1, #2 \rangle}$

$\newcommand{\Span}{\mathrm{span}}$ $\newcommand{\AA}{\unicode[.8,0]{x212B}}$

$\newcommand{\vectorA}[1]{\vec{#1}} % arrow$

$\newcommand{\vectorAt}[1]{\vec{\text{#1}}} % arrow$

$\newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} }$

$\newcommand{\vectorC}[1]{\textbf{#1}}$

$\newcommand{\vectorD}[1]{\overrightarrow{#1}}$

$\newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}}$

$\newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}}$

$\newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} }$

$\newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}}$

$\newcommand{\avec}{\mathbf a}$

$\newcommand{\bvec}{\mathbf b}$

$\newcommand{\cvec}{\mathbf c}$

$\newcommand{\dvec}{\mathbf d}$

$\newcommand{\dtil}{\widetilde{\mathbf d}}$

$\newcommand{\evec}{\mathbf e}$

$\newcommand{\fvec}{\mathbf f}$

$\newcommand{\nvec}{\mathbf n}$

$\newcommand{\pvec}{\mathbf p}$

$\newcommand{\qvec}{\mathbf q}$

$\newcommand{\svec}{\mathbf s}$

$\newcommand{\tvec}{\mathbf t}$

$\newcommand{\uvec}{\mathbf u}$

$\newcommand{\vvec}{\mathbf v}$

$\newcommand{\wvec}{\mathbf w}$

$\newcommand{\xvec}{\mathbf x}$

$\newcommand{\yvec}{\mathbf y}$

$\newcommand{\zvec}{\mathbf z}$

$\newcommand{\rvec}{\mathbf r}$

$\newcommand{\mvec}{\mathbf m}$

$\newcommand{\zerovec}{\mathbf 0}$

$\newcommand{\onevec}{\mathbf 1}$

$\newcommand{\real}{\mathbb R}$

$\newcommand{\twovec}[2]{\left[\begin{array}{r}#1 \\ #2 \end{array}\right]}$

$\newcommand{\ctwovec}[2]{\left[\begin{array}{c}#1 \\ #2 \end{array}\right]}$

$\newcommand{\threevec}[3]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \end{array}\right]}$

$\newcommand{\cthreevec}[3]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \end{array}\right]}$

$\newcommand{\fourvec}[4]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}$

$\newcommand{\cfourvec}[4]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}$

$\newcommand{\fivevec}[5]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}$

$\newcommand{\cfivevec}[5]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}$

$\newcommand{\mattwo}[4]{\left[\begin{array}{rr}#1 \amp #2 \\ #3 \amp #4 \\ \end{array}\right]}$

$\newcommand{\laspan}[1]{\text{Span}\{#1\}}$

$\newcommand{\bcal}{\cal B}$

$\newcommand{\ccal}{\cal C}$

$\newcommand{\scal}{\cal S}$

$\newcommand{\wcal}{\cal W}$

$\newcommand{\ecal}{\cal E}$

$\newcommand{\coords}[2]{\left\{#1\right\}_{#2}}$

$\newcommand{\gray}[1]{\color{gray}{#1}}$

$\newcommand{\lgray}[1]{\color{lightgray}{#1}}$

$\newcommand{\rank}{\operatorname{rank}}$

$\newcommand{\row}{\text{Row}}$

$\newcommand{\col}{\text{Col}}$

$\renewcommand{\row}{\text{Row}}$

$\newcommand{\nul}{\text{Nul}}$

$\newcommand{\var}{\text{Var}}$

$\newcommand{\corr}{\text{corr}}$

$\newcommand{\len}[1]{\left|#1\right|}$

$\newcommand{\bbar}{\overline{\bvec}}$

$\newcommand{\bhat}{\widehat{\bvec}}$

$\newcommand{\bperp}{\bvec^\perp}$

$\newcommand{\xhat}{\widehat{\xvec}}$

$\newcommand{\vhat}{\widehat{\vvec}}$

$\newcommand{\uhat}{\widehat{\uvec}}$

$\newcommand{\what}{\widehat{\wvec}}$

$\newcommand{\Sighat}{\widehat{\Sigma}}$

$\newcommand{\lt}{<}$

$\newcommand{\gt}{>}$

$\newcommand{\amp}{&}$

$\definecolor{fillinmathshade}{gray}{0.9}$

Learning Objectives

By the end of this section you should be able to

Describe the Pandas library.
Create a DataFrame and a Series object.
Choose appropriate Pandas functions to gain insight from heterogeneous data.

Pandas library

Pandas is an open-source Python library used for data cleaning, processing, and analysis. Pandas provides data structures and data analysis tools to analyze structured data efficiently. The name "Pandas" is derived from the term "panel data," which refers to multidimensional structured datasets. Key features of Pandas include:

Data structure: Pandas implements two main data structures:
- Series: A Series is a one-dimensional labeled array.
- DataFrame: A DataFrame is a two-dimensional labeled data structure that consists of columns and rows. A DataFrame can be thought of as a spreadsheet-like data structure where each column represents a Series. DataFrame is a heterogeneous data structure where each column can have a different data type.
Data processing functionality: Pandas provides various functionalities for data processing, such as data selection, filtering, slicing, sorting, merging, joining, and reshaping.
Integration with other libraries: Pandas integrates well with other Python libraries, such as NumPy. The integration capability allows for data exchange between different data analysis and visualization tools.

The conventional alias for importing Pandas is pd. In other words, Pandas is imported as import pandas as pd. Examples of DataFrame and Series objects are shown below.

DataFrame example

Series example

	Name	Age	City
0	Emma	15	Dubai
1	Gireeja	28	London
2	Sophia	22	San Jose

0	Emma
1	Gireeja
2	Sophia
dtype:	object

Table 15.1

Data input and output

A DataFrame can be created from a dictionary, list, NumPy array, or a CSV file. Column names and column data types can be specified at the time of DataFrame instantiation.

Description Example Output Explanation

DataFrame from a dictionary

import pandas as pd

# Create a dictionary of columns
data = {
  "Name": ["Emma", "Gireeja", "Sophia"],
  "Age": [15, 28, 22],
  "City": ["Dubai", "London", "San Jose"]
}

# Create a DataFrame from the dictionary
df = pd.DataFrame(data)

# Display the DataFrame
df

	Name	Age	City
0	Emma	15	Dubai
1	Gireeja	28	London
2	Sophia	22	San Jose

The pd.DataFrame() function takes in a dictionary and converts it into a DataFrame. Dictionary keys will be column labels and values are stored in respective columns.

DataFrame from a list

import pandas as pd

# Create a list of rows
data = [
  ["Emma", 15, "Dubai"],
  ["Gireeja", 28, "London"],
  ["Sophia", 22, "San Jose"]
]

# Define column labels
columns = ["Name", "Age", "City"]

# Create a DataFrame from list using column labels
df = pd.DataFrame(data, columns=columns)

# Display the DataFrame
df

	Name	Age	City
0	Emma	15	Dubai
1	Gireeja	28	London
2	Sophia	22	San Jose

The pd.DataFrame() function takes in a list containing the records in different rows of a DataFrame, along with a list of column labels, and creates a DataFrame with the given rows and column labels.

DataFrame from a NumPy array

import numpy as np
import pandas as pd

# Create a NumPy array
data = np.array([
  [1, 0, 0],
  [0, 1, 0],
  [2, 3, 4]
])

# Define column labels
columns = ["A", "B", "C"]

# Create a DataFrame from the NumPy array
df = pd.DataFrame(data, columns=columns)

# Display the DataFrame
df

	A	B	C
0	1	0	0
1	0	1	0
2	2	3	4

A NumPy array, along with column labels, are passed to the pd.DataFrame() function to create a DataFrame object.

DataFrame from a CSV file

import pandas as pd

# Read the CSV file into a DataFrame
df = pd.read_csv("data.csv")

# Display the DataFrame
df

The content of the CSV file will be printed in a tabular format.

The pd.read_csv() function reads a CSV file into a DataFrame and organizes the content in a tabular format.

DataFrame from a Excel file

import pandas as pd

# Read the Excel file into a DataFrame
df = pd.read_excel("data.xlsx")

# Display the DataFrame
df

The content of the Excel file will be printed in a tabular format.

The pd.read_excel() function reads an Excel file into a DataFrame and organizes the content in a tabular format.

Table 15.2 DataFrame creation.

Concepts in Practice: Pandas basics

Which of the following is a Pandas data structure?

Series

dictionary

list

A DataFrame object can be considered as a collection of Series objects.

true
false

What are the benefits of Pandas over NumPy?

Pandas provides integration with other libraries.
Pandas supports heterogeneous data whereas NumPy supports homogenous numerical data.
Pandas supports both one-dimensional and two-dimensional data structures.

Pandas for data manipulation and analysis

The Pandas library provides functions and techniques to explore, manipulate, and gain insights from the data. Key DataFrame functions that analyze this code are described in the following table.

    import pandas as pd
    import numpy as np
    
    # Create a sample DataFrame
    days = {
      'Season': ['Summer', 'Summer', 'Fall', 'Winter', 'Fall', 'Winter'],
      'Month': ['July', 'June', 'September', 'January', 'October', 'February'],
      'Month-day': [1, 12, 3, 7, 20, 28],
      'Year': [2000, 1990, 2020, 1998, 2001, 2022]
    }
    df = pd.DataFrame(days)

	Season	Month	Month-day	Year
0	Summer	July	1	2000
1	Summer	June	12	1990
2	Fall	September	3	2020
3	Winter	January	7	1998
4	Fall	October	20	2001
5	Winter	February	28	2022

Table 15.3

Function name Explanation Example Output

head(n)

Returns the first n rows. If a value is not passed, the first 5 rows will be shown.

df.head(4)

	Season	Month	Month-day	Year
0	Summer	July	1	2000
1	Summer	June	12	1990
2	Fall	September	3	2020
3	Winter	January	7	1998

tail(n)

Returns the last n rows. If a value is not passed, the last 5 rows will be shown.

df.tail(3)

	Season	Month	Month-day	Year
3	Winter	January	7	1998
4	Fall	October	20	2001
5	Winter	February	28	2022

info()

Provides a summary of the DataFrame, including the column names, data types, and the number of non-null values. The function also returns the DataFrame's memory usage.

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 6 entries, 0 to 5
Data columns (total 4 columns):
 #    Column        Non-Null Count    Dtype
---   ------        --------------    -----
 0    Season        6 non-null        object
 1    Month         6 non-null        object
 2    Month-day     6 non-null        int64
 3    Year          6 non-null        int64
dtypes: int64(2), object(2)
memory usage: 320.0+ bytes

describe()

Generates the column count, mean, standard deviation, minimum, maximum, and quartiles.

df.describe()

	Month-day	Year
count	6.000000	6.000000
mean	11.833333	2005.166667
std	10.457852	12.875040
min	1.000000	1990.000000
25%	4.000000	1998.500000
50%	9.500000	2000.500000
75%	18.000000	2015.250000
max	28.000000	2022.000000

value_counts()

Counts the occurrences of unique values in a column when a column is passed as an argument and presents them in descending order.

df.value_counts \('Season')

Season
Fall  2
Summer  2
Winter  2
dtype: int64

unique()

Returns an array of unique values in a column when called on a column.

df['Season'] \.unique()

['Summer' 'Fall' 'Winter']

Table 15.4 DataFrame functions.

Concepts in Practice: DataFrame operations

Which of the following returns the top five rows of a DataFrame?

df.head()

df.head(5)

both

What does the unique() function do in a DataFrame when applied to a column?

returns the number of unique columns
returns the number of unique values in the given column
returns the unique values in the given column

Which function generates statistical information of columns with numerical data types?

describe()
info()
unique()

Exploring further

Please refer to the Pandas user guide for more information about the Pandas library.

Pandas User Guide

Programming practice with Google

Use the Google Colaboratory document below to practice Pandas functionalities to extract insights from a dataset.

Google Colaboratory document