Section 1

1 Before you start

This is not graded, and it does not count towards anything. Answer every question even if you are only guessing — having a go at something before you have been taught it is one of the best ways to remember it afterwards. The lesson ends with another short check, and there you will be shown the correct answer for anything you missed. Choose an option for every question and confidence rating, then click Submit answers.

Pre-test

You write df = pd.DataFrame({"a": [1, 2, 3], "b": [10, 20, 30]}). What does df["a"] return?

Pre-test

Which of these reads a tab-separated file into a DataFrame?

Pre-test

A column of strings like "2024-01-15" is in your DataFrame. After running df["d"] = pd.to_datetime(df["d"]), what dtype does d have?

Pre-test

Which call reports the dtype and the number of non-null values for every column of df?

Pre-test

df["clinic"].value_counts() returns which kind of object?

Pre-test

A DataFrame is indexed by patient_id ("P001", "P002", "P003", ...). Which call returns the row for patient "P002"?

Pre-test

You want every row in df where age > 50 AND bmi > 25. Which expression is correct?

Pre-test

You want every row whose clinic is one of "A", "B", or "C". Which expression works?

Pre-confidence

I can read a CSV or Excel file into a DataFrame and use head, info, and describe to get a quick feel for what is in it.

Not at all confident
Fully confident
Pre-confidence

I can filter a DataFrame on one or more boolean conditions and pull out the rows that match.

Not at all confident
Fully confident
Pre-confidence

I can parse a column of date strings with pd.to_datetime and pull out the year or month with .dt.

Not at all confident
Fully confident
Section 2

2 Introduction

Pandas is a package for data analysis in Python. It stores a table of data in an object called a DataFrame. In precision medicine, a table can hold patient records, laboratory results, gene counts or survey responses.

This lesson is the first of three on pandas. It has three parts. First, you load data from a file. Next, you check what the data holds. Then you pick out the rows and columns you need.

Every pandas operation falls into one of three intents, getting data in, understanding what you have, and getting what you want
Every pandas operation falls into one of three intents, getting data in, understanding what you have, and getting what you want

Try every example in the Python Scratchpad on the right.

Section 3

3 Introduction to Pandas DataFrames

A DataFrame is a table with named columns, numbered rows and a value in every cell. It is the main object in pandas.

The usual way to import the package is import pandas as pd. This gives pandas the short name pd. The function pd.DataFrame builds a table from a dictionary. Each dictionary key becomes a column name. The list beside the key fills the cells of that column:

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
print(labs)
# patient ward potassium
# 0 Tan 5A 4.6
# 1 Lim 7B 3.9
# 2 Kumar 5A 5.8
# 3 Wong 7B 5.2
# 4 Ng 5A 6.1

The numbers 0 to 4 down the left side are the index. Pandas adds them when you do not give an index. Many people name their table df, short for DataFrame. Here the table is called labs. This is because it holds laboratory results. Each potassium result is in mmol/L.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

data = {
    "patient_id": ["P001", "P002", "P003"],
    "age": [54, 61, 47],
    "bmi": [27.3, 24.1, 31.2],
}
df = pd.DataFrame(data)
print(df)
A pandas DataFrame with each key forming a column and pandas adding the row index automatically.
A pandas DataFrame with each key forming a column and pandas adding the row index automatically.

One column of a DataFrame is a Series. A Series is a list of values with an index attached. You take a Series out of a DataFrame with the column name in square brackets:

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
levels = labs["potassium"]
print(levels)
# 0 4.6
# 1 3.9
# 2 5.8
# 3 5.2
# 4 6.1
# Name: potassium, dtype: float64
print(levels.mean())  # 5.12

The Series keeps the index 0 to 4 beside its values. The method .mean() gives the average of the five results, with no loop. The methods .sum(), .median(), .std(), .min() and .max() work in the same way.

A column of a pandas DataFrame is a Series, and the Series knows its own aggregation methods such as .mean(), .sum(), and .max().
A column of a pandas DataFrame is a Series, and the Series knows its own aggregation methods such as .mean(), .sum(), and .max().
Section 4

4 Reading CSV and Excel files with Pandas

Real data often comes as a CSV file from a colleague, an Excel workbook from a clinical study or a TSV file from a sequencing pipeline. A CSV file separates its values with commas. A TSV file separates them with tabs. Pandas can load all of these.

The function pd.read_csv loads a CSV file into a DataFrame. It takes the column names from the first line. It also works out the type of each column.

The Python Scratchpad already holds several data files, including patients.csv and patients.tsv.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.read_csv("patients.csv")
print(df)

Several arguments of pd.read_csv are useful from the start. The argument sep tells pandas which character separates the values. Python writes the tab character as "\t". The next example first writes a small file with tabs between its values. Then it loads the file twice:

import pandas as pd

with open("labs.tsv", "w") as f:
    print("patient\tward\tpotassium", file=f)
    print("Tan\t5A\t4.6", file=f)
    print("Lim\t7B\t3.9", file=f)
    print("Kumar\t5A\t5.8", file=f)

with_sep = pd.read_csv("labs.tsv", sep="\t")
print(with_sep.shape)  # (3, 3)

without_sep = pd.read_csv("labs.tsv")
print(without_sep.shape)  # (3, 1)

The first call has sep="\t", so the table has three columns. The second call has no sep, so pandas looks for commas. The file has no commas, so each whole line stays in one column. Four more arguments are useful:

  • header=None says that the file has no header row. Pandas then makes up column names.
  • index_col="patient_id" uses that column as the row index, in place of 0, 1, 2.
  • na_values=["NA", "missing", -999] lists the entries that mean a missing value.
  • nrows=100 reads only the first 100 rows. This is a quick way to look at a huge file.
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.read_csv(
    "patients.tsv",
    sep="\t",
    index_col="patient_id",
    na_values=["NA", "missing", -999],
    nrows=100,
)

print(df)

Excel files use a similar function, pd.read_excel. A workbook can hold several sheets, so the function has an extra argument, sheet_name. It takes the name of a sheet as a string, or the position of a sheet as a whole number. The first sheet is 0.

pd.read_csv and pd.read_excel both load tabular files into a DataFrame, with pd.read_excel taking an extra sheet_name argument to pick a sheet from a workbook.
pd.read_csv and pd.read_excel both load tabular files into a DataFrame, with pd.read_excel taking an extra sheet_name argument to pick a sheet from a workbook.

Writing a DataFrame to a file is the mirror image of reading it. The method .to_csv writes a CSV file. The method .to_excel writes an Excel file. Add index=False to both. Without it, pandas writes the index as an extra first column with no name. The next example saves a table twice and loads each file again:

import pandas as pd

results = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar"],
    "potassium": [4.6, 3.9, 5.8],
})
results.to_csv("results_with_index.csv")
print(pd.read_csv("results_with_index.csv"))
# Unnamed: 0 patient potassium
# 0 0 Tan 4.6
# 1 1 Lim 3.9
# 2 2 Kumar 5.8

results.to_csv("results.csv", index=False)
print(pd.read_csv("results.csv"))
# patient potassium
# 0 Tan 4.6
# 1 Lim 3.9
# 2 Kumar 5.8

The first file has an extra column, which pandas names Unnamed: 0. This is because the file gives that column no name. The second file has only the two columns of the table.

If pandas reports a FileNotFoundError, the file is not in the folder where Python is running. Move the file there, or give the full path to the file.

Data is also published at a web address. Outside this Scratchpad, you load it with pd.read_csv(url), with the web address in place of the path to a file.

Section 5

5 Working with dates: pd.to_datetime

A column of dates read from a file is usually text. Pandas stores text in the type object. Pandas does not know that "2026-03-04" is a date until you tell it. Until then, the .dt accessor does not work:

import pandas as pd

collected = pd.Series(["2026-03-04", "2026-03-11", "2026-04-02"])
print(collected.dt.month_name())
# AttributeError: Can only use .dt accessor with datetimelike values

The function pd.to_datetime converts a column of text into dates. The attribute dtypes shows the type of each column. After the conversion, the type of the column collected is datetime64[ns]. Then the .dt accessor gives one part of each date, such as the year or the name of the month:

import pandas as pd

visits = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar"],
    "collected": ["2026-03-04", "2026-03-11", "2026-04-02"],
})
print(visits.dtypes)
# patient object
# collected object
# dtype: object

visits["collected"] = pd.to_datetime(visits["collected"])
print(visits.dtypes)
# patient object
# collected datetime64[ns]
# dtype: object
print(visits["collected"].dt.year)
# 0 2026
# 1 2026
# 2 2026
# Name: collected, dtype: int32
print(visits["collected"].dt.month_name())
# 0 March
# 1 March
# 2 April
# Name: collected, dtype: object
Parsing a string date column in pandas into proper datetimes with pd.to_datetime, then using the .dt accessor to extract the year.
Parsing a string date column in pandas into proper datetimes with pd.to_datetime, then using the .dt accessor to extract the year.
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003"],
    "visit": ["2024-01-15", "2024-03-02", "2024-06-21"],
})
df["visit"] = pd.to_datetime(df["visit"])
print(df.dtypes)
print(df["visit"].dt.year)
print(df["visit"].dt.month_name())

Dates written with slashes need care. In Singapore, the day usually comes first, so 04/03/2026 is the 4th of March. Pandas guesses that the month comes first. The second line below gives the layout with the argument format:

import pandas as pd

print(pd.to_datetime("04/03/2026"))  # 2026-04-03 00:00:00
print(pd.to_datetime("04/03/2026", format="%d/%m/%Y"))  # 2026-03-04 00:00:00

The first line reads 04/03/2026 as the 3rd of April. The second line reads it as the 4th of March, which is what was written. Two keyword arguments make pd.to_datetime reliable:

  • format="%d/%m/%Y" states the layout of the dates. Here %d is the day, %m is the month and %Y is the year with four digits.
  • errors="coerce" turns each entry that pandas cannot read into NaT. The name NaT stands for "not a time". It is the date version of NaN. Without errors="coerce", one such entry stops the whole conversion with a ValueError. Afterwards, .isna().sum() counts the NaT entries.
import pandas as pd

visits = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar"],
    "collected": ["04/03/2026", "11/03/2026", "pending"],
})
visits["collected"] = pd.to_datetime(visits["collected"], format="%d/%m/%Y", errors="coerce")
print(visits["collected"])
# 0 2026-03-04
# 1 2026-03-11
# 2 NaT
# Name: collected, dtype: datetime64[ns]
print(visits["collected"].isna().sum())  # 1
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003", "P004"],
    "visit": ["15/01/2024", "02/03/2024", "not a date", "21/06/2024"],
})
df["visit"] = pd.to_datetime(df["visit"], format="%d/%m/%Y", errors="coerce")
print(df)
print(df["visit"].isna().sum())

Always check that count after you convert a date column. If the count is larger than you expected, the cause is often a wrong format string, and not messy data.

Section 6

6 Inspecting dataframes (head, info, describe)

When you load a new file, check what is in it first. Printing the whole table does not help. This is because a real table can have thousands of rows. You cannot read them all.

Run these five quick checks on every new file, in this order. A problem then shows early, before it spreads through your analysis:

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
print(labs.head(3))
# patient ward potassium
# 0 Tan 5A 4.6
# 1 Lim 7B 3.9
# 2 Kumar 5A 5.8
print(labs.tail(2))
# patient ward potassium
# 3 Wong 7B 5.2
# 4 Ng 5A 6.1
print(labs.shape)  # (5, 3)
print(labs.columns)  # Index(['patient', 'ward', 'potassium'], dtype='object')
print(labs.dtypes)
# patient object
# ward object
# potassium float64
# dtype: object

Here is what each check does:

  • labs.head() shows the first five rows, and labs.head(3) shows the first three.
  • labs.tail() shows the last five rows. This helps you find a row of totals at the bottom of a spreadsheet.
  • labs.shape gives the number of rows and the number of columns, as a tuple. It has no round brackets. This is because it is an attribute and not a method.
  • labs.columns lists the column names.
  • labs.dtypes shows the type of each column.
Six pandas DataFrame inspection tools demonstrated on a sample patient dataset.
Six pandas DataFrame inspection tools demonstrated on a sample patient dataset.
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.read_csv("patients.csv")
print(df.shape)
print(df.columns)
print(df.dtypes)
print(df.head())

The method info puts these checks together. For each column, it shows the type and the number of values that are not missing. It helps when a file loads badly. For example, a column of numbers shows up as object when some text gets into it:

import pandas as pd

messy = pd.DataFrame({"potassium": [4.6, "5.8 mmol/L", 3.9]})
print(messy.dtypes)
# potassium object
# dtype: object

The entry "5.8 mmol/L" is text. So the whole column gets the type object. A calculation on that column, such as its mean, then fails.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.read_csv("patients.csv")
df.info()
df.info() reports the dtype and non-null count for every column, so when a column you expected to be numeric appears as object, you know a non-numeric value, like a stray unit, has slipped in and needs cleaning.
df.info() reports the dtype and non-null count for every column, so when a column you expected to be numeric appears as object, you know a non-numeric value, like a stray unit, has slipped in and needs cleaning.

The method describe gives summary values for every numeric column: the count, the mean, the standard deviation, the smallest value, the 25th, 50th and 75th percentiles, and the largest value. These values help you find a value that cannot be right, such as a negative age, a heart rate of zero or a BMI of 9000.

For the text columns, add include="object". Then you get the number of values, the number of different values, the most frequent value and how often it appears:

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
print(labs.describe())
# potassium
# count 5.000000
# mean 5.120000
# std 0.892749
# min 3.900000
# 25% 4.600000
# 50% 5.200000
# 75% 5.800000
# max 6.100000
print(labs.describe(include="object"))
# patient ward
# count 5 5
# unique 5 2
# top Tan 5A
# freq 1 3

For the std row, pandas divides by the count minus one. The NumPy method std divides by the count. So NumPy gives 0.798 for the same five results.

df.describe() reveals impossible values like a negative age, a zero heart rate, and a BMI of 9000, with the categorical variant of the same call highlighting the most frequent category and its count.
df.describe() reveals impossible values like a negative age, a zero heart rate, and a BMI of 9000, with the categorical variant of the same call highlighting the most frequent category and its count.
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({
    "age": [54, 61, 47, 72, 38],
    "bmi": [27.3, 24.1, 31.2, 22.8, 29.5],
})
print(df.shape)

A first look at a new file is often four lines. First, you read the file. Then you print the shape, call info and print the first rows with head.

First-pass workflow with pandas to understand the dataset
First-pass workflow with pandas to understand the dataset
Section 7

7 Counting categories with value_counts

Some columns hold labels, such as smoking status, treatment arm or blood type. For these columns, the first question is how many rows have each label.

The method head cannot answer this. This is because the first five rows may all hold the same label. You need a count over the whole column.

The method value_counts gives back a Series with one count for each different label. The counts are sorted from largest to smallest:

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
print(labs["ward"].value_counts())
# ward
# 5A 3
# 7B 2
# Name: count, dtype: int64

So the label with the largest count comes first. Rare labels come last, where they stand out. A rare label is often interesting, or it is a typing error.

value_counts() walks every row of a categorical column and increments a running tally for each category, then returns those tallies sorted high to low so common categories sit at the top and rare values or data-entry typos surface at the bottom.
value_counts() walks every row of a categorical column and increments a running tally for each category, then returns those tallies sorted high to low so common categories sit at the top and rare values or data-entry typos surface at the bottom.

Two keyword arguments are useful from the start.

  • normalize=True gives proportions in place of counts. This helps you compare groups of different sizes.
  • dropna=False counts missing values as a label of their own. This helps when you check the quality of the data. In the example below, None stands for a missing value.
import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
print(labs["ward"].value_counts(normalize=True))
# ward
# 5A 0.6
# 7B 0.4
# Name: proportion, dtype: float64

gaps = pd.DataFrame({"ward": ["5A", "7B", "5A", None, "5A"]})
print(gaps["ward"].value_counts(dropna=False))
# ward
# 5A 3
# 7B 1
# None 1
# Name: count, dtype: int64
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({"clinic": ["A", "B", "A", "C", "B", "B", "A"]})
print(df["clinic"].value_counts())
print(df["clinic"].value_counts(normalize=True))
How pandas value_counts() behaves with normalize=True and dropna=False,
How pandas value_counts() behaves with normalize=True and dropna=False,
Section 8

8 Data selection and filtering

You rarely want the whole table. You want some columns or some rows. Pandas gives you short forms to pick them.

Section 8.1

8.1 Picking columns

The number of square brackets decides what you get back.

One pair of brackets with one column name gives a Series. Two pairs give a smaller DataFrame.

The outer pair asks pandas for a selection. The inner pair is an ordinary Python list of the column names you want. So a list with one name still gives a DataFrame with one column:

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
print(type(labs["potassium"]))  # <class 'pandas.core.series.Series'>
print(type(labs[["potassium"]]))  # <class 'pandas.core.frame.DataFrame'>
print(labs[["patient", "potassium"]])
# patient potassium
# 0 Tan 4.6
# 1 Lim 3.9
# 2 Kumar 5.8
# 3 Wong 5.2
# 4 Ng 6.1
print(labs["patient", "potassium"])
# KeyError: ('patient', 'potassium')

The last line has no inner brackets. So pandas reads the two names as one tuple. No column has that name.

A single string returns a Series while a list of names returns a DataFrame.
A single string returns a Series while a list of names returns a DataFrame.
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003"],
    "age": [54, 61, 47],
    "bmi": [27.3, 24.1, 31.2],
})
print(df["age"])
print(df[["patient_id", "bmi"]])

A real table can have forty columns. You do not want to type forty names. The attribute columns is a sequence of strings. You can build the list of names with a list comprehension and a string method. You met them in the lessons Control Flow II and String Manipulation:

  • [col for col in df.columns if col.startswith("gluc")] keeps the names that start with gluc.
  • [col for col in df.columns if "date" in col] keeps the names that contain date.
  • [col for col in df.columns if not col.startswith("patient")] drops the names that start with patient.
import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
picked = [name for name in labs.columns if name.startswith("p")]
print(picked)  # ['patient', 'potassium']
print(labs[picked])
# patient potassium
# 0 Tan 4.6
# 1 Lim 3.9
# 2 Kumar 5.8
# 3 Wong 5.2
# 4 Ng 6.1
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.read_csv("patients.csv")
glucose_cols = [col for col in df.columns if col.startswith("gluc")]
date_cols = [col for col in df.columns if "date" in col]
front = ["patient_id", "age"] + date_cols
print(glucose_cols)
print(date_cols)
print(df[front])

The line that makes front uses a trick. The + joins two lists. The list of two names comes first, and date_cols follows it.

Section 8.2

8.2 Picking rows

A row has a label and a position. The label is its entry in the index. The position is its number, counted from zero. A table gets an index of names from the argument index_col of pd.read_csv, or from the argument index of pd.DataFrame. The example below uses index. Two tools pick a row:

  • labs.loc["Kumar"] gives the row whose label is Kumar.
  • labs.iloc[2] gives the row at position 2, which is the third row.
import pandas as pd

labs = pd.DataFrame(
    {
        "ward": ["5A", "7B", "5A", "7B", "5A"],
        "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
    },
    index=["Tan", "Lim", "Kumar", "Wong", "Ng"],
)
print(labs.loc["Kumar"])
# ward 5A
# potassium 5.8
# Name: Kumar, dtype: object
print(labs.iloc[2])
# ward 5A
# potassium 5.8
# Name: Kumar, dtype: object

With the default index 0, 1, 2 and so on, the label and position of a row are the same number. The two tools look alike. With an index of names, they differ.

To remember the difference: .loc works by name. .iloc works by number. A name inside .iloc raises a TypeError. The message is Cannot index by location index with a non-integer key.

.loc selects rows by their index label while .iloc selects by integer position. When item not found, a KeyError is presented.
.loc selects rows by their index label while .iloc selects by integer position. When item not found, a KeyError is presented.
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame(
    {"age": [54, 61, 47]},
    index=["P001", "P002", "P003"],
)

print(df.loc["P002"])
print(df.iloc[1])
print(df.iloc[0])

.iloc also takes a slice, as a list does. For example, labs.iloc[0:2] gives rows 0 and 1. It leaves out row 2.

Both .iloc and .loc take a second part after a comma. Rows come first. Columns come second. This is the same order as in shape.

  • labs.iloc[0:2, 1] picks rows 0 and 1 and column 1, by position.
  • labs.loc[labs["potassium"] > 5.5, ["ward"]] picks the rows where potassium is above 5.5, and the column ward. The test labs["potassium"] > 5.5 is True for each row with a result above 5.5.
import pandas as pd

labs = pd.DataFrame(
    {
        "ward": ["5A", "7B", "5A", "7B", "5A"],
        "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
    },
    index=["Tan", "Lim", "Kumar", "Wong", "Ng"],
)
print(labs.iloc[0:2])
# ward potassium
# Tan 5A 4.6
# Lim 7B 3.9
print(labs.iloc[0:2, 1])
# Tan 4.6
# Lim 3.9
# Name: potassium, dtype: float64
print(labs.loc[labs["potassium"] > 5.5, ["ward"]])
# ward
# Kumar 5A
# Ng 5A
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.read_csv("patients.csv")
print(df.iloc[0:5])
print(df.iloc[0:5, 0:3])
print(df.loc[df["age"] > 60, ["patient_id", "age", "bmi"]])

A .loc line with a test and a list of columns filters the rows and picks the columns in one step.

Section 8.3

8.3 Boolean filtering

Often, you want every row where a condition is true. This is called boolean filtering. A boolean is a value that is True or False.

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
high = labs["potassium"] > 5.5
print(high)
# 0 False
# 1 False
# 2 True
# 3 False
# 4 True
# Name: potassium, dtype: bool
print(labs[high])
# patient ward potassium
# 2 Kumar 5A 5.8
# 4 Ng 5A 6.1

Read the example in two steps. First, labs["potassium"] > 5.5 gives a Series of True and False, with one value for each row. This Series is called a mask. Second, labs[high] keeps only the rows where the value is True. The kept rows keep index numbers 2 and 4. You can also write both steps in one line as labs[labs["potassium"] > 5.5].

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003", "P004"],
    "age": [54, 61, 47, 72],
    "bmi": [27.3, 24.1, 31.2, 22.8],
})
older = df[df["age"] > 60]
print(older)
Boolean filtering, with True or False badges appearing next to each row to form a mask, then unmatched rows fading out and matching rows collapsing to form the filtered result.
Boolean filtering, with True or False badges appearing next to each row to form a mask, then unmatched rows fading out and matching rows collapsing to form the filtered result.
Section 8.4

8.4 Combining conditions

To join two conditions, use & instead of and, and | instead of or. Put each condition in its own pair of round brackets:

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
both = labs[(labs["potassium"] > 5.0) & (labs["ward"] == "5A")]
print(both)
# patient ward potassium
# 2 Kumar 5A 5.8
# 4 Ng 5A 6.1

either = labs[(labs["potassium"] > 5.5) | (labs["ward"] == "7B")]
print(either)
# patient ward potassium
# 1 Lim 7B 3.9
# 2 Kumar 5A 5.8
# 3 Wong 7B 5.2
# 4 Ng 5A 6.1

The keyword and does not work between two conditions on a Series. It needs one True or False from each condition. A Series holds one value for each row. So Python raises a ValueError. The function bool shows this:

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
test = labs["potassium"] > 5.0
print(bool(test))
# ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

If you leave out the round brackets, Python applies & before the comparisons. The line then raises a TypeError. The message starts with Cannot perform 'rand_'.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np

arr = np.zeros(4)
print(arr)

arr = arr.reshape(2, 2)
print(arr)
To combine two conditions in pandas, use & for AND and | for OR, with each condition wrapped in its own parentheses.
To combine two conditions in pandas, use & for AND and | for OR, with each condition wrapped in its own parentheses.

The method .query() is a second way to filter. You write the whole condition as one string:

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
print(labs.query("potassium > 5.0 and ward == '5A'"))
# patient ward potassium
# 2 Kumar 5A 5.8
# 4 Ng 5A 6.1
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.read_csv("patients.csv")
bracket = df[(df["age"] > 60) & (df["bmi"] > 25)]
query = df.query("age > 60 and bmi > 25")
print(bracket)
print(query)

The two forms give the same rows. Inside the string, column names need no table name around them. The words and and or replace & and |. So round brackets around each condition are not needed.

Section 8.5

8.5 Filtering by multiple categories with .isin

To keep rows whose value appears in a list, use the method isin. Give it the list inside its round brackets. It gives back True for each row whose value appears in the list. A tilde ~ before the condition flips it. Then you get every row not in the list:

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
print(labs[labs["patient"].isin(["Tan", "Ng"])])
# patient ward potassium
# 0 Tan 5A 4.6
# 4 Ng 5A 6.1
print(labs[~labs["patient"].isin(["Tan", "Ng"])])
# patient ward potassium
# 1 Lim 7B 3.9
# 2 Kumar 5A 5.8
# 3 Wong 7B 5.2
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003", "P004", "P005"],
    "clinic": ["A", "B", "C", "D", "A"],
    "age": [54, 61, 47, 72, 38],
})

in_main_clinics = df[df["clinic"].isin(["A", "B", "C"])]
print(in_main_clinics)

elsewhere = df[~df["clinic"].isin(["A", "B", "C"])]
print(elsewhere)
The .isin() method tests each value against a list and returns a boolean Series, which is used to filter rows, and the tilde operator inverts it to select the rows not in the list.
The .isin() method tests each value against a list and returns a boolean Series, which is used to filter rows, and the tilde operator inverts it to select the rows not in the list.
Section 9

9 Renaming columns and changing their type

The rename method changes column names. It takes a dictionary of old and new names:

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
renamed = labs.rename(columns={"potassium": "K"})
print(renamed.columns)  # Index(['patient', 'ward', 'K'], dtype='object')
print(labs.columns)  # Index(['patient', 'ward', 'potassium'], dtype='object')

The rename method gives back a new DataFrame. It leaves the old one unchanged. So store the result under a name, as in df = df.rename(columns={...}).

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.read_csv("patients.csv")
df = df.rename(columns={"chol": "cholesterol", "bmi": "body_mass_index"})
print(df.columns)

To replace every name at once, give a new list to the columns attribute. The list needs one name for each column, in the same order.

For a long table, build the list from names you already have. For example, [c.replace("lab_", "") for c in df.columns] removes the prefix lab_ from every name.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({
    "lab_glucose": [95, 110, 88],
    "lab_hba1c": [5.4, 6.1, 5.0],
    "patient_id": ["P001", "P002", "P003"],
})
df.columns = [c.replace("lab_", "") for c in df.columns]
print(df.columns)

Be careful with this form. If the order is wrong, each column gets the wrong name. Python raises no error for this. If the list has the wrong length, Python raises a ValueError:

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
labs.columns = ["patient", "potassium", "ward"]
print(labs.head(2))
# patient potassium ward
# 0 Tan 5A 4.6
# 1 Lim 7B 3.9
labs.columns = ["patient", "ward"]
# ValueError: Length mismatch: Expected axis has 3 elements, new values have 2 elements

In the first output, the heading potassium is over the wards. The heading ward is over the results. The second list has two names for three columns. So Python raises an error.

The astype method changes the type of a column. For example, astype("float64") converts whole numbers to decimals. In the other direction, astype("int64") cuts off decimals and does not round. So 5.8 becomes 5, and not 6:

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
print(labs["potassium"].astype("int64"))
# 0 4
# 1 3
# 2 5
# 3 5
# 4 6
# Name: potassium, dtype: int64
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({"age": [54, 61, 47]})
print(df["age"].dtype)
print(df["age"].astype("float64"))

bmi = pd.Series([27.9, 24.1, 31.6])
print(bmi.astype("int64"))

A common use of astype is to convert a column to the category type. A column such as ward holds a small set of labels that repeat. The category type keeps one copy of each label. It also keeps a short code for each row. This saves memory on a large table.

The category type also tells pandas that the column holds groups, and not free text.

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
labs["ward"] = labs["ward"].astype("category")
print(labs.dtypes)
# patient object
# ward category
# potassium float64
# dtype: object
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.read_csv("patients.csv")
print(df.dtypes)
df["clinic"] = df["clinic"].astype("category")
print(df.dtypes)
print(df["clinic"].cat.categories)

The select_dtypes method picks columns by type, and not by name. The argument include=["number"] keeps the numeric columns. The argument include=["category", "object"] keeps the label columns:

import pandas as pd

labs = pd.DataFrame({
    "patient": ["Tan", "Lim", "Kumar", "Wong", "Ng"],
    "ward": ["5A", "7B", "5A", "7B", "5A"],
    "potassium": [4.6, 3.9, 5.8, 5.2, 6.1],
})
print(labs.select_dtypes(include=["number"]))
# potassium
# 0 4.6
# 1 3.9
# 2 5.8
# 3 5.2
# 4 6.1
print(labs.select_dtypes(include=["category", "object"]))
# patient ward
# 0 Tan 5A
# 1 Lim 7B
# 2 Kumar 5A
# 3 Wong 7B
# 4 Ng 5A
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.read_csv("patients.csv")
numeric_only = df.select_dtypes(include=["number"])
labels_only = df.select_dtypes(include=["category", "object"])
print(numeric_only.columns)
print(labels_only.columns)

Machine-learning code usually needs numeric columns on their own. Many models cannot use a text column as input. So select_dtypes(include=["number"]) is a quick way to take the numeric part of a table.

Six types cover nearly everything you meet in this course:

  • int64: whole numbers.
  • float64: decimal numbers.
  • object: text, and anything else pandas has no more specific type for.
  • category: a column of repeated labels, stored compactly.
  • bool: True / False.
  • datetime64[ns]: the type pd.to_datetime produces.
Section 10

10 Check your understanding

You have reached the end of the module. Try the same questions again — your answers here, paired with your pre-test answers, are how we measure what the module taught you. Answer every question and confidence rating, then click Submit and see results to view your score.

Post-test

You write df = pd.DataFrame({"a": [1, 2, 3], "b": [10, 20, 30]}). What does df["a"] return?

Post-test

Which of these reads a tab-separated file into a DataFrame?

Post-test

A column of strings like "2024-01-15" is in your DataFrame. After running df["d"] = pd.to_datetime(df["d"]), what dtype does d have?

Post-test

Which call reports the dtype and the number of non-null values for every column of df?

Post-test

df["clinic"].value_counts() returns which kind of object?

Post-test

A DataFrame is indexed by patient_id ("P001", "P002", "P003", ...). Which call returns the row for patient "P002"?

Post-test

You want every row in df where age > 50 AND bmi > 25. Which expression is correct?

Post-test

You want every row whose clinic is one of "A", "B", or "C". Which expression works?

Post-confidence

I can read a CSV or Excel file into a DataFrame and use head, info, and describe to get a quick feel for what is in it.

Not at all confident
Fully confident
Post-confidence

I can filter a DataFrame on one or more boolean conditions and pull out the rows that match.

Not at all confident
Fully confident
Post-confidence

I can parse a column of date strings with pd.to_datetime and pull out the year or month with .dt.

Not at all confident
Fully confident
Section 11

11 Your results

Here is how your post-test answers compare with your pre-test answers. The pre/post pairing is the most reliable way to see what this module actually taught you.

Your score

Submit the post-test to see your results.

Muddiest point

What is the one thing from this module that is still unclear to you?

Rate this module

Overall, how would you rate this module?

How likely are you to recommend this module to a peer? (0 = not at all, 10 = extremely likely)