Section 1

1 Before you start

This is not graded, and it does not count towards anything. Answer every question even if you are only guessing — having a go at something before you have been taught it is one of the best ways to remember it afterwards. The lesson ends with another short check, and there you will be shown the correct answer for anything you missed. Choose an option for every question and confidence rating, then click Submit answers.

Pre-test

You have a wide DataFrame with columns patient_id, v1, v2, v3 (weights at three visits). Which call turns it into long form with one row per (patient, visit, weight)?

Pre-test

You have a long DataFrame with columns patient_id, timepoint, weight. Which call turns it into wide form — one row per patient, one column per timepoint?

Pre-test

pd.crosstab(df["clinic"], df["sex"]) returns what?

Pre-test

You merge a one-row-per-patient demographics table with a multiple-rows-per-patient labs table using an inner join on patient_id. How many rows does the result have?

Pre-test

You want to keep EVERY row of the demographics table, attaching lab values where a patient has them and NaN where they do not. Which how= do you pass to pd.merge?

Pre-test

pd.get_dummies(df, columns=["clinic"]) does what to the DataFrame?

Pre-test

What is the role of the round brackets in a chained pandas pipeline like (df.pipe(...).merge(...).assign(...))?

Pre-test

df.to_numpy() on a DataFrame of numeric columns returns what?

Pre-confidence

I can pivot a long-form DataFrame to wide form and melt a wide-form DataFrame to long form.

Not at all confident
Fully confident
Pre-confidence

I can merge two tables on a key column and pick the right join type (inner, left, full outer) for the question I am asking.

Not at all confident
Fully confident
Pre-confidence

I can write a small chained pipeline that joins two tables, one-hot encodes the categorical columns, and bridges into NumPy with .to_numpy().

Not at all confident
Fully confident
Section 2

2 Introduction

Parts I and II mostly worked on one DataFrame at a time. In real projects, the data you need is often in several tables, or in a layout your next step cannot use.

This part covers reshaping, two-way count tables, joins, one-hot encoding, method chaining and the bridge to NumPy. It is the third and final part on pandas.

Different six pandas operations, namely pivot and melt, crosstab, merge, get_dummies, method chains, and the DataFrame to NumPy bridge.
Different six pandas operations, namely pivot and melt, crosstab, merge, get_dummies, method chains, and the DataFrame to NumPy bridge.

Try every example in the Python Scratchpad on the right.

Section 3

3 Reshaping data (pivot and melt; wide to long)

The same information can sit in two shapes. In wide form, each test has its own column, and each sample has one row. In long form, each result has its own row. A long table has one column for the test name and one column for the result. Wide form is usually easier for a person to read. Long form suits many pandas methods. Reshaping changes a table from one form to the other.

The same dataset can be arranged in wide form (one row per subject, separate columns for each measurement) or long form (one row per measurement, with the measurement type as its own column).
The same dataset can be arranged in wide form (one row per subject, separate columns for each measurement) or long form (one row per measurement, with the measurement type as its own column).
Section 3.1

3.1 Wide to long form

The function pd.melt changes a wide table into a long table. The argument id_vars lists the columns that stay as they are. The argument value_vars lists the columns to stack into one column. The arguments var_name and value_name name the two new columns. The first holds the old column names, and the second holds the values.

import pandas as pd

wide = pd.DataFrame({
    "sample_id": ["S01", "S02", "S03"],
    "glucose": [5.4, 6.0, 4.8],
    "cholesterol": [4.8, 5.4, 4.2],
})
long = pd.melt(
    wide,
    id_vars=["sample_id"],
    value_vars=["glucose", "cholesterol"],
    var_name="test",
    value_name="result",
)
print(long)
# sample_id test result
# 0 S01 glucose 5.4
# 1 S02 glucose 6.0
# 2 S03 glucose 4.8
# 3 S01 cholesterol 4.8
# 4 S02 cholesterol 5.4
# 5 S03 cholesterol 4.2

Each sample now has two rows, one for each test. The old column names, glucose and cholesterol, are now values in the test column.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

wide = pd.DataFrame({
    "patient_id": ["P001", "P002"],
    "weight_baseline": [72, 80],
    "weight_3month": [70, 78],
    "weight_6month": [69, 77],
})
long = pd.melt(
    wide,
    id_vars=["patient_id"],
    value_vars=["weight_baseline", "weight_3month", "weight_6month"],
    var_name="timepoint",
    value_name="weight",
)
print(long)
pandas pd.melt reshapes a wide DataFrame into long format by unpivoting one measurement column at a time into stacked variable and value columns.
pandas pd.melt reshapes a wide DataFrame into long format by unpivoting one measurement column at a time into stacked variable and value columns.

In long form, the test name is a value in a column. So the tools from Part II can use it. For example, groupby gives the mean result for each test:

import pandas as pd

long = pd.DataFrame({
    "sample_id": ["S01", "S02", "S03", "S01", "S02", "S03"],
    "test": ["glucose", "glucose", "glucose", "cholesterol", "cholesterol", "cholesterol"],
    "result": [5.4, 6.0, 4.8, 4.8, 5.4, 4.2],
})
print(long.groupby("test")["result"].mean())
# test
# cholesterol 4.8
# glucose 5.4
# Name: result, dtype: float64
Section 3.2

3.2 Long to wide form

The method pivot changes a long table into a wide table. The argument index names the column that gives the rows. The argument columns names the column whose values become the new column headings. The argument values names the column that fills the cells.

import pandas as pd

long = pd.DataFrame({
    "sample_id": ["S01", "S02", "S03", "S01", "S02", "S03"],
    "test": ["glucose", "glucose", "glucose", "cholesterol", "cholesterol", "cholesterol"],
    "result": [5.4, 6.0, 4.8, 4.8, 5.4, 4.2],
})
wide = long.pivot(index="sample_id", columns="test", values="result")
print(wide)
# test cholesterol glucose
# sample_id
# S01 4.8 5.4
# S02 5.4 6.0
# S03 4.2 4.8

Each sample now has one row, and each test has one column. The sample_id values are now the row labels. The new columns are in alphabetical order.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

long = pd.DataFrame({
    "patient_id": ["P001", "P001", "P002", "P002"],
    "timepoint": ["baseline", "3month", "baseline", "3month"],
    "weight": [72, 70, 80, 78],
})
wide = long.pivot(index="patient_id", columns="timepoint", values="weight")
print(wide)
How DataFrame pivot uses its index, columns, and values arguments to reshape long data into wide form.
How DataFrame pivot uses its index, columns, and values arguments to reshape long data into wide form.

If a sample has two results for the same test, pivot cannot choose between them. It reports an error:

import pandas as pd

repeated = pd.DataFrame({
    "sample_id": ["S01", "S01", "S02"],
    "test": ["glucose", "glucose", "glucose"],
    "result": [5.4, 5.8, 6.0],
})
print(repeated.pivot(index="sample_id", columns="test", values="result"))
# ValueError: Index contains duplicate entries, cannot reshape

The method pivot_table solves this. Its argument aggfunc says how to combine the repeated results. The default is "mean". Other choices include "median", "max" and "first". Here, S01 shows 5.6, the mean of 5.4 and 5.8:

import pandas as pd

repeated = pd.DataFrame({
    "sample_id": ["S01", "S01", "S02"],
    "test": ["glucose", "glucose", "glucose"],
    "result": [5.4, 5.8, 6.0],
})
table = repeated.pivot_table(index="sample_id", columns="test", values="result", aggfunc="mean")
print(table)
# test glucose
# sample_id
# S01 5.6
# S02 6.0
Section 4

4 Two-way categorical tables with pd.crosstab

The method value_counts counts the values of one column. The function pd.crosstab counts the combinations of two columns. For example, it can count how many patients in each study arm responded to a treatment. The rows of the result are the categories of the first column. The columns are the categories of the second column.

import pandas as pd

trial = pd.DataFrame({
    "arm": ["drug", "drug", "drug", "placebo", "placebo", "placebo", "drug", "placebo"],
    "response": ["yes", "yes", "no", "no", "no", "yes", "yes", "no"],
})
print(pd.crosstab(trial["arm"], trial["response"]))
# response no yes
# arm
# drug 1 3
# placebo 3 1

Each number counts the rows that have both labels. For example, 3 patients in the drug arm responded, and 1 did not.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({
    "clinic": ["A", "A", "B", "B", "C", "C"],
    "sex": ["F", "M", "F", "F", "M", "F"],
})
print(pd.crosstab(df["clinic"], df["sex"]))
pd.crosstab counts each row's combination of two categorical columns to produce a 2D summary table, extending the 1D logic of value_counts.
pd.crosstab counts each row's combination of two categorical columns to produce a 2D summary table, extending the 1D logic of value_counts.
Section 5

5 Joining data (inner, left, full outer)

Different tables hold different facts about the same people. One table can hold ages, and another table can hold lab results. Each table has a column that identifies the person, such as a patient ID.

To use the facts together, you need one DataFrame. A join matches the rows of two tables on a shared column, called the key. The pandas function for a join is pd.merge.

Section 5.1

5.1 Inner join

By default, pd.merge does an inner join. It keeps only the rows whose key is in both tables. Use it when you need only the people who have both kinds of information.

import pandas as pd

roster = pd.DataFrame({
    "subject": ["S01", "S02", "S03"],
    "age": [54, 61, 47],
})
results = pd.DataFrame({
    "subject": ["S02", "S03", "S04"],
    "potassium": [4.1, 5.3, 3.8],
})
print(pd.merge(roster, results, on="subject"))
# subject age potassium
# 0 S02 61 4.1
# 1 S03 47 5.3

The subjects S02 and S03 are in both tables. So they are the only rows in the result. S01 has no result and S04 has no age. So both are dropped.

How pd.merge with an inner join keeps only the patient rows that appear in both source tables and discards the rest.
How pd.merge with an inner join keeps only the patient rows that appear in both source tables and discards the rest.
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

demographics = pd.DataFrame({
 "patient_id": ["P001", "P002", "P003"],
 "age": [54, 61, 47],
})
labs = pd.DataFrame({
 "patient_id": ["P002", "P003", "P004"],
 "glucose": [5.4, 6.1, 4.9],
})
combined = pd.merge(demographics, labs, on="patient_id")
print(combined)
Section 5.2

5.2 Left and outer join

An inner join can drop rows you need. The argument how chooses another type of join. The value "left" keeps every row of the left table, which is the first table you pass. The value "right" does the same for the right table. The value "outer" keeps every row of both tables. Where the other table has no match, pandas fills the gap with NaN.

import pandas as pd

roster = pd.DataFrame({
    "subject": ["S01", "S02", "S03"],
    "age": [54, 61, 47],
})
results = pd.DataFrame({
    "subject": ["S02", "S03", "S04"],
    "potassium": [4.1, 5.3, 3.8],
})
left = pd.merge(roster, results, on="subject", how="left")
outer = pd.merge(roster, results, on="subject", how="outer")
print(left)
# subject age potassium
# 0 S01 54 NaN
# 1 S02 61 4.1
# 2 S03 47 5.3
print(outer)
# subject age potassium
# 0 S01 54.0 NaN
# 1 S02 61.0 4.1
# 2 S03 47.0 5.3
# 3 S04 NaN 3.8

The left join keeps S01, who has no result. So potassium is NaN in that row. The outer join also keeps S04, who has no age. In the outer result, age prints as 54.0 instead of 54. This is because a column of numbers that holds NaN is stored as decimal numbers.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

demographics = pd.DataFrame({
 "patient_id": ["P001", "P002", "P003"],
 "age": [54, 61, 47],
})
labs = pd.DataFrame({
 "patient_id": ["P002", "P003", "P004"],
 "glucose": [5.4, 6.1, 4.9],
})
left = pd.merge(demographics, labs, on="patient_id", how="left")
outer = pd.merge(demographics, labs, on="patient_id", how="outer")
print(left)
print(outer)
The how= argument in pandas merge controls which rows survive a join between two tables, with NaN filling any cell that has no match on the other side.
The how= argument in pandas merge controls which rows survive a join between two tables, with NaN filling any cell that has no match on the other side.
Section 5.3

5.3 How to pick

Use an inner join when you need only complete records. Use a left join when one table is your main list and every row of it must stay. Use an outer join when you compare two sources and want to see which records are missing from each.

pandas DataFrame.merge() with how='inner', 'left', and 'outer', showing how each option combines two DataFrames and produces NaN where rows have no match.
pandas DataFrame.merge() with how='inner', 'left', and 'outer', showing how each option combines two DataFrames and produces NaN where rows have no match.

Sometimes the key column has a different name in each table. Then use left_on for the name in the left table and right_on for the name in the right table, instead of on.

import pandas as pd

samples = pd.DataFrame({
    "sample_id": ["S01", "S02"],
    "sample_type": ["serum", "plasma"],
})
results = pd.DataFrame({
    "id": ["S01", "S02"],
    "potassium": [4.6, 3.9],
})
print(pd.merge(samples, results, left_on="sample_id", right_on="id"))
# sample_id sample_type id potassium
# 0 S01 serum S01 4.6
# 1 S02 plasma S02 3.9

The result keeps both key columns, sample_id and id.

How pd.merge joins two DataFrames on a shared key, comparing inner (matches only) and outer (all rows, with NaN for missing values) joins
How pd.merge joins two DataFrames on a shared key, comparing inner (matches only) and outer (all rows, with NaN for missing values) joins

If a key appears several times in one table, its row from the other table is repeated once for each match. Check the key column of each table before you merge. The attribute is_unique is True when no value repeats:

import pandas as pd

samples = pd.DataFrame({
    "sample_id": ["S01", "S02"],
    "sample_type": ["serum", "plasma"],
})
readings = pd.DataFrame({
    "sample_id": ["S01", "S01", "S01", "S02"],
    "potassium": [4.6, 4.7, 4.5, 3.9],
})
print(samples["sample_id"].is_unique)  # True
print(readings["sample_id"].is_unique)  # False
print(pd.merge(samples, readings, on="sample_id"))
# sample_id sample_type potassium
# 0 S01 serum 4.6
# 1 S01 serum 4.7
# 2 S01 serum 4.5
# 3 S02 plasma 3.9

The key S01 appears three times in readings. So the row for S01 from samples is repeated three times, once for each reading.

Section 6

6 One-hot encoding with pd.get_dummies

Many methods that work on numbers cannot use a column of labels, such as ward names. One-hot encoding replaces the column with one new column for each label. The function pd.get_dummies does this. In a new column, a row holds True when it has that label and False when it does not.

import pandas as pd

samples = pd.DataFrame({
    "sample_id": ["S01", "S02", "S03", "S04"],
    "ward": ["5A", "7B", "5A", "9C"],
})
print(pd.get_dummies(samples, columns=["ward"]))
# sample_id ward_5A ward_7B ward_9C
# 0 S01 True False False
# 1 S02 False True False
# 2 S03 True False False
# 3 S04 False False True

Each row has exactly one True. The new column names join the old column name and the label. Add dtype=int to get 1 and 0 instead. Add drop_first=True to drop the first new column. A row with no 1 in the other columns has the label of the dropped column. So no information is lost. Some models need one column to be dropped. This is because the full set of columns repeats information.

import pandas as pd

samples = pd.DataFrame({
    "sample_id": ["S01", "S02", "S03", "S04"],
    "ward": ["5A", "7B", "5A", "9C"],
})
print(pd.get_dummies(samples, columns=["ward"], dtype=int))
# sample_id ward_5A ward_7B ward_9C
# 0 S01 1 0 0
# 1 S02 0 1 0
# 2 S03 1 0 0
# 3 S04 0 0 1
print(pd.get_dummies(samples, columns=["ward"], drop_first=True, dtype=int))
# sample_id ward_7B ward_9C
# 0 S01 0 0
# 1 S02 1 0
# 2 S03 0 0
# 3 S04 0 1
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003", "P004"],
    "clinic": ["A", "B", "A", "C"],
    "age": [54, 61, 47, 72],
})
print(pd.get_dummies(df, columns=["clinic"]))
print(pd.get_dummies(df, columns=["clinic"], drop_first=True))
pd.get_dummies one-hot encodes the clinic column into three binary columns, then dropping the first as the reference category.
pd.get_dummies one-hot encodes the clinic column into three binary columns, then dropping the first as the reference category.
Section 7

7 Building processing pipelines with method chaining

Most analyses use several pandas calls in a row. Most pandas methods return a new DataFrame or Series. So you can call the next method on the result. Calling methods one after another like this is called method chaining. A chain is a single expression that reads from top to bottom, like a recipe.

In pandas, method chaining replaces a string of intermediate variables with a single expression that reads top to bottom, because each method returns a DataFrame the next one can run on.
In pandas, method chaining replaces a string of intermediate variables with a single expression that reads top to bottom, because each method returns a DataFrame the next one can run on.

The steps below find the mean potassium of each ward for patients over 50, and sort the result. First they are separate lines. Then they are one chain, which does not name the in-between results. Both give the same result.

import pandas as pd

records = pd.DataFrame({
    "ward": ["5A", "5A", "5A", "7B", "7B", "7B"],
    "age": [54, 47, 63, 61, 72, 38],
    "potassium": [4.6, 5.8, 5.1, 3.9, 5.2, 4.1],
})

# Separate steps
older = records[records["age"] > 50]
means = older.groupby("ward")["potassium"].mean()
step_result = means.sort_values(ascending=False)
print(step_result)
# ward
# 5A 4.85
# 7B 4.55
# Name: potassium, dtype: float64

# One chain
chain_result = (
    records
    .loc[records["age"] > 50]
    .groupby("ward")["potassium"]
    .mean()
    .sort_values(ascending=False)
)
print(chain_result)
# ward
# 5A 4.85
# 7B 4.55
# Name: potassium, dtype: float64
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({
    "clinic": ["A", "A", "B", "B", "B"],
    "age": [54, 47, 61, 72, 38],
    "bmi": [27.3, 31.2, 24.1, 22.8, 29.5],
})

# Unchained
filtered = df[df["age"] > 40]
grouped = filtered.groupby("clinic")["bmi"].mean()
result = grouped.sort_values(ascending=False)
print(result)
Three pandas operations (filter, groupby mean, and sort) applied step by step to a small DataFrame.
Three pandas operations (filter, groupby mean, and sort) applied step by step to a small DataFrame.
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({
    "clinic": ["A", "A", "B", "B", "B"],
    "age": [54, 47, 61, 72, 38],
    "bmi": [27.3, 31.2, 24.1, 22.8, 29.5],
})

# Chained
result = (
    df
    .loc[df["age"] > 40]
    .groupby("clinic")["bmi"]
    .mean()
    .sort_values(ascending=False)
)
print(result)
A chained pandas pipeline applying filter, groupby, mean, and sort in sequence to produce per-clinic average BMI.
A chained pandas pipeline applying filter, groupby, mean, and sort in sequence to produce per-clinic average BMI.

Write a chain over several lines inside round brackets. Without them, Python reports an error:

result = records
    .loc[records["age"] > 50]
    .groupby("ward")["potassium"]
    .mean()
# IndentationError: unexpected indent

The error is at the second line. This is because the first line is already a complete statement. Round brackets tell Python that the statement continues until the closing bracket.

Inside the brackets, put one method on each line, and line up the dots. You can then comment out one step to test the rest.

A chain is one expression. So you cannot put an assignment, such as records["flag"] = 5, between two steps. The method .assign() adds a column inside a chain:

import pandas as pd

bp = pd.DataFrame({
    "subject": ["S01", "S02", "S03"],
    "systolic": [128, 142, 118],
    "diastolic": [82, 90, 76],
})
result = (
    bp
    .assign(pulse_pressure=lambda d: d["systolic"] - d["diastolic"])
)
print(result)
# subject systolic diastolic pulse_pressure
# 0 S01 128 82 46
# 1 S02 142 90 52
# 2 S03 118 76 42

The method returns a new DataFrame with the extra column. Each keyword argument names a new column. Its value is usually a lambda. The lambda receives the DataFrame as d. Here, the new column pulse_pressure is the systolic pressure minus the diastolic pressure, in mmHg.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003"],
    "weight_kg": [72, 80, 65],
    "height_m": [1.75, 1.80, 1.60],
})
result = (
    df
    .assign(bmi=lambda d: d["weight_kg"] / d["height_m"] ** 2)
)
print(result)

Write the lambda as lambda d: ..., and use d for the DataFrame. This is because d is the DataFrame as it is at that point in the chain, after every earlier filter or .dropna() has run. If you use the original name instead, the calculation reaches back past the filter. It then uses rows that the chain has already removed.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003", "P004"],
    "clinic": ["A", "A", "B", "B"],
    "weight_kg": [72, 58, 65, 90],
})

# Safe: the lambda sees the DataFrame as it stands after the filter, so
# .sum() only adds up clinic A's own two weights.
safe = (
    df
    .loc[df["clinic"] == "A"]
    .assign(pct_of_group=lambda d: d["weight_kg"] / d["weight_kg"].sum() * 100)
)
print(safe)

# Risky: df here still names the ORIGINAL four-row table, so .sum() silently
# adds up all four patients, not just the filtered two.
risky = (
    df
    .loc[df["clinic"] == "A"]
    .assign(pct_of_group=df["weight_kg"] / df["weight_kg"].sum() * 100)
)
print(risky)

The first result gives about 55.4 and 44.6 per cent, which add up to 100. The second result gives about 25.3 and 20.4 per cent. This is because the sum also includes the two patients that the filter removed.

The next pipeline has three steps. The method .dropna() removes the rows that lack a value that you need. The method .rename(columns=...) gives a column a clearer name. Then .assign() adds the column that you want. Cleaning first means that the later steps do not work on rows that you are about to remove:

import pandas as pd

labs = pd.DataFrame({
    "sample_id": ["S01", "S02", "S03", None],
    "total_chol": [5.2, 6.1, None, 4.8],
    "hdl": [1.3, 1.1, 1.5, 1.4],
})
clean = (
    labs
    .dropna(subset=["sample_id", "total_chol"])
    .rename(columns={"sample_id": "id"})
    .assign(non_hdl=lambda d: d["total_chol"] - d["hdl"])
)
print(clean)
# id total_chol hdl non_hdl
# 0 S01 5.2 1.3 3.9
# 1 S02 6.1 1.1 5.0

Two rows remain. The method .dropna() removed S03, which has no total_chol, and the last row, which has no sample_id. The new column non_hdl is the total cholesterol minus the HDL cholesterol, in mmol/L.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003", None],
    "weight_kg": [72, 80, 65, 90],
    "height_m": [1.75, 1.80, None, 1.95],
})

clean = (
    df
    .dropna(subset=["patient_id", "height_m"])
    .rename(columns={"patient_id": "id"})
    .assign(bmi=lambda d: d["weight_kg"] / d["height_m"] ** 2)
)
print(clean)

If a chain grows long, or if you need to look at an in-between result, break it into steps with named variables. This is often clearer.

Section 8

8 From pandas to NumPy and back

Once the data is clean, .to_numpy() turns a DataFrame into a 2D NumPy array. The array has no row labels and no column names. To go back, pd.DataFrame(arr, columns=[...]) names each column. The older attribute .values gives the same array.

import pandas as pd

vitals = pd.DataFrame({
    "heart_rate": [72, 80, 64],
    "temp_c": [36.8, 37.4, 36.6],
})
arr = vitals.to_numpy()
print(arr)
# [[72. 36.8]
# [80. 37.4]
# [64. 36.6]]
print(arr.shape)  # (3, 2)

back = pd.DataFrame(arr, columns=["heart_rate", "temp_c"])
print(back)
# heart_rate temp_c
# 0 72.0 36.8
# 1 80.0 37.4
# 2 64.0 36.6

The whole numbers in heart_rate now print with a decimal point. This is because an array has one type for all its values.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd
import numpy as np

df = pd.DataFrame({"age": [54, 61, 47], "bmi": [27.4, 31.1, 24.8]})
arr = df.to_numpy()
print(arr)
print(arr.shape)

back = pd.DataFrame(arr, columns=["age", "bmi"])
print(back)
Converting between a pandas DataFrame and a NumPy array preserves the underlying numeric values while only the column and row labels are removed and reattached.
Converting between a pandas DataFrame and a NumPy array preserves the underlying numeric values while only the column and row labels are removed and reattached.

A NumPy function can also work on a column directly. A pandas Series keeps its values in a NumPy array. So a function such as np.log reads that array and returns a Series with the same row labels. You do not need .to_numpy() first:

import pandas as pd
import numpy as np

crp = pd.Series([2.0, 8.0, 20.0])
print(np.log(crp))
# 0 0.693147
# 1 2.079442
# 2 2.995732
# dtype: float64
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd
import numpy as np

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003"],
    "glucose": [5.4, 6.1, 4.9],
})
df["log_glucose"] = np.log(df["glucose"])
print(df)
print(type(np.log(df["glucose"])))

The rules from the NumPy lessons carry over to a column. Arithmetic on a column gives one result for each row, with no loop. A comparison such as df["glucose"] > 7 gives a Series of True and False, just as comparing an array gives an array. The function np.where also accepts a column.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import pandas as pd
import numpy as np

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003", "P004"],
    "glucose": [5.4, 7.8, 6.1, 9.2],
})
df["glucose_mgdl"] = df["glucose"] * 18
print(df["glucose"] > 7)
df["flag"] = np.where(df["glucose"] > 7, "high", "normal")
print(df)
Section 9

9 Check your understanding

You have reached the end of the module. Try the same questions again — your answers here, paired with your pre-test answers, are how we measure what the module taught you. Answer every question and confidence rating, then click Submit and see results to view your score.

Post-test

You have a wide DataFrame with columns patient_id, v1, v2, v3 (weights at three visits). Which call turns it into long form with one row per (patient, visit, weight)?

Post-test

You have a long DataFrame with columns patient_id, timepoint, weight. Which call turns it into wide form — one row per patient, one column per timepoint?

Post-test

pd.crosstab(df["clinic"], df["sex"]) returns what?

Post-test

You merge a one-row-per-patient demographics table with a multiple-rows-per-patient labs table using an inner join on patient_id. How many rows does the result have?

Post-test

You want to keep EVERY row of the demographics table, attaching lab values where a patient has them and NaN where they do not. Which how= do you pass to pd.merge?

Post-test

pd.get_dummies(df, columns=["clinic"]) does what to the DataFrame?

Post-test

What is the role of the round brackets in a chained pandas pipeline like (df.pipe(...).merge(...).assign(...))?

Post-test

df.to_numpy() on a DataFrame of numeric columns returns what?

Post-confidence

I can pivot a long-form DataFrame to wide form and melt a wide-form DataFrame to long form.

Not at all confident
Fully confident
Post-confidence

I can merge two tables on a key column and pick the right join type (inner, left, full outer) for the question I am asking.

Not at all confident
Fully confident
Post-confidence

I can write a small chained pipeline that joins two tables, one-hot encodes the categorical columns, and bridges into NumPy with .to_numpy().

Not at all confident
Fully confident
Section 10

10 Your results

Here is how your post-test answers compare with your pre-test answers. The pre/post pairing is the most reliable way to see what this module actually taught you.

Your score

Submit the post-test to see your results.

Muddiest point

What is the one thing from this module that is still unclear to you?

Rate this module

Overall, how would you rate this module?

How likely are you to recommend this module to a peer? (0 = not at all, 10 = extremely likely)