Python Fundamentals — Pandas I

Before you begin, take a few minutes to check what you already know and how confident you feel. You will see the same questions again at the end of the module — this helps both you and us measure what you have learned. Click an option for every question and confidence rating, then click Next to continue.

Pandas is the workhorse of data analysis in Python. Almost any tabular data you will meet in precision medicine — patient records, lab results, gene counts, survey responses — lives most comfortably in a pandas DataFrame.

This is the first of three parts on pandas. Part I focuses on getting data into a DataFrame and understanding it. Part II covers cleaning and transforming a single DataFrame. Part III covers reshaping, combining, and exporting.

This part covers seven connected pieces. Treat them as a path — each one builds on the one before, and by the end you can read a real CSV, look it over, and pull out the rows that matter for the question you are asking.

• Introduction to Pandas DataFrames — what a DataFrame is, how to build one from scratch, and what a Series is.
• Reading CSV and Excel files — pull a real file off disk in one line.
• Working with dates: pd.to_datetime — parse a column of date strings into proper datetime values you can sort and filter on.
• Inspecting dataframes — the four short calls (head, shape, info, describe) that tell you what you have.
• Counting categories with value_counts — the call you reach for most often on a categorical column.
• Data selection and filtering — pick the rows and columns you actually want, by label, by position, or by condition.
• Filtering by multiple categories with .isin — keep every row whose value matches one of a list, in one short expression.

Try every snippet in the Python Scratchpad on the right. By the end of this part you will read a CSV, inspect it, parse its dates, and pull out the rows that matter.

A DataFrame is the central object in pandas. Picture it as a table with named columns, numbered rows, and a value in every cell.

By convention, pandas is imported under the short alias pd, so the line at the top of your script will look the same as the line at the top of everyone else's:

import pandas as pd

data = {
    "patient_id": ["P001", "P002", "P003"],
    "age": [54, 61, 47],
    "bmi": [27.3, 24.1, 31.2],
}
df = pd.DataFrame(data)
print(df)

Each entry in data becomes a column: the name on the left is the column header, and the list on the right fills the cells beneath it.

The 0, 1, 2 down the left side is the index, which pandas adds for you when you do not specify one. Naming the variable df is another convention you will see everywhere.

A single column is its own object, called a Series: a one-dimensional labelled array, or a list of values with an index attached. You pull one out of a DataFrame by passing the column name in square brackets.

That last line shows why pandas is worth the import. Once your data is in a Series, calling .mean() gives you the average. No loop, no sum divided by length, no manual conversion to floats. The Series knows its own values and the operations that make sense on them. The same applies to .sum(), .median(), .std(), .min(), .max(), and dozens of others.

Building a DataFrame from a dictionary is fine for examples and tiny tables, but real data almost always lives in a file. The next subtopic covers how to load one.

Almost no real-world data starts its life as a Python literal. Instead, it arrives as a CSV from a colleague, an Excel workbook from a clinical study, or a TSV exported from a genomic sequencing pipeline. Fortunately, pandas can read all of these formats with a single, easily memorable line of code.

To load a comma-separated file, use the pd.read_csv() function. Simply pass in the file path, and pandas handles the heavy lifting: it identifies the columns from the header row, infers the data type for each column, and returns a fully structured DataFrame.

Note: We have provided several csv files that have beep pre-loaded for you in the Python Scratchpad

import pandas as pd

df = pd.read_csv("patients.csv")
print(df)

Several keyword arguments to read_csv come up so often that they are worth knowing on day one:

• sep="\t" reads a tab-separated file (TSV) instead of the default comma.
• header=None tells pandas the file has no header row, so it should make up column names itself.
• index_col="patient_id" uses a specific column as the row index instead of the auto-numbered 0, 1, 2.
• na_values=["NA", "missing", -999] tells pandas which strings to treat as missing (more on missing values in a later subtopic).
• nrows=100 reads only the first hundred rows, perfect for a quick peek at a huge file before you commit to loading the whole thing.

import pandas as pd

df = pd.read_csv(
    "patients.tsv",
    sep="\t",
    index_col="patient_id",
    na_values=["NA", "missing", -999],
    nrows=100,
)

print(df)

Excel files use a sister function called. pd.read_excel. The shape is the same. Excel workbooks can have multiple sheets, so the extra argument you reach for most often is sheet_name: pass the sheet's name as a string, or its position as an integer (0 for the first sheet).

Writing a DataFrame back out is the mirror image.

• df.to_csv("output.csv") writes a CSV
• df.to_excel("output.xlsx") writes an Excel file.

One small habit worth forming: pass index=False unless the index carries information. Without it, pandas writes the row numbers as a first column, which the next person to read your file will then have to clean up.

The syntax to import Excel or CSV files as Dataframes is given as below.

df.to_csv("output.csv", index=False)
df.to_excel("output.xlsx", index=False)

If you get a FileNotFoundError, pandas is looking in your current working directory. Either move the file there, pass a full path, or use pathlib to build a path from the script's location. The easiest first move is print(__file__) at the top of your script to remind yourself where Python thinks it is running from.

A column of dates that comes out of CSV is almost always a string. pandas does not know that "2024-01-15" is a date until you tell it.

The pd.to_datetime function parses a column into proper datetime values. Once converted, you can easily sort by date, filter by specific ranges, and extract the year, month, or day using the .dt accessor. Proper date handling is essential in clinical data, where you constantly manage admission dates, follow-up intervals, and dates of birth.

import pandas as pd

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003"],
    "visit": ["2024-01-15", "2024-03-02", "2024-06-21"],
})
df["visit"] = pd.to_datetime(df["visit"])
print(df.dtypes)
print(df["visit"].dt.year)
print(df["visit"].dt.month_name())

The first thing you should do with any newly loaded DataFrame is inspect it.

Printing the entire dataset is counterproductive. This is because a DataFrame can hold millions of rows and will quickly clutter your screen with unreadable text.

Instead, use these method calls to immediately understand what you are working with. Run them in this order on every new file. If something is wrong, you will catch it early on, before the error propagates through the rest of your analysis.

• df.head() shows the first five rows.
• df.head(10) shows the first ten.
• df.tail() shows the last five, useful for spotting a stray totals row at the bottom of a spreadsheet.
• df.shape is a tuple of (rows, columns), no parentheses, it is an attribute, not a method.
• df.columns lists the column names.
• df.dtypes shows the type of each column.

import pandas as pd

df = pd.read_csv("patients.csv")
print(df.shape)
print(df.columns)
print(df.dtypes)
print(df.head())

The df.info() method bundles these diagnostics together, adding the count of non-null values per column and the total memory usage of the DataFrame. It is the single most useful tool when an import goes wrong. For instance, if a column you expected to be numeric shows up as an object, df.info() flags it immediately.

import pandas as pd

df = pd.read_csv("patients.csv")
df.info()

df.describe() gives you summary statistics for every numeric column at once: count, mean, standard deviation, minimum, the 25th, 50th, and 75th percentiles, and the maximum. It is the fastest way to spot impossible values — a negative age, a heart rate of zero, a BMI of 9000 — that always seem to lurk in real data.

For the non-numeric columns, df.describe(include="object") gives you count, number of unique values, the most frequent value, and how often it occurs.

import pandas as pd

df = pd.DataFrame({
    "age": [54, 61, 47, 72, 38],
    "bmi": [27.3, 24.1, 31.2, 22.8, 29.5],
})
print(df.shape)

A typical first pass on a new file is exactly four lines:

• read the file,
• print the shape,
• print info,
• print head.

Five seconds of looking saves an hour of debugging.

When a column holds categorical values, smoking status, treatment arm, blood type, the question you ask first is almost always "what are the categories and how often does each one show up?"

Eyeballing head() will not tell you, because the first five rows might all happen to be the same category. You need a count over the whole column.

The call you reach for is .value_counts(). It returns a Series of counts, one per unique value, sorted high to low.

That ordering matters. The most common categories sit at the top where you will see them, and rare ones (often the interesting ones, or the data-entry typos) sit at the bottom where they stand out.

Two keyword arguments are worth knowing from the start.

• normalize=True returns proportions instead of raw counts, useful when you want to compare the distribution across two cohorts of different sizes.
• dropna=False makes missing values a category of their own and counts them alongside the rest, which is what you want when you are auditing data quality rather than describing the observed sample.

import pandas as pd

df = pd.DataFrame({"clinic": ["A", "B", "A", "C", "B", "B", "A"]})
print(df["clinic"].value_counts())
print(df["clinic"].value_counts(normalize=True))

You almost never want the entire DataFrame. You want the patients over sixty, the rows where BMI is above thirty, or the two columns called age and bmi but not the other twenty. pandas gives you a small set of notations for picking out the bits you care about, and they cover almost every selection you will ever make.

When selecting columns in pandas, the number of brackets you use determines what you get back. If you ask for a single column with df["age"], pandas hands you back a 1D Series. But if you want multiple columns, you must use double square brackets—like df[["age", "bmi"]]—which returns a smaller 2D DataFrame. If the double brackets look strange, just remember that the outer pair tells pandas you want to select data, while the inner pair is simply a standard Python list of the column names you are asking for.

import pandas as pd

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003"],
    "age": [54, 61, 47],
    "bmi": [27.3, 24.1, 31.2],
})
print(df["age"])
print(df[["patient_id", "bmi"]])

To pick rows by label or position, you use two cousins:

• df.loc[2] gives you the row whose label is 2
• df.iloc[2] gives you the third row by position (index).

For a DataFrame with the default 0, 1, 2, ... index, the two look identical. As soon as you set a meaningful index (patient_id, for instance), they diverge and the difference matters.

Rule of thumb: .loc is by name, .iloc is by number.

import pandas as pd

df = pd.DataFrame(
    {"age": [54, 61, 47]},
    index=["P001", "P002", "P003"],
)

print(df.loc["P002"])
print(df.iloc[1])
print(df.iloc[0])

The most common thing you will ask is not "give me row three" but "give me every row where some condition is true." That is called boolean filtering, and it is the move you will reach for ten times a day.

import pandas as pd

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003", "P004"],
    "age": [54, 61, 47, 72],
    "bmi": [27.3, 24.1, 31.2, 22.8],
})
older = df[df["age"] > 60]
print(older)

Read it in two steps:

• df["age"] > 60 produces a Series of True and False, one entry per row, True wherever the condition holds.
• Passing that Series back into df[...] keeps only the rows where the entry was True.

Once your eye is trained for this pattern, you will read df[df["age"] > 60] without thinking.

Use & for and, | for or, and put each condition in its own pair of round brackets. The brackets are not optional. & has higher precedence than the comparison operators, and without them Python parses the expression in a way that almost always raises an error.

import numpy as np

arr = np.zeros(4)
print(arr)

arr = arr.reshape(2, 2)
print(arr)

When you want every row whose clinic is one of "A", "B", or "C", you could write three conditions joined by | - but the cleaner way is .isin. Pass it a list of accepted values and it returns a boolean Series, True wherever the column value is in the list. Prefix with ~ to flip it ("not in"). This pairs perfectly with the boolean filtering pattern from earlier in this module.

import pandas as pd

df = pd.DataFrame({
    "patient_id": ["P001", "P002", "P003", "P004", "P005"],
    "clinic": ["A", "B", "C", "D", "A"],
    "age": [54, 61, 47, 72, 38],
})

in_main_clinics = df[df["clinic"].isin(["A", "B", "C"])]
print(in_main_clinics)

elsewhere = df[~df["clinic"].isin(["A", "B", "C"])]
print(elsewhere)

The seven tools in Part I are usually used together. A typical first pass on a new dataset reads the file, prints the shape and dtypes, looks at the first few rows, counts the categories of one column, parses any date columns, and filters down to the rows that matter for the question you are asking.

import pandas as pd

df = pd.read_csv("patients.csv")
print(df.shape)
print(df.info())
print(df["clinic"].value_counts())
df["visit_date"] = pd.to_datetime(df["visit_date"])
selected = df[(df["age"] > 60) & df["clinic"].isin(["A", "B", "C"])]
print(selected)

You have reached the end of the module. Try the same questions again — your answers here, paired with your pre-test answers, are how we measure what the module taught you. Answer every question and confidence rating, then click Submit and see results to view your score.

Here is how your post-test answers compare with your pre-test answers. The pre/post pairing is the most reliable way to see what this module actually taught you.