Section 1

1 Before you start

This is not graded, and it does not count towards anything. Answer every question even if you are only guessing — having a go at something before you have been taught it is one of the best ways to remember it afterwards. The lesson ends with another short check, and there you will be shown the correct answer for anything you missed. Choose an option for every question and confidence rating, then click Submit answers.

Pre-test

ages is a NumPy array. Which of these raises an error because the method does not exist?

Pre-test

patients is a 2D array with one row per patient and columns age, weight, height. Which call returns the mean age, mean weight, and mean height (one value per column)?

Pre-test

You write ages[ages >= 50 and ages < 70] and Python raises a ValueError. What is the fix?

Pre-test

You want a new array the same length as ages, where each element is "senior" or "adult" depending on whether the age is at least 65. Which call does this?

Pre-test

In np.where(mask, a / b, a) some entries of b are 0, and Python warns about dividing by zero even for elements that take the False branch. Why?

Pre-test

bmi is a NumPy array. What does bmi.argmax() return?

Pre-test

values contains some np.nan entries. Why does values == np.nan fail to find them?

Pre-test

ages = np.array([54, 61, 47, 72, 38]). What does ages[ages >= 60] return?

Pre-confidence

I can compute the mean, median, and standard deviation of a 1D array, and use the axis argument to summarise a 2D array down its rows or columns.

Not at all confident
Fully confident
Pre-confidence

I can filter an array with a boolean condition, combine two conditions with the symbol & or |, and use np.where to build a new array of transformed values.

Not at all confident
Fully confident
Section 2

2 Introduction

In Part I, you saw that a NumPy array can do arithmetic on every value at once. This part covers three more things you can do with an array.

You can reduce an array to one summary number, such as its average. You can filter an array, keeping only the values that pass a test. You can also transform an array, building a new array where each value depends on a test.

NumPy array allows for statistic calculations along an axis, Boolean indexing and filtering, and transform cell by cell with np.where.
NumPy array allows for statistic calculations along an axis, Boolean indexing and filtering, and transform cell by cell with np.where.

Try every example in the Python Scratchpad on the right.

Section 3

3 Statistical operations (mean, median, std)

A NumPy array can summarise itself without a loop. Here are five potassium results in mmol/L:

import numpy as np
levels = np.array([4.6, 3.9, 5.8, 5.2, 6.1])
print(levels.mean())  # 5.12
print(np.median(levels))  # 5.2
print(levels.std())  # 0.7984985911070851

The mean is the sum of the values divided by their count. The median is the middle value of the sorted values. The standard deviation measures how far the values spread out around the mean.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np
ages = np.array([54, 61, 47, 72, 38])
print(ages.mean())
print(np.median(ages))
print(ages.std())

In the exercise, the median of whole numbers prints with a decimal point. This is because NumPy gives the median as a decimal number.

The mean and the standard deviation can be written as a method, like levels.mean(), or as a function, like np.mean(levels). Both give the same number.

The median is different. NumPy has only the function np.median. So the method call below raises an AttributeError:

import numpy as np
levels = np.array([4.6, 3.9, 5.8, 5.2, 6.1])
print(levels.median())
# AttributeError: 'numpy.ndarray' object has no attribute 'median'
Mean reports the average, median reports the middle value when the array is sorted, and standard deviation reports the typical distance of values from the mean.
Mean reports the average, median reports the middle value when the array is sorted, and standard deviation reports the typical distance of values from the mean.

Four more methods work in the same way: sum, min, max and var:

import numpy as np
levels = np.array([4.6, 3.9, 5.8, 5.2, 6.1])
print(levels.sum())  # 25.6
print(levels.min())  # 3.9
print(levels.max())  # 6.1
print(levels.var())  # 0.6376

The variance is the standard deviation squared.

Aggregation methods reduce an array of many values to a single summary number.
Aggregation methods reduce an array of many values to a single summary number.

The argument axis tells a summary method which direction to use on a 2D array. Here is a grid of potassium results, with one row per patient and one column per day:

import numpy as np
readings = np.array([
    [4.6, 4.4, 4.2],
    [3.9, 4.1, 4.0],
    [5.8, 5.1, 4.7],
    [5.2, 5.0, 4.8],
    [6.1, 5.3, 4.8]
])
print(readings.mean())  # 4.8
print(readings.mean(axis=0))  # [5.12 4.78 4.5 ]
print(readings.mean(axis=1))  # [4.4 4.  5.2 5.  5.4]

With no axis, NumPy uses all fifteen values and gives one number. With axis=0, NumPy combines the five rows into one row. So each column gives one average, one per day. With axis=1, NumPy combines the three columns into one column. So each row gives one average, one per patient.

How NumPy's axis argument controls 2D array summaries
How NumPy's axis argument controls 2D array summaries

The axis you name is the axis that disappears. The shape of the grid is (5, 3). So axis=0 leaves three numbers. And axis=1 leaves five.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np
patients = np.array([
    [54, 72.5, 1.75],
    [61, 68.0, 1.80],
    [47, 80.2, 1.68]
])
print(patients.mean())
print(patients.mean(axis=0))
print(patients.mean(axis=1))

The first line gives one number for all nine values. The second line gives one number per column. The third line gives one number per patient. Each number mixes quantities with different units. So these numbers have no meaning.

Naming an axis collapses the array along that direction, so axis=0 averages down each column into one value per feature while axis=1 averages across each row, which only makes sense when a row's values share the same units.
Naming an axis collapses the array along that direction, so axis=0 averages down each column into one value per feature while axis=1 averages across each row, which only makes sense when a row's values share the same units.
Section 3.1

3.1 argmin and argmax

The methods argmax and argmin give the position of the largest and smallest value, not the value itself. Here are the results again, with patient names in the same order:

import numpy as np
levels = np.array([4.6, 3.9, 5.8, 5.2, 6.1])
names = np.array(["Tan", "Lim", "Kumar", "Wong", "Ng"])
print(levels.max())  # 6.1
print(levels.argmax())  # 4
print(names[levels.argmax()])  # Ng
print(names[levels.argmin()])  # Lim

A position is useful when a second array has values in the same order. Here the position of the largest result picks the matching name. On a 2D array, argmin and argmax take the axis argument just like mean.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np
bmi = np.array([27.4, 31.1, 24.8, 29.5, 33.2])
print(bmi.max())
print(bmi.argmax())
print(bmi[bmi.argmax()])
.argmax() returns the index where the maximum sits, not the maximum itself, which is why bmi[bmi.argmax()] indexes back to recover the value that .max() reports directly.
.argmax() returns the index where the maximum sits, not the maximum itself, which is why bmi[bmi.argmax()] indexes back to recover the value that .max() reports directly.
Section 3.2

3.2 np.unique for counting categories

Some arrays hold categories, such as ward names or blood types. The function np.unique gives back each different value once in sorted order. With the argument return_counts=True, it also gives back how many times each value appears. Here is an array of wards, with one entry per patient:

import numpy as np
wards = np.array(["5A", "7B", "5A", "7B", "5A"])
print(np.unique(wards))  # ['5A' '7B']
ward_names, how_many = np.unique(wards, return_counts=True)
print(ward_names)  # ['5A' '7B']
print(how_many)  # [3 2]

The two arrays line up by position. So ward 5A has three patients, and ward 7B has two.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np
clinics = np.array(["A", "B", "A", "C", "B", "B", "A"])
values, counts = np.unique(clinics, return_counts=True)
print(values)
print(counts)
np.unique with return_counts=True gives back two arrays aligned by position, the sorted distinct values and a parallel count of how many times each occurs, so values[i] and counts[i] describe the same category rather than forming a dictionary.
np.unique with return_counts=True gives back two arrays aligned by position, the sorted distinct values and a parallel count of how many times each occurs, so values[i] and counts[i] describe the same category rather than forming a dictionary.
Section 3.3

3.3 np.percentile and np.quantile

A percentile is a cut point in the sorted values. For example, the 50th percentile is the median. You can pass a list of percentages to np.percentile to get several cut points at once:

import numpy as np
levels = np.array([4.6, 3.9, 5.8, 5.2, 6.1])
print(np.percentile(levels, 50))  # 5.2
print(np.percentile(levels, [25, 50, 75]))  # [4.6 5.2 5.8]
print(np.quantile(levels, 0.5))  # 5.2

The function np.quantile takes a fraction from 0 to 1 instead of a percentage. Percentiles are useful for skewed data, such as lengths of stay, costs or lab values. This is because a few large values pull the mean upwards. They move the median much less.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np
length_of_stay = np.array([2, 3, 3, 4, 5, 5, 7, 9, 12, 30])
print(np.percentile(length_of_stay, [25, 50, 75]))
print(np.percentile(length_of_stay, 95))
Percentiles are cut-points in sorted data that np.percentile finds by rank and linear interpolation between neighbouring values, so on a skewed distribution the median stays near the typical case while the 95th is pulled far out toward the tail, which is exactly why percentiles beat mean and SD here.
Percentiles are cut-points in sorted data that np.percentile finds by rank and linear interpolation between neighbouring values, so on a skewed distribution the median stays near the typical case while the 95th is pulled far out toward the tail, which is exactly why percentiles beat mean and SD here.
Section 3.4

3.4 np.isnan for detecting missing values

NumPy marks a missing value with np.nan, which stands for not a number. Here are the results of a later visit, when two patients did not come:

import numpy as np
followup = np.array([4.3, np.nan, 4.6, 4.8, np.nan])
print(followup == np.nan)  # [False False False False False]
print(np.isnan(followup))  # [False  True False False  True]

A test with two equals signs finds nothing. This is because np.nan is not equal to anything, and that includes itself. So you use the function np.isnan. It gives True at each missing value.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np
values = np.array([54, np.nan, 47, 72, np.nan])
mask = np.isnan(values)
print(mask)
print(values[~mask])

The symbol ~ flips each True to False and each False to True. An array of True and False values inside square brackets keeps the values at the True positions. So the last line of the exercise keeps the values that are not missing.

Missing values are found with np.isnan, and the resulting boolean mask is inverted with ~ so that boolean indexing keeps only the real values.
Missing values are found with np.isnan, and the resulting boolean mask is inverted with ~ so that boolean indexing keeps only the real values.
Section 4

4 Boolean indexing and filtering

Often you want to pick values by a test, such as the results above 5.5 mmol/L. In plain Python, this needs a loop with an if. In NumPy, you write the test on the whole array:

import numpy as np
levels = np.array([4.6, 3.9, 5.8, 5.2, 6.1])
high = levels > 5.5
print(high)  # [False False  True False  True]
print(levels[high])  # [5.8 6.1]
print(levels[levels > 5.5])  # [5.8 6.1]

The array of True and False values is a boolean mask. It does not remove anything. It only records which positions pass the test. Inside square brackets, it keeps the values at the True positions and drops the rest. The last line above puts the test directly inside the square brackets.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np
ages = np.array([54, 61, 47, 72, 38])
is_senior = ages >= 65
print(is_senior)
print(ages[is_senior])
A boolean array of the same length acts as a per-element on/off switch, and only the elements at True positions are kept.
A boolean array of the same length acts as a per-element on/off switch, and only the elements at True positions are kept.

Every comparison gives a mask in this way. The sign == finds the values that match exactly. The sign != finds all the other values:

import numpy as np
levels = np.array([4.6, 3.9, 5.8, 5.2, 6.1])
print(levels < 4.6)  # [False  True False False False]
print(levels == 5.2)  # [False False False  True False]
print(levels != 5.2)  # [ True  True  True False  True]
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np
ages = np.array([54, 61, 47, 72, 38])
print(ages <= 60)
print(ages == 47)
print(ages != 47)

Often you need two tests at once, such as the results in a reference range of 3.5 to 5.0 mmol/L. In plain Python, you join two tests with the keyword and. With an array, this raises a ValueError:

import numpy as np
levels = np.array([4.6, 3.9, 5.8, 5.2, 6.1])
print(levels[levels >= 3.5 and levels <= 5.0])
# ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

This is because the keyword and expects one answer. But a test on an array gives one answer for each value. So you use the symbol & for AND. You use the symbol | for OR. Each test must also be inside round brackets. Without them, Python applies & before the comparisons. So the line fails:

import numpy as np
levels = np.array([4.6, 3.9, 5.8, 5.2, 6.1])
print(levels[levels >= 3.5 & levels <= 5.0])
# TypeError: ufunc 'bitwise_and' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

With round brackets, the symbol & keeps the results in the reference range. The symbol | keeps the results below 4.0 or above 5.5:

import numpy as np
levels = np.array([4.6, 3.9, 5.8, 5.2, 6.1])
in_range = levels[(levels >= 3.5) & (levels <= 5.0)]
print(in_range)  # [4.6 3.9]
low_or_high = levels[(levels < 4.0) | (levels > 5.5)]
print(low_or_high)  # [3.9 5.8 6.1]
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np
ages = np.array([54, 61, 47, 72, 38])
between = ages[(ages >= 50) & (ages < 70)]
print(between)
A comparison on an array produces a boolean mask of the same shape, which can be passed straight into the brackets to keep only the elements that satisfy the condition.
A comparison on an array produces a boolean mask of the same shape, which can be passed straight into the brackets to keep only the elements that satisfy the condition.

A mask also works on any other array of the same length. On a 2D array, you can test one column, or the whole grid:

import numpy as np
names = np.array(["Tan", "Lim", "Kumar", "Wong", "Ng"])
readings = np.array([
    [4.6, 4.4, 4.2],
    [3.9, 4.1, 4.0],
    [5.8, 5.1, 4.7],
    [5.2, 5.0, 4.8],
    [6.1, 5.3, 4.8]
])
day_1 = readings[:, 0]
print(names[day_1 > 5.5])  # ['Kumar' 'Ng']
print(readings[day_1 > 5.5, :].shape)  # (2, 3)
print(readings[readings > 5.5])  # [5.8 6.1]

The test on the first column gives one True or False for each patient. So it picks two names. It also keeps two whole rows of the grid. The test on the whole grid gives back the values that pass, as a flat 1D array.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np
patients = np.array([
    [54, 72.5, 1.75],
    [61, 68.0, 1.80],
    [47, 80.2, 1.68]
])
ages = patients[:, 0]
print(patients[ages >= 50, :])
A boolean mask the same shape as a 2D array picks out True positions in row-by-row order and returns them as a flat 1D array.
A boolean mask the same shape as a 2D array picks out True positions in row-by-row order and returns them as a flat 1D array.
Section 5

5 Conditional transforms with np.where

A filter gives back fewer values. Sometimes you want one value for each position instead, chosen by a test. For example, you may want to label each potassium result as "review" or "ok". The function np.where does this:

import numpy as np
levels = np.array([4.6, 3.9, 5.8, 5.2, 6.1])
labels = np.where(levels > 5.5, "review", "ok")
print(labels)  # ['ok' 'ok' 'review' 'ok' 'review']

The function np.where takes three things: the test, the value for each position that passes the test, and the value for each position that fails.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np
ages = np.array([54, 61, 47, 72, 38])
labels = np.where(ages >= 65, "senior", "adult")
print(labels)
NumPy's np.where shown in both forms, returning the positions where a condition is True, and selecting one of two values for each element of an array.
NumPy's np.where shown in both forms, returning the positions where a condition is True, and selecting one of two values for each element of an array.

The second and third values can also be whole arrays. Here, NumPy takes the result from levels at each missing value of followup, and the result from followup at every other position:

import numpy as np
levels = np.array([4.6, 3.9, 5.8, 5.2, 6.1])
followup = np.array([4.3, np.nan, 4.6, 4.8, np.nan])
filled = np.where(np.isnan(followup), levels, followup)
print(filled)  # [4.3 3.9 4.6 4.8 6.1]
Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np
doses = np.array([5, 12, 7, 20, 3])
boosted = np.where(doses > 10, doses * 2, doses)
print(boosted)

The function np.where works out both sets of values in full, for every position. Only after that does it pick between them. This matters when one set of values divides by zero.

Here, samples is the number of blood samples that each patient gave in a week. Then 7 / samples is the number of days between samples:

import numpy as np
samples = np.array([7, 0, 2, 1, 0])
gap = np.where(samples > 0, 7 / samples, 0)
# RuntimeWarning: divide by zero encountered in divide
print(gap)  # [1.  0.  3.5 7.  0. ]

The test samples > 0 skips the zeros. But 7 / samples has already been worked out at every position. So Python prints a warning about dividing by zero. The result itself is correct.

np.where computes the True and False result arrays in full before selecting, so unsafe math in a branch you never use (like 100/0) still triggers its warning or error.
np.where computes the True and False result arrays in full before selecting, so unsafe math in a branch you never use (like 100/0) still triggers its warning or error.

A filter gives back only the values that pass the test. So the result is usually shorter than the array. The function np.where gives back one value for each position. So the result has the same length as the array.

A z-score says how many standard deviations a value is from the average. You subtract the average from every value, and then divide by the standard deviation. Here are eight results:

import numpy as np
batch = np.array([4.1, 4.6, 3.9, 4.4, 5.2, 4.8, 4.3, 9.5])
z = (batch - batch.mean()) / batch.std()
print(round(z[-1], 2))  # 2.58

The last result is 2.58 standard deviations above the average. A z-score of 2 or more, in either direction, is often treated as an outlier.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np
glucose = np.array([4.5, 5.1, 4.9, 9.8, 5.3, 5.0, 4.7])
z_scores = (glucose - glucose.mean()) / glucose.std()
print(z_scores)

In the exercise, the fourth reading, 9.8, has a z-score of about 2.4. The next exercise flags it as an outlier. The function np.abs removes the minus signs. So one test catches a z-score of 2 or more in either direction.

Try it out

Try this snippet in the Python Scratchpad on the right.

Try this snippet
import numpy as np
glucose = np.array([4.5, 5.1, 4.9, 9.8, 5.3, 5.0, 4.7])
z_scores = (glucose - glucose.mean()) / glucose.std()
outliers = np.where(np.abs(z_scores) >= 2, 1, 0)
print(z_scores)
print(outliers)

The test np.abs(z_scores) >= 2 is True only for the fourth reading. So np.where gives that reading 1. It gives every other reading 0.

Section 6

6 Check your understanding

You have reached the end of the module. Try the same questions again — your answers here, paired with your pre-test answers, are how we measure what the module taught you. Answer every question and confidence rating, then click Submit and see results to view your score.

Post-test

ages is a NumPy array. Which of these raises an error because the method does not exist?

Post-test

patients is a 2D array with one row per patient and columns age, weight, height. Which call returns the mean age, mean weight, and mean height (one value per column)?

Post-test

You write ages[ages >= 50 and ages < 70] and Python raises a ValueError. What is the fix?

Post-test

You want a new array the same length as ages, where each element is "senior" or "adult" depending on whether the age is at least 65. Which call does this?

Post-test

In np.where(mask, a / b, a) some entries of b are 0, and Python warns about dividing by zero even for elements that take the False branch. Why?

Post-test

bmi is a NumPy array. What does bmi.argmax() return?

Post-test

values contains some np.nan entries. Why does values == np.nan fail to find them?

Post-test

ages = np.array([54, 61, 47, 72, 38]). What does ages[ages >= 60] return?

Post-confidence

I can compute the mean, median, and standard deviation of a 1D array, and use the axis argument to summarise a 2D array down its rows or columns.

Not at all confident
Fully confident
Post-confidence

I can filter an array with a boolean condition, combine two conditions with the symbol & or |, and use np.where to build a new array of transformed values.

Not at all confident
Fully confident
Section 7

7 Your results

Here is how your post-test answers compare with your pre-test answers. The pre/post pairing is the most reliable way to see what this module actually taught you.

Your score

Submit the post-test to see your results.

Muddiest point

What is the one thing from this module that is still unclear to you?

Rate this module

Overall, how would you rate this module?

How likely are you to recommend this module to a peer? (0 = not at all, 10 = extremely likely)