Question: How do transformations and classification methods affect our representation of the same values?
Concepts: Min-max normalisation, z-scores, quantile and equal-interval classes. Data: 2023 HDB teaching extract.
Try the tasks before opening the companion solution. Class membership is sensitive to the scheme and should not be confused with intrinsic groups in the data.
After you attempt the exercises: Open the complete solution notebook on GitHub.
from pathlib import Path
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# Works in a repository checkout and remotely (including Colab).
name = "hdb_data_2023.csv.xz"
candidates = [
Path("../datasets/singapore_hdb_ura") / name,
Path("src/content/datasets/singapore_hdb_ura") / name,
Path("datasets/singapore_hdb_ura") / name,
]
source = next((p for p in candidates if p.is_file()), "https://raw.githubusercontent.com/wcchin/geo-stat-vis/8cf3ddb0a7eff7d94018940e91f32d7482f32aaa/src/content/datasets/singapore_hdb_ura/" + name)
df = pd.read_csv(source, index_col=0)
assert {"resale_price", "resale_age"}.issubset(df.columns)
print(f"Loaded {len(df):,} records from {source}")
df[["resale_price", "resale_age"]].describe()1. Reference summaries¶
Start by checking price range and distribution.
price = df["resale_price"].dropna().astype(float)
print(price.describe())
print("Missing values:", df["resale_price"].isna().sum())2. Exercise — normalisation¶
TODO: Compute (a) min-max normalisation to [0, 1] and (b) standard scores. Verify the minimum/maximum of the first and the mean/standard deviation of the second.
# TODO: create price_minmax and price_z using `price`.
3. Exercise — classification¶
TODO: Classify resale prices into 5 classes using both equal-width intervals and quantiles. Compare counts per class. Why are these not the same?
# TODO: use pd.cut and pd.qcut (duplicates='drop') and compare value_counts().
Interpretation¶
Explain one circumstance in which a quantile map could mislead readers about differences in actual prices. You do not need geometry to answer this representation question.
Check your understanding¶
Try the exercises and write down your interpretation before comparing your work with the complete answers.