Lesson 21 — Working with Data

Basic Data Handling

Python excels at data handling. The csv module, statistics module, and pandas library cover everything from simple file processing to sophisticated analysis.

PYTHON — CSV 
import csv

# Write CSV:
rows = [
    ["name",    "course",   "mark"],
    ["Thandi",  "Java",     74.5],
    ["Sipho",   "Python",   81.0],
    ["Lerato",  "SQL",      68.0]
]

with open("learners.csv", "w", newline="") as f:
    csv.writer(f).writerows(rows)

# Read CSV as dictionaries:
learners = []
with open("learners.csv") as f:
    for row in csv.DictReader(f):
        learners.append({
            "name":   row["name"],
            "course": row["course"],
            "mark":   float(row["mark"])   # convert from string
        })

for l in learners:
    print(f"{l['name']:<10} {l['course']:<10} {l['mark']:.1f}%")

statistics Module

PYTHON — statistics module 
import statistics

marks = [65, 72, 80, 58, 91, 44, 77, 69, 85, 73]

print(f"Mean:    {statistics.mean(marks):.1f}")
print(f"Median:  {statistics.median(marks)}")
print(f"Mode:    {statistics.mode(marks)}")
print(f"StdDev:  {statistics.stdev(marks):.2f}")
print(f"Variance:{statistics.variance(marks):.2f}")

# Min and max without a library:
print(f"Min: {min(marks)}, Max: {max(marks)}")
pass_rate = sum(1 for m in marks if m >= 50) / len(marks)
print(f"Pass rate: {pass_rate:.0%}")

Introduction to pandas

PYTHON — pandas 
# Install: pip install pandas
import pandas as pd

# Load CSV:
df = pd.read_csv("learners.csv")

# Explore:
print(df.head())       # first 5 rows
print(df.shape)        # (rows, columns)
print(df.dtypes)       # column types
print(df.describe())   # statistical summary

# Access columns:
print(df["mark"].mean())
print(df["mark"].max())

# Filter rows:
passing = df[df["mark"] >= 50]
print(f"Passing: {len(passing)}/{len(df)}")

# Group by course:
course_stats = df.groupby("course")["mark"].agg(["mean","min","max","count"])
print(course_stats)

# Sort:
df_sorted = df.sort_values("mark", ascending=False)
print(df_sorted.head(3))

JSON Handling

PYTHON — JSON 
import json

# Python dict to JSON string:
learner = {"name": "Thandi", "course": "Python", "mark": 74.5}
json_str = json.dumps(learner, indent=2)
print(json_str)

# JSON string to Python dict:
back = json.loads(json_str)
print(back["name"])

# Read/write JSON files:
with open("learner.json", "w") as f:
    json.dump(learner, f, indent=2)

with open("learner.json") as f:
    loaded = json.load(f)
print(loaded)

Practice Task

Your Turn

Create a CSV with 10 learners (name, city, course, mark). Use pandas to: load it, print descriptive statistics, filter to passing learners, find the top 3 per course, compute the pass rate per city, and save results to a new CSV. Then load the original as Python dicts using the csv module and compute the same pass rate without pandas.

Common Mistakes

  • DictReader gives all values as strings — always convert numeric fields explicitly.
  • pandas DataFrames are not lists — use vectorised operations, not for loops.
  • Reading a large CSV all at once — use chunksize parameter for files over a few hundred MB.
  • json.dumps returns a string; json.dump writes to a file. Easy to confuse.

Professional Tip

pandas is the reason Python dominates data science. A single groupby().agg() replaces 20 lines of dictionary manipulation. Install it in every data project.

Mini Quiz

Which Python module provides mean, median, and standard deviation built-in?