pandas Essentials

Data Manipulation in Python

STAT 540 — Computing in Statistics Yen-Yi Ho · Department of Statistics, University of South Carolina

---

Thirty minutes, seven ideas. Everything else in pandas is a variation on these.

| | Idea |---|---| | 1 | Series and DataFrame — labels travel with the data| | 2 | Look before you leap — info, describe, value_counts| | 3 | Selection — loc, iloc, and Boolean masks| | 4 | New columns — assign and map | | 5 | Missing data — isna, dropna, fillna | | 6 | Split–apply–combine — groupby | | 7 | Joining and reshaping — merge, pivot_table |

Three exercises at the end. The long version of this lecture covers each idea in depth, with hierarchical indexing, string methods, reshaping, and the R-to-pandas translation tables.

Why pandas?

It is the data.frame of the Python world: built on NumPy, but designed for heterogeneous tables where row and column labels travel with the data.

import numpy as np
import pandas as pd

pd.set_option("display.max_rows", 10)
pd.set_option("display.precision", 3)
print("pandas", pd.__version__)

pandas 2.3.3

The example dataset

A simulated "restaurant tips" table — one row per check. Building it here means the notebook runs anywhere, and the seed makes it reproducible.

rng = np.random.default_rng(540)
n = 244
tips = pd.DataFrame({
    "total_bill": np.round(rng.gamma(7.0, 2.9, n) + 3.0, 2),
    "day":    rng.choice(["Thur", "Fri", "Sat", "Sun"], n, p=[.25, .08, .36, .31]),
    "time":   rng.choice(["Lunch", "Dinner"], n, p=[.28, .72]),
    "smoker": rng.choice(["No", "Yes"], n, p=[.62, .38]),
    "size":   rng.choice([1, 2, 3, 4, 5, 6], n, p=[.02, .64, .15, .15, .03, .01]),
})
tips["tip"] = np.round(tips["total_bill"] * rng.uniform(.08, .28, n), 2)
tips["tip_pct"] = (tips["tip"] / tips["total_bill"]).round(3)
tips = tips[["total_bill", "tip", "tip_pct", "smoker", "day", "time", "size"]]

tips.head()

	total_bill	tip	tip_pct	smoker	day	time	size
0	23.57	4.21	0.179	No	Sat	Lunch	4
1	17.18	4.03	0.235	Yes	Sun	Dinner	2
2	17.40	3.62	0.208	Yes	Thur	Dinner	2
3	19.85	3.75	0.189	No	Sun	Dinner	4
4	51.79	13.05	0.252	Yes	Thur	Dinner	2

---

1. Series and DataFrame

A Series is a one-dimensional labeled array: values plus an index. The label is a meaningful handle on the value — unlike a plain NumPy array.

s = pd.Series([4, 7, -5, 3], index=["d", "b", "a", "c"])
print(s)
print("\ns['a'] =", s["a"])

d    4
b    7
a   -5
c    3
dtype: int64

s['a'] = -5

Filtering and arithmetic both preserve the index–value link.

print(s[s > 0])
print()
print(s * 2)

d    4
b    7
c    3
dtype: int64

d     8
b    14
a   -10
c     6
dtype: int64

The one behavior that surprises everyone

Operations align on labels, not on position. Labels present in only one Series give NaN. This is pandas' most important behavior, and the source of most early confusion.

s

d    4
b    7
a   -5
c    3
dtype: int64

s2 = pd.Series([10, 20, 30], index=["a", "b", "z"])
s + s2

a     5.0
b    27.0
c     NaN
d     NaN
z     NaN
dtype: float64

The DataFrame

df = pd.DataFrame({"state": ["Ohio", "Ohio", "Nevada"],
                   "year":  [2000, 2001, 2001],
                   "pop":   [1.5, 1.7, 2.4]})
df

	state	year	pop
0	Ohio	2000	1.5
1	Ohio	2001	1.7
2	Nevada	2001	2.4

---

2. Explore Data

Five lines to run on every dataset, every time, before you analyze anything. They take thirty seconds and they will save you hours.

print("shape     :", tips.shape)
print()
print(tips.dtypes)

shape     : (244, 7)

total_bill    float64
tip           float64
tip_pct       float64
smoker         object
day            object
time           object
size            int64
dtype: object

tips.shape

(244, 8)

A numeric column arriving as object is the most common data-loading defect — usually a currency symbol or a thousands separator — and dtypes catches it in one line.

tips.describe().round(2)

	total_bill	tip	tip_pct	size
count	244.00	244.00	244.00	244.00
mean	23.22	4.11	0.18	2.59
std	8.00	1.98	0.06	0.92
min	7.77	0.98	0.08	1.00
25%	17.66	2.77	0.13	2.00
50%	21.93	3.78	0.18	2.00
75%	26.91	5.12	0.23	3.00
max	57.18	13.05	0.28	6.00

tips.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 244 entries, 0 to 243
Data columns (total 8 columns):
 #   Column      Non-Null Count  Dtype  
---  ------      --------------  -----  
 0   total_bill  244 non-null    float64
 1   tip         244 non-null    float64
 2   tip_pct     244 non-null    float64
 3   smoker      244 non-null    object 
 4   day         244 non-null    object 
 5   time        244 non-null    object 
 6   size        244 non-null    int64  
 7   bill_pp     244 non-null    float64
dtypes: float64(4), int64(1), object(3)
memory usage: 15.4+ KB

print(tips["day"].value_counts())
print()
print("missing values:", tips.isna().sum().sum())
print("duplicate rows:", tips.duplicated().sum())

day
Sat     84
Sun     75
Thur    59
Fri     26
Name: count, dtype: int64

missing values: 0
duplicate rows: 0

	Question	One line that answers it
1	How big is it?	df.shape
2	What type is each column?	df.dtypes / df.info()
3	What is missing?	df.isna().sum()
4	Are there repeated rows?	df.duplicated().sum()
5	What values does each category take?	df[c].value_counts()

Which is which

Attribute — no ()	Method — needs ()
.shape	.info()
.dtypes	.describe()
.columns	.head()
.index	.mean()
.values	.sum()
.size	.count()
.ndim	.sort_values()
.empty	.groupby()
.T	.isna()
.loc .iloc .at	.drop_duplicates()

Attributes report the structure; methods do work on the data. shape and dtypes are facts about the frame's skeleton and need no decisions from you. describe() and mean() compute something and accept arguments (numeric_only=, axis=, skipna=) — anything that takes arguments has to be a method.

.loc and .iloc look like exceptions but aren't: they are attributes holding indexer objects, which is why they take square brackets rather than round ones — tips.loc[0, "tip"].

tips.columns

Index(['total_bill', 'tip', 'tip_pct', 'smoker', 'day', 'time', 'size',
       'bill_pp'],
      dtype='object')

tips.index

RangeIndex(start=0, stop=244, step=1)

test=tips.T

test.head()

	0	1	2	3	4	5	6	7	8	9	...	234	235	236	237	238	239	240	241	242	243
total_bill	23.57	17.18	17.4	19.85	51.79	31.13	36.91	22.63	25.58	35.44	...	28.72	15.81	21.9	16.0	40.76	16.84	31.13	30.96	29.33	22.57
tip	4.21	4.03	3.62	3.75	13.05	2.77	9.55	2.61	3.15	3.68	...	6.44	4.17	2.03	4.42	6.9	1.91	6.58	8.55	5.25	5.92
tip_pct	0.179	0.235	0.208	0.189	0.252	0.089	0.259	0.115	0.123	0.104	...	0.224	0.264	0.093	0.276	0.169	0.113	0.211	0.276	0.179	0.262
smoker	No	Yes	Yes	No	Yes	No	Yes	Yes	Yes	Yes	...	No	No	Yes	Yes	Yes	No	Yes	No	Yes	No
day	Sat	Sun	Thur	Sun	Thur	Sat	Sun	Sun	Sat	Sat	...	Sat	Sun	Sat	Sun	Sat	Sat	Sat	Sat	Sun	Sun

5 rows × 244 columns

---

3. Selection

Syntax	Selects by	Typical use
df["col"]	column label	one column
df[["a","b"]]	column labels	several columns
df[mask]	Boolean mask	filtering rows
df.loc[rows, cols]	labels	general, explicit
df.iloc[rows, cols]	integer positions	positional

Rule of thumb: use loc and iloc for anything beyond picking columns. Plain [] becomes ambiguous the moment the index is integer-valued.

print(tips.loc[0:2, ["total_bill", "tip"]])    # labels -- endpoint INCLUDED
print()
print(tips.iloc[0:2, 0:2])                      # positions -- endpoint EXCLUDED

   total_bill   tip
0       23.57  4.21
1       17.18  4.03
2       17.40  3.62

   total_bill   tip
0       23.57  4.21
1       17.18  4.03

Note the difference in the two slices: loc[0:2] returned three rows, iloc[0:2] returned two. Label slices include the endpoint; positional slices follow the usual Python rule.

Boolean filtering

A Boolean Series is the most useful object in pandas: build it once, reuse it anywhere.

big = tips["total_bill"] > 40
print("checks over $40:", big.sum())
tips.loc[big, ["total_bill", "tip", "day", "tip_pct"]].head()

checks over $40: 8

	total_bill	tip	day	tip_pct
4	51.79	13.05	Thur	0.252
10	48.84	8.14	Thur	0.167
27	44.13	6.69	Thur	0.152
34	54.47	10.53	Thur	0.193
62	45.97	12.45	Thur	0.271

Combining conditions

Use & (and), | (or), ~ (not) — not Python's and / or / not.

Parenthesize every condition. & binds more tightly than >, so without the parentheses Python evaluates the wrong thing and raises.

mask = (tips["total_bill"] > 30) & (tips["day"] == "Sun")
print("big Sunday checks:", mask.sum())
print("mean tip_pct     :", tips.loc[mask, "tip_pct"].mean().round(4))

big Sunday checks: 11
mean tip_pct     : 0.1873

The trap: chained assignment

tips[tips["tip"] > 10]["tip_pct"] = 0.5   # WRONG -- may modify a temporary copy
tips.loc[tips["tip"] > 10, "tip_pct"] = 0.5   # RIGHT -- a single .loc call

If you ever see a SettingWithCopyWarning, you have written a chained assignment. Write one loc call instead.

---

4. New Columns

Column arithmetic is vectorized — no loop, and the labels line up automatically.

tips["bill_pp"] = tips["total_bill"] / tips["size"]
tips[["total_bill", "size", "bill_pp"]].head()

	total_bill	size	bill_pp
0	23.57	4	5.893
1	17.18	2	8.590
2	17.40	2	8.700
3	19.85	4	4.963
4	51.79	2	25.895

Several at once with assign

assign returns a new frame, so it chains — and lambda d: lets each new column refer to the frame being built.

out= tips.assign()
out.head()

	total_bill	tip	tip_pct	smoker	day	time	size	bill_pp
0	23.57	4.21	0.179	No	Sat	Lunch	4	5.893
1	17.18	4.03	0.235	Yes	Sun	Dinner	2	8.590
2	17.40	3.62	0.208	Yes	Thur	Dinner	2	8.700
3	19.85	3.75	0.189	No	Sun	Dinner	4	4.963
4	51.79	13.05	0.252	Yes	Thur	Dinner	2	25.895

out.shape

(244, 10)

out = tips.assign(
    is_weekend = lambda d: d["day"].isin(["Sat", "Sun"]),
    bill_q     = lambda d: pd.qcut(d["total_bill"], 4, labels=["Q1","Q2","Q3","Q4"]))
out[["day", "is_weekend", "total_bill", "bill_q"]].head()

	day	is_weekend	total_bill	bill_q
0	Sat	True	23.57	Q3
1	Sun	True	17.18	Q1
2	Thur	False	17.40	Q1
3	Sun	True	19.85	Q2
4	Thur	False	51.79	Q4

out.shape

(244, 10)

Recoding with map

map is the lookup workhorse — R's match-into-a-table idiom in a single call.

kind = {"Thur": "Weekday", "Fri": "Weekday", "Sat": "Weekend", "Sun": "Weekend"}
recoded = tips["day"].map(kind)

print(recoded.value_counts())
print("\nNaNs introduced:", recoded.isna().sum())

day
Weekend    159
Weekday     85
Name: count, dtype: int64

NaNs introduced: 0

The one dangerous property of map: anything missing from the dictionary becomes NaN silently — no error, no warning. A spelling you did not anticipate just disappears from your groupby. Always count the NaNs afterwards, as above.

---

5. Missing Data

Both np.nan and Python's None count as missing.

x = pd.Series([1.0, np.nan, 3.5, None, 7.0])
print(x)
print("\nmissing:", x.isna().sum())

0    1.0
1    NaN
2    3.5
3    NaN
4    7.0
dtype: float64

missing: 2

print("dropna :", x.dropna().tolist())
print("fillna :", x.fillna(0).tolist())
print("ffill  :", x.ffill().tolist())
print("mean   :", x.fillna(round(x.mean(), 2)).tolist())

dropna : [1.0, 3.5, 7.0]
fillna : [1.0, 0.0, 3.5, 0.0, 7.0]
ffill  : [1.0, 1.0, 3.5, 3.5, 7.0]
mean   : [1.0, 3.83, 3.5, 3.83, 7.0]

tips.isna().sum()

total_bill    0
tip           0
tip_pct       0
smoker        0
day           0
time          0
size          0
bill_pp       0
dtype: int64

For a DataFrame, df.isna().sum() gives the count per column. dropna(how="all"), dropna(thresh=2) and fillna({col: value}) cover most of the rest.

df = pd.DataFrame([[1., 6.5, 3.],
                   [1., np.nan, np.nan],
                   [np.nan, np.nan, np.nan],
                   [np.nan, 6.5, 3.]])

df.head()

	0	1	2
0	1.0	6.5	3.0
1	1.0	NaN	NaN
2	NaN	NaN	NaN
3	NaN	6.5	3.0

df.notna().sum(axis="columns") ## colapse by column, sum row-wise non-NA values

0    3
1    1
2    0
3    2
dtype: int64

df.dropna() 

	0	1	2
0	1.0	6.5	3.0

df.dropna(how="all")

	0	1	2
0	1.0	6.5	3.0
1	1.0	NaN	NaN
3	NaN	6.5	3.0

df.dropna(thresh=2)

	0	1	2
0	1.0	6.5	3.0
3	NaN	6.5	3.0

---

6. Split–Apply–Combine

Split the data by a key → apply a function to each piece → combine the results. This is group_by() %>% summarize() from dplyr, or PROC MEANS BY in SAS.

tips.groupby("day")["tip_pct"].mean().round(4)

day
Fri     0.166
Sat     0.172
Sun     0.188
Thur    0.179
Name: tip_pct, dtype: float64

Two keys give a hierarchical index; unstack() turns it into a readable table.

tips.groupby(["day", "smoker"])["tip_pct"].mean().round(3)

day   smoker
Fri   No        0.164
      Yes       0.169
Sat   No        0.174
      Yes       0.169
Sun   No        0.186
      Yes       0.191
Thur  No        0.181
      Yes       0.177
Name: tip_pct, dtype: float64

tips.groupby(["day", "smoker"])["tip_pct"].mean().unstack().round(3)

smoker	No	Yes
day
Fri	0.164	0.169
Sat	0.174	0.169
Sun	0.186	0.191
Thur	0.181	0.177

Named aggregation

Prefer this form. It names each output column, so you never end up parsing a hierarchical column index afterwards.

tips.head()

	total_bill	tip	tip_pct	smoker	day	time	size	bill_pp
0	23.57	4.21	0.179	No	Sat	Lunch	4	5.893
1	17.18	4.03	0.235	Yes	Sun	Dinner	2	8.590
2	17.40	3.62	0.208	Yes	Thur	Dinner	2	8.700
3	19.85	3.75	0.189	No	Sun	Dinner	4	4.963
4	51.79	13.05	0.252	Yes	Thur	Dinner	2	25.895

tips["tip"].size

244

tips.groupby("smoker").agg(
    n        = ("tip", "size"),
    avg_pct  = ("tip_pct", "mean"),
    max_bill = ("total_bill", "max")).round(3)

	n	avg_pct	max_bill
smoker
No	143	0.178	57.18
Yes	101	0.177	51.79

agg vs transform

agg returns one row per group. transform returns one row per original row, broadcasting the group result back — which is exactly what you need for group-centering and within-group standardization.

tips.head()

	total_bill	tip	tip_pct	smoker	day	time	size	bill_pp	day_mean	vs_day
0	23.57	4.21	0.179	No	Sat	Lunch	4	5.893	0.172	0.750
1	17.18	4.03	0.235	Yes	Sun	Dinner	2	8.590	0.188	4.687
2	17.40	3.62	0.208	Yes	Thur	Dinner	2	8.700	0.179	2.853
3	19.85	3.75	0.189	No	Sun	Dinner	4	4.963	0.188	0.087
4	51.79	13.05	0.252	Yes	Thur	Dinner	2	25.895	0.179	7.253

tips["day_mean"] = tips.groupby("day")["tip_pct"].transform("mean")
tips["vs_day"]   = (tips["tip_pct"] - tips["day_mean"])*100
tips[["day", "tip_pct", "day_mean", "vs_day"]].head()

	day	tip_pct	day_mean	vs_day
0	Sat	0.179	0.172	0.750
1	Sun	0.235	0.188	4.687
2	Thur	0.208	0.179	2.853
3	Sun	0.189	0.188	0.087
4	Thur	0.252	0.179	7.253

Always report the group sizes

A mean over 18 observations is not the same kind of number as a mean over 87. Putting size in the agg call gets you n for free.

tips.groupby("day").agg(n=("tip", "size"),
                        avg_pct=("tip_pct", "mean")).round(4)

	n	avg_pct
day
Fri	26	0.166
Sat	84	0.172
Sun	75	0.188
Thur	59	0.179

---

7. Joining and Reshaping

tips.head()

	total_bill	tip	tip_pct	smoker	day	time	size	bill_pp	day_mean	vs_day
0	23.57	4.21	0.179	No	Sat	Lunch	4	5.893	0.172	0.750
1	17.18	4.03	0.235	Yes	Sun	Dinner	2	8.590	0.188	4.687
2	17.40	3.62	0.208	Yes	Thur	Dinner	2	8.700	0.179	2.853
3	19.85	3.75	0.189	No	Sun	Dinner	4	4.963	0.188	0.087
4	51.79	13.05	0.252	Yes	Thur	Dinner	2	25.895	0.179	7.253

tips["day"].value_counts()

day
Sat     84
Sun     75
Thur    59
Fri     26
Name: count, dtype: int64

staff = pd.DataFrame({"day": ["Thur", "Fri", "Sat", "Sun"],
                      "on_shift": [4, 5, 9, 8]})

joined = tips.merge(staff, on="day", how="left")
print("rows before:", len(tips), " after:", len(joined))
joined[["day", "tip", "on_shift"]].head()

rows before: 244  after: 244

	day	tip	on_shift
0	Sat	4.21	9
1	Sun	4.03	8
2	Thur	3.62	4
3	Sun	3.75	8
4	Thur	13.05	4

joined["on_shift"].value_counts()

on_shift
9    84
8    75
4    59
5    26
Name: count, dtype: int64

how=	Keys kept	R
"inner" (default)	in both tables	inner_join
"left"	all of the left table	left_join
"outer"	union of both	full_join

The merge that eats your data

If the right table has duplicate keys, a merge silently adds rows. Nothing warns you.

dup_lookup = pd.DataFrame({"day": ["Thur", "Thur", "Fri", "Sat", "Sun"],
                           "on_shift": [4, 7, 5, 9, 8]})
print("rows before:", len(tips))
print("rows after :", len(tips.merge(dup_lookup, on="day", how="left")))   # grew!

rows before: 244
rows after : 303

# make it fail loudly instead
try:
    tips.merge(dup_lookup, on="day", how="left", validate="m:1")
except Exception as e:
    print(type(e).__name__, "->", e)

MergeError -> Merge keys are not unique in right dataset; not a many-to-one merge

Check lookup["key"].is_unique before you join, or pass validate="m:1" and let pandas refuse. Checking shape before and after every merge is the cheapest insurance in pandas.

pivot_table and crosstab

pivot_table is groupby + unstack in one call. crosstab is the counting special case — R's table().

print(tips.pivot_table(index="day", columns="smoker",
                       values="tip_pct", aggfunc="mean").round(3))
print()
print(pd.crosstab(tips["day"], tips["smoker"], margins=True))

smoker     No    Yes
day                 
Fri     0.164  0.169
Sat     0.174  0.169
Sun     0.186  0.191
Thur    0.181  0.177

smoker   No  Yes  All
day                  
Fri      15   11   26
Sat      45   39   84
Sun      45   30   75
Thur     38   21   59
All     143  101  244

Method chaining

Wrap the chain in parentheses and put each step on its own line — pandas' answer to the |> pipe.

test=tips.query("size >= 4")
test.head()

	total_bill	tip	tip_pct	smoker	day	time	size	bill_pp	day_mean	vs_day
0	23.57	4.21	0.179	No	Sat	Lunch	4	5.893	0.172	0.750
3	19.85	3.75	0.189	No	Sun	Dinner	4	4.963	0.188	0.087
8	25.58	3.15	0.123	Yes	Sat	Dinner	4	6.395	0.172	-4.850
9	35.44	3.68	0.104	Yes	Sat	Lunch	4	8.860	0.172	-6.750
11	29.26	6.13	0.210	Yes	Sun	Dinner	4	7.315	0.188	2.187

(tips
   .query("size >= 2")
   .groupby("day")
   .agg(n=("tip", "size"), avg=("tip_pct", "mean"))
   .sort_values("avg", ascending=False)
   .round(3))

	n	avg
day
Sun	74	0.189
Thur	59	0.179
Fri	24	0.172
Sat	84	0.172

---

Wrapping Up

pandas and dplyr side by side

dplyr / tidyr	pandas
filter(df, x > 1)	df[df["x"] > 1]
select(df, a, b)	df[["a", "b"]]
mutate(df, z = x + y)	df.assign(z=df["x"] + df["y"])
arrange(df, desc(x))	df.sort_values("x", ascending=False)
group_by(g) %>% summarize(m = mean(x))	df.groupby("g")["x"].mean()
left_join(a, b, by = "k")	pd.merge(a, b, on="k", how="left")
distinct(df)	df.drop_duplicates()
count(df, g)	df["g"].value_counts()
x %in% y	s.isin(y)

The Five Silent Failures

1. Alignment. Operations match on labels, not position. Two Series with different indexes give you NaN, not an error.
2. Chained assignment. df[mask]["col"] = x may do nothing. Write one .loc call.
3. and / or. Use &, |, ~, and parenthesize each condition.
4. Silent NaN from map. A key missing from the dictionary becomes missing data with no warning. Count afterwards.
5. Merges that grow. Duplicate keys multiply rows silently. Check shape before and after, every time.

---

Your Turn

Exercise 1. Select the checks where size is at least 4 and time is "Dinner". How many are there, and what is their mean tip_pct?

(tips
    .query("size>=4")
    .groupby("time")
    .agg(n=("tip", "size"), avg=("tip_pct", "mean")))
# your code here

	n	avg
time
Dinner	34	0.187
Lunch	16	0.159

Exercise 2. Add a column bill_pp = total_bill / size. Then report, for each day, the number of checks and the mean bill_pp — in a single groupby().agg() call with named output columns.

tips.head()

	total_bill	tip	tip_pct	smoker	day	time	size	bill_pp	day_mean	vs_day
0	23.57	4.21	0.179	No	Sat	Lunch	4	5.893	0.172	0.750
1	17.18	4.03	0.235	Yes	Sun	Dinner	2	8.590	0.188	4.687
2	17.40	3.62	0.208	Yes	Thur	Dinner	2	8.700	0.179	2.853
3	19.85	3.75	0.189	No	Sun	Dinner	4	4.963	0.188	0.087
4	51.79	13.05	0.252	Yes	Thur	Dinner	2	25.895	0.179	7.253

(tips
    .assign(bill_pp=lambda d: d["total_bill"]/d["size"])
    .groupby("day").agg(n=("tip", "size"), avg_pp=("bill_pp", "mean")))

	n	avg_pp
day
Fri	26	9.772
Sat	84	9.400
Sun	75	10.221
Thur	59	10.392

Exercise 3. Build a lookup table mapping each day to "Weekend" or "Weekday". Attach it to tips twice — once with map, once with merge — and confirm that the row count did not change.

look_up=pd.DataFrame({"day": ["Thur", "Fri", "Sat", "Sun"],
                           "isWeekend": [False, False, True, True]})
look_up.head()

	day	isWeekend
0	Thur	False
1	Fri	False
2	Sat	True
3	Sun	True

look_up.set_index("day")

	isWeekend
day
Thur	False
Fri	False
Sat	True
Sun	True

look_up=pd.DataFrame({"day": ["Thur", "Fri", "Sat", "Sun"],
                           "isWeekend": [False, False, True, True]})
key=look_up.set_index("day")["isWeekend"]
print("row before map", len(tips))
tips["isWeekend"]=tips["day"].map(key)
print("row after map", len(tips))
tips.head()

row before map 244
row after map 244

	total_bill	tip	tip_pct	smoker	day	time	size	bill_pp	day_mean	vs_day	isWeekend
0	23.57	4.21	0.179	No	Sat	Lunch	4	5.893	0.172	0.750	True
1	17.18	4.03	0.235	Yes	Sun	Dinner	2	8.590	0.188	4.687	True
2	17.40	3.62	0.208	Yes	Thur	Dinner	2	8.700	0.179	2.853	False
3	19.85	3.75	0.189	No	Sun	Dinner	4	4.963	0.188	0.087	True
4	51.79	13.05	0.252	Yes	Thur	Dinner	2	25.895	0.179	7.253	False

tips_merge=tips.merge(look_up, on="day", how="left")
tips_merge.head()

	total_bill	tip	tip_pct	smoker	day	time	size	bill_pp	day_mean	vs_day	isWeekend_x	isWeekend_y
0	23.57	4.21	0.179	No	Sat	Lunch	4	5.893	0.172	0.750	True	True
1	17.18	4.03	0.235	Yes	Sun	Dinner	2	8.590	0.188	4.687	True	True
2	17.40	3.62	0.208	Yes	Thur	Dinner	2	8.700	0.179	2.853	False	False
3	19.85	3.75	0.189	No	Sun	Dinner	4	4.963	0.188	0.087	True	True
4	51.79	13.05	0.252	Yes	Thur	Dinner	2	25.895	0.179	7.253	False	False

---

Where to go next

• McKinney, W. Python for Data Analysis, 3rd ed. — https://wesmckinney.com/book/ — Ch. 5 getting started · Ch. 7 cleaning · Ch. 8 join/reshape · Ch. 10 group operations
• VanderPlas, J. Python Data Science Handbook, Ch. 3 — https://jakevdp.github.io/PythonDataScienceHandbook/
• pandas user guide
• Comparison with R

The long version of this lecture covers each of these ideas in depth, plus hierarchical indexing, stack/unstack, string methods, categoricals, and the full R-to-pandas translation tables.