{ "cells": [ { "cell_type": "markdown", "id": "e000", "metadata": {}, "source": [ "# pandas Essentials\n", "## Data Manipulation in Python\n", "\n", "**STAT 540 — Computing in Statistics**\n", "Yen-Yi Ho · Department of Statistics, University of South Carolina\n", "\n", "---\n", "\n", "Thirty minutes, seven ideas. Everything else in pandas is a variation on these.\n", "\n", "| | Idea \n", "|---|---|\n", "| 1 | Series and DataFrame — labels travel with the data| \n", "| 2 | Look before you leap — `info`, `describe`, `value_counts`| \n", "| 3 | Selection — `loc`, `iloc`, and Boolean masks|\n", "| 4 | New columns — `assign` and `map` | \n", "| 5 | Missing data — `isna`, `dropna`, `fillna` | \n", "| 6 | Split–apply–combine — `groupby` | \n", "| 7 | Joining and reshaping — `merge`, `pivot_table` | \n", "\n", "Three exercises at the end. The **long version** of this lecture covers each idea in\n", "depth, with hierarchical indexing, string methods, reshaping, and the R-to-pandas\n", "translation tables.\n", "\n", "### Why pandas?\n", "\n", "It is the `data.frame` of the Python world: built on NumPy, but designed for\n", "**heterogeneous** tables where row and column **labels travel with the data**." ] }, { "cell_type": "code", "execution_count": 3, "id": "e001", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pandas 2.3.3\n" ] } ], "source": [ "import numpy as np\n", "import pandas as pd\n", "\n", "pd.set_option(\"display.max_rows\", 10)\n", "pd.set_option(\"display.precision\", 3)\n", "print(\"pandas\", pd.__version__)" ] }, { "cell_type": "markdown", "id": "e002", "metadata": {}, "source": [ "### The example dataset\n", "\n", "A simulated \"restaurant tips\" table — one row per check. Building it here means the\n", "notebook runs anywhere, and the seed makes it reproducible." ] }, { "cell_type": "code", "execution_count": 4, "id": "e003", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
total_billtiptip_pctsmokerdaytimesize
023.574.210.179NoSatLunch4
117.184.030.235YesSunDinner2
217.403.620.208YesThurDinner2
319.853.750.189NoSunDinner4
451.7913.050.252YesThurDinner2
\n", "
" ], "text/plain": [ " total_bill tip tip_pct smoker day time size\n", "0 23.57 4.21 0.179 No Sat Lunch 4\n", "1 17.18 4.03 0.235 Yes Sun Dinner 2\n", "2 17.40 3.62 0.208 Yes Thur Dinner 2\n", "3 19.85 3.75 0.189 No Sun Dinner 4\n", "4 51.79 13.05 0.252 Yes Thur Dinner 2" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rng = np.random.default_rng(540)\n", "n = 244\n", "tips = pd.DataFrame({\n", " \"total_bill\": np.round(rng.gamma(7.0, 2.9, n) + 3.0, 2),\n", " \"day\": rng.choice([\"Thur\", \"Fri\", \"Sat\", \"Sun\"], n, p=[.25, .08, .36, .31]),\n", " \"time\": rng.choice([\"Lunch\", \"Dinner\"], n, p=[.28, .72]),\n", " \"smoker\": rng.choice([\"No\", \"Yes\"], n, p=[.62, .38]),\n", " \"size\": rng.choice([1, 2, 3, 4, 5, 6], n, p=[.02, .64, .15, .15, .03, .01]),\n", "})\n", "tips[\"tip\"] = np.round(tips[\"total_bill\"] * rng.uniform(.08, .28, n), 2)\n", "tips[\"tip_pct\"] = (tips[\"tip\"] / tips[\"total_bill\"]).round(3)\n", "tips = tips[[\"total_bill\", \"tip\", \"tip_pct\", \"smoker\", \"day\", \"time\", \"size\"]]\n", "\n", "tips.head()" ] }, { "cell_type": "markdown", "id": "e004", "metadata": {}, "source": [ "---\n", "# 1. Series and DataFrame\n", "\n", "A **Series** is a one-dimensional labeled array: values plus an **index**. The label is a\n", "meaningful handle on the value — unlike a plain NumPy array." ] }, { "cell_type": "code", "execution_count": 5, "id": "e005", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "d 4\n", "b 7\n", "a -5\n", "c 3\n", "dtype: int64\n", "\n", "s['a'] = -5\n" ] } ], "source": [ "s = pd.Series([4, 7, -5, 3], index=[\"d\", \"b\", \"a\", \"c\"])\n", "print(s)\n", "print(\"\\ns['a'] =\", s[\"a\"])" ] }, { "cell_type": "markdown", "id": "e006", "metadata": {}, "source": [ "Filtering and arithmetic both **preserve the index–value link**." ] }, { "cell_type": "code", "execution_count": 4, "id": "e007", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "d 4\n", "b 7\n", "c 3\n", "dtype: int64\n", "\n", "d 8\n", "b 14\n", "a -10\n", "c 6\n", "dtype: int64\n" ] } ], "source": [ "print(s[s > 0])\n", "print()\n", "print(s * 2)" ] }, { "cell_type": "markdown", "id": "e008", "metadata": {}, "source": [ "### The one behavior that surprises everyone\n", "\n", "**Operations align on labels, not on position.** Labels present in only one Series give\n", "`NaN`. This is pandas' most important behavior, and the source of most early confusion." ] }, { "cell_type": "code", "execution_count": 6, "id": "8d277152-0d1b-4dd0-8e9d-3e0d39af0aee", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "d 4\n", "b 7\n", "a -5\n", "c 3\n", "dtype: int64" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s" ] }, { "cell_type": "code", "execution_count": 5, "id": "e009", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "a 5.0\n", "b 27.0\n", "c NaN\n", "d NaN\n", "z NaN\n", "dtype: float64" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s2 = pd.Series([10, 20, 30], index=[\"a\", \"b\", \"z\"])\n", "s + s2" ] }, { "cell_type": "markdown", "id": "e010", "metadata": {}, "source": [ "### The DataFrame" ] }, { "cell_type": "code", "execution_count": 7, "id": "e011", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
stateyearpop
0Ohio20001.5
1Ohio20011.7
2Nevada20012.4
\n", "
" ], "text/plain": [ " state year pop\n", "0 Ohio 2000 1.5\n", "1 Ohio 2001 1.7\n", "2 Nevada 2001 2.4" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.DataFrame({\"state\": [\"Ohio\", \"Ohio\", \"Nevada\"],\n", " \"year\": [2000, 2001, 2001],\n", " \"pop\": [1.5, 1.7, 2.4]})\n", "df" ] }, { "cell_type": "markdown", "id": "e012", "metadata": {}, "source": [ "---\n", "# 2. Explore Data\n", "\n", "Five lines to run on **every** dataset, every time, before you analyze anything. They\n", "take thirty seconds and they will save you hours." ] }, { "cell_type": "code", "execution_count": 6, "id": "e013", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "shape : (244, 7)\n", "\n", "total_bill float64\n", "tip float64\n", "tip_pct float64\n", "smoker object\n", "day object\n", "time object\n", "size int64\n", "dtype: object\n" ] } ], "source": [ "print(\"shape :\", tips.shape)\n", "print()\n", "print(tips.dtypes)" ] }, { "cell_type": "code", "execution_count": 34, "id": "b6101a16-db78-483e-adb6-e13f0d2bc2b3", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(244, 8)" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips.shape" ] }, { "cell_type": "markdown", "id": "e014", "metadata": {}, "source": [ "A numeric column arriving as `object` is the most common data-loading defect — usually a\n", "currency symbol or a thousands separator — and `dtypes` catches it in one line." ] }, { "cell_type": "code", "execution_count": 9, "id": "e015", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
total_billtiptip_pctsize
count244.00244.00244.00244.00
mean23.224.110.182.59
std8.001.980.060.92
min7.770.980.081.00
25%17.662.770.132.00
50%21.933.780.182.00
75%26.915.120.233.00
max57.1813.050.286.00
\n", "
" ], "text/plain": [ " total_bill tip tip_pct size\n", "count 244.00 244.00 244.00 244.00\n", "mean 23.22 4.11 0.18 2.59\n", "std 8.00 1.98 0.06 0.92\n", "min 7.77 0.98 0.08 1.00\n", "25% 17.66 2.77 0.13 2.00\n", "50% 21.93 3.78 0.18 2.00\n", "75% 26.91 5.12 0.23 3.00\n", "max 57.18 13.05 0.28 6.00" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips.describe().round(2)" ] }, { "cell_type": "code", "execution_count": 31, "id": "3c2bc5f1-e872-47f2-9b1e-bb0eece87f58", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 244 entries, 0 to 243\n", "Data columns (total 8 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 total_bill 244 non-null float64\n", " 1 tip 244 non-null float64\n", " 2 tip_pct 244 non-null float64\n", " 3 smoker 244 non-null object \n", " 4 day 244 non-null object \n", " 5 time 244 non-null object \n", " 6 size 244 non-null int64 \n", " 7 bill_pp 244 non-null float64\n", "dtypes: float64(4), int64(1), object(3)\n", "memory usage: 15.4+ KB\n" ] } ], "source": [ "tips.info()" ] }, { "cell_type": "code", "execution_count": 9, "id": "e016", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "day\n", "Sat 84\n", "Sun 75\n", "Thur 59\n", "Fri 26\n", "Name: count, dtype: int64\n", "\n", "missing values: 0\n", "duplicate rows: 0\n" ] } ], "source": [ "print(tips[\"day\"].value_counts())\n", "print()\n", "print(\"missing values:\", tips.isna().sum().sum())\n", "print(\"duplicate rows:\", tips.duplicated().sum())" ] }, { "cell_type": "markdown", "id": "e017", "metadata": {}, "source": [ "| | Question | One line that answers it |\n", "|---|---|---|\n", "| 1 | How big is it? | `df.shape` |\n", "| 2 | What type is each column? | `df.dtypes` / `df.info()` |\n", "| 3 | What is missing? | `df.isna().sum()` |\n", "| 4 | Are there repeated rows? | `df.duplicated().sum()` |\n", "| 5 | What values does each category take? | `df[c].value_counts()` |" ] }, { "cell_type": "markdown", "id": "e013300c-e9e3-4fdf-af2c-92c77322a5ad", "metadata": {}, "source": [ "### Which is which\n", "\n", "| Attribute — no `()` | Method — needs `()` |\n", "|:---|:---|\n", "| `.shape` | `.info()` |\n", "| `.dtypes` | `.describe()` |\n", "| `.columns` | `.head()` |\n", "| `.index` | `.mean()` |\n", "| `.values` | `.sum()` |\n", "| `.size` | `.count()` |\n", "| `.ndim` | `.sort_values()` |\n", "| `.empty` | `.groupby()` |\n", "| `.T` | `.isna()` |\n", "| `.loc` `.iloc` `.at` | `.drop_duplicates()` |\n", "\n", "**Attributes report the structure; methods do work on the data.** `shape` and `dtypes` are\n", "facts about the frame's skeleton and need no decisions from you. `describe()` and `mean()`\n", "compute something and accept arguments (`numeric_only=`, `axis=`, `skipna=`) — anything\n", "that takes arguments has to be a method.\n", "\n", "`.loc` and `.iloc` look like exceptions but aren't: they are attributes holding *indexer\n", "objects*, which is why they take **square** brackets rather than round ones —\n", "`tips.loc[0, \"tip\"]`." ] }, { "cell_type": "code", "execution_count": 36, "id": "c04f310c-fc48-4da0-866d-6a6d3d688e8c", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Index(['total_bill', 'tip', 'tip_pct', 'smoker', 'day', 'time', 'size',\n", " 'bill_pp'],\n", " dtype='object')" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips.columns" ] }, { "cell_type": "code", "execution_count": 37, "id": "61948b02-c569-45f7-b049-a50be640dbc4", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "RangeIndex(start=0, stop=244, step=1)" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips.index" ] }, { "cell_type": "code", "execution_count": 39, "id": "3420bfe8-4498-4b5d-934d-d7ccc5d99d99", "metadata": {}, "outputs": [], "source": [ "test=tips.T" ] }, { "cell_type": "code", "execution_count": 41, "id": "86f267e0-9faa-4e0d-b826-be44d00e5081", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
0123456789...234235236237238239240241242243
total_bill23.5717.1817.419.8551.7931.1336.9122.6325.5835.44...28.7215.8121.916.040.7616.8431.1330.9629.3322.57
tip4.214.033.623.7513.052.779.552.613.153.68...6.444.172.034.426.91.916.588.555.255.92
tip_pct0.1790.2350.2080.1890.2520.0890.2590.1150.1230.104...0.2240.2640.0930.2760.1690.1130.2110.2760.1790.262
smokerNoYesYesNoYesNoYesYesYesYes...NoNoYesYesYesNoYesNoYesNo
daySatSunThurSunThurSatSunSunSatSat...SatSunSatSunSatSatSatSatSunSun
\n", "

5 rows × 244 columns

\n", "
" ], "text/plain": [ " 0 1 2 3 4 5 6 7 8 \\\n", "total_bill 23.57 17.18 17.4 19.85 51.79 31.13 36.91 22.63 25.58 \n", "tip 4.21 4.03 3.62 3.75 13.05 2.77 9.55 2.61 3.15 \n", "tip_pct 0.179 0.235 0.208 0.189 0.252 0.089 0.259 0.115 0.123 \n", "smoker No Yes Yes No Yes No Yes Yes Yes \n", "day Sat Sun Thur Sun Thur Sat Sun Sun Sat \n", "\n", " 9 ... 234 235 236 237 238 239 240 \\\n", "total_bill 35.44 ... 28.72 15.81 21.9 16.0 40.76 16.84 31.13 \n", "tip 3.68 ... 6.44 4.17 2.03 4.42 6.9 1.91 6.58 \n", "tip_pct 0.104 ... 0.224 0.264 0.093 0.276 0.169 0.113 0.211 \n", "smoker Yes ... No No Yes Yes Yes No Yes \n", "day Sat ... Sat Sun Sat Sun Sat Sat Sat \n", "\n", " 241 242 243 \n", "total_bill 30.96 29.33 22.57 \n", "tip 8.55 5.25 5.92 \n", "tip_pct 0.276 0.179 0.262 \n", "smoker No Yes No \n", "day Sat Sun Sun \n", "\n", "[5 rows x 244 columns]" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test.head()" ] }, { "cell_type": "markdown", "id": "e018", "metadata": {}, "source": [ "---\n", "# 3. Selection\n", "\n", "| Syntax | Selects by | Typical use |\n", "|---|---|---|\n", "| `df[\"col\"]` | column label | one column |\n", "| `df[[\"a\",\"b\"]]` | column labels | several columns |\n", "| `df[mask]` | Boolean mask | filtering rows |\n", "| `df.loc[rows, cols]` | **labels** | general, explicit |\n", "| `df.iloc[rows, cols]` | **integer positions** | positional |\n", "\n", "**Rule of thumb:** use `loc` and `iloc` for anything beyond picking columns. Plain `[]`\n", "becomes ambiguous the moment the index is integer-valued." ] }, { "cell_type": "code", "execution_count": 11, "id": "e019", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " total_bill tip\n", "0 23.57 4.21\n", "1 17.18 4.03\n", "2 17.40 3.62\n", "\n", " total_bill tip\n", "0 23.57 4.21\n", "1 17.18 4.03\n" ] } ], "source": [ "print(tips.loc[0:2, [\"total_bill\", \"tip\"]]) # labels -- endpoint INCLUDED\n", "print()\n", "print(tips.iloc[0:2, 0:2]) # positions -- endpoint EXCLUDED" ] }, { "cell_type": "markdown", "id": "e020", "metadata": {}, "source": [ "Note the difference in the two slices: `loc[0:2]` returned **three** rows, `iloc[0:2]`\n", "returned **two**. Label slices include the endpoint; positional slices follow the usual\n", "Python rule.\n", "\n", "### Boolean filtering\n", "\n", "A Boolean Series is the most useful object in pandas: build it once, reuse it anywhere." ] }, { "cell_type": "code", "execution_count": 14, "id": "e021", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "checks over $40: 8\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
total_billtipdaytip_pct
451.7913.05Thur0.252
1048.848.14Thur0.167
2744.136.69Thur0.152
3454.4710.53Thur0.193
6245.9712.45Thur0.271
\n", "
" ], "text/plain": [ " total_bill tip day tip_pct\n", "4 51.79 13.05 Thur 0.252\n", "10 48.84 8.14 Thur 0.167\n", "27 44.13 6.69 Thur 0.152\n", "34 54.47 10.53 Thur 0.193\n", "62 45.97 12.45 Thur 0.271" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "big = tips[\"total_bill\"] > 40\n", "print(\"checks over $40:\", big.sum())\n", "tips.loc[big, [\"total_bill\", \"tip\", \"day\", \"tip_pct\"]].head()" ] }, { "cell_type": "markdown", "id": "e022", "metadata": {}, "source": [ "### Combining conditions\n", "\n", "Use `&` (and), `|` (or), `~` (not) — **not** Python's `and` / `or` / `not`.\n", "\n", "**Parenthesize every condition.** `&` binds more tightly than `>`, so without the\n", "parentheses Python evaluates the wrong thing and raises." ] }, { "cell_type": "code", "execution_count": 15, "id": "e023", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "big Sunday checks: 11\n", "mean tip_pct : 0.1873\n" ] } ], "source": [ "mask = (tips[\"total_bill\"] > 30) & (tips[\"day\"] == \"Sun\")\n", "print(\"big Sunday checks:\", mask.sum())\n", "print(\"mean tip_pct :\", tips.loc[mask, \"tip_pct\"].mean().round(4))" ] }, { "cell_type": "markdown", "id": "e024", "metadata": {}, "source": [ "### The trap: chained assignment\n", "\n", "```python\n", "tips[tips[\"tip\"] > 10][\"tip_pct\"] = 0.5 # WRONG -- may modify a temporary copy\n", "tips.loc[tips[\"tip\"] > 10, \"tip_pct\"] = 0.5 # RIGHT -- a single .loc call\n", "```\n", "\n", "If you ever see a `SettingWithCopyWarning`, you have written a chained assignment.\n", "Write one `loc` call instead." ] }, { "cell_type": "markdown", "id": "e025", "metadata": {}, "source": [ "---\n", "# 4. New Columns\n", "\n", "Column arithmetic is vectorized — no loop, and the labels line up automatically." ] }, { "cell_type": "code", "execution_count": 17, "id": "e026", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
total_billsizebill_pp
023.5745.893
117.1828.590
217.4028.700
319.8544.963
451.79225.895
\n", "
" ], "text/plain": [ " total_bill size bill_pp\n", "0 23.57 4 5.893\n", "1 17.18 2 8.590\n", "2 17.40 2 8.700\n", "3 19.85 4 4.963\n", "4 51.79 2 25.895" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips[\"bill_pp\"] = tips[\"total_bill\"] / tips[\"size\"]\n", "tips[[\"total_bill\", \"size\", \"bill_pp\"]].head()" ] }, { "cell_type": "markdown", "id": "e027", "metadata": {}, "source": [ "### Several at once with `assign`\n", "\n", "`assign` returns a **new** frame, so it chains — and `lambda d:` lets each new column\n", "refer to the frame being built." ] }, { "cell_type": "code", "execution_count": 25, "id": "5c844170-c41c-461d-a7b4-74be47f8a419", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
total_billtiptip_pctsmokerdaytimesizebill_pp
023.574.210.179NoSatLunch45.893
117.184.030.235YesSunDinner28.590
217.403.620.208YesThurDinner28.700
319.853.750.189NoSunDinner44.963
451.7913.050.252YesThurDinner225.895
\n", "
" ], "text/plain": [ " total_bill tip tip_pct smoker day time size bill_pp\n", "0 23.57 4.21 0.179 No Sat Lunch 4 5.893\n", "1 17.18 4.03 0.235 Yes Sun Dinner 2 8.590\n", "2 17.40 3.62 0.208 Yes Thur Dinner 2 8.700\n", "3 19.85 3.75 0.189 No Sun Dinner 4 4.963\n", "4 51.79 13.05 0.252 Yes Thur Dinner 2 25.895" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "out= tips.assign()\n", "out.head()" ] }, { "cell_type": "code", "execution_count": 29, "id": "5288b8bf-66ed-481b-9b83-4956ef041ae3", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(244, 10)" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "out.shape" ] }, { "cell_type": "code", "execution_count": 27, "id": "e028", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
dayis_weekendtotal_billbill_q
0SatTrue23.57Q3
1SunTrue17.18Q1
2ThurFalse17.40Q1
3SunTrue19.85Q2
4ThurFalse51.79Q4
\n", "
" ], "text/plain": [ " day is_weekend total_bill bill_q\n", "0 Sat True 23.57 Q3\n", "1 Sun True 17.18 Q1\n", "2 Thur False 17.40 Q1\n", "3 Sun True 19.85 Q2\n", "4 Thur False 51.79 Q4" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "out = tips.assign(\n", " is_weekend = lambda d: d[\"day\"].isin([\"Sat\", \"Sun\"]),\n", " bill_q = lambda d: pd.qcut(d[\"total_bill\"], 4, labels=[\"Q1\",\"Q2\",\"Q3\",\"Q4\"]))\n", "out[[\"day\", \"is_weekend\", \"total_bill\", \"bill_q\"]].head()" ] }, { "cell_type": "code", "execution_count": 28, "id": "1b91a0d4-ed8b-4974-b839-dac3c57b455d", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(244, 10)" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "out.shape" ] }, { "cell_type": "markdown", "id": "e029", "metadata": {}, "source": [ "### Recoding with `map`\n", "\n", "`map` is the lookup workhorse — R's `match`-into-a-table idiom in a single call." ] }, { "cell_type": "code", "execution_count": 35, "id": "e030", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "day\n", "Weekend 159\n", "Weekday 85\n", "Name: count, dtype: int64\n", "\n", "NaNs introduced: 0\n" ] } ], "source": [ "kind = {\"Thur\": \"Weekday\", \"Fri\": \"Weekday\", \"Sat\": \"Weekend\", \"Sun\": \"Weekend\"}\n", "recoded = tips[\"day\"].map(kind)\n", "\n", "print(recoded.value_counts())\n", "print(\"\\nNaNs introduced:\", recoded.isna().sum())" ] }, { "cell_type": "markdown", "id": "e031", "metadata": {}, "source": [ "> **The one dangerous property of `map`:** anything missing from the dictionary becomes\n", "> `NaN` **silently** — no error, no warning. A spelling you did not anticipate just\n", "> disappears from your `groupby`. Always count the `NaN`s afterwards, as above." ] }, { "cell_type": "markdown", "id": "e032", "metadata": {}, "source": [ "---\n", "# 5. Missing Data\n", "\n", "Both `np.nan` and Python's `None` count as missing." ] }, { "cell_type": "code", "execution_count": 42, "id": "e033", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0 1.0\n", "1 NaN\n", "2 3.5\n", "3 NaN\n", "4 7.0\n", "dtype: float64\n", "\n", "missing: 2\n" ] } ], "source": [ "x = pd.Series([1.0, np.nan, 3.5, None, 7.0])\n", "print(x)\n", "print(\"\\nmissing:\", x.isna().sum())" ] }, { "cell_type": "code", "execution_count": 43, "id": "e034", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dropna : [1.0, 3.5, 7.0]\n", "fillna : [1.0, 0.0, 3.5, 0.0, 7.0]\n", "ffill : [1.0, 1.0, 3.5, 3.5, 7.0]\n", "mean : [1.0, 3.83, 3.5, 3.83, 7.0]\n" ] } ], "source": [ "print(\"dropna :\", x.dropna().tolist())\n", "print(\"fillna :\", x.fillna(0).tolist())\n", "print(\"ffill :\", x.ffill().tolist())\n", "print(\"mean :\", x.fillna(round(x.mean(), 2)).tolist())" ] }, { "cell_type": "code", "execution_count": 44, "id": "d1df5c92-3afe-4754-9a34-026ce2e1024e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "total_bill 0\n", "tip 0\n", "tip_pct 0\n", "smoker 0\n", "day 0\n", "time 0\n", "size 0\n", "bill_pp 0\n", "dtype: int64" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips.isna().sum()" ] }, { "cell_type": "markdown", "id": "e035", "metadata": {}, "source": [ "For a DataFrame, `df.isna().sum()` gives the count per column. `dropna(how=\"all\")`,\n", "`dropna(thresh=2)` and `fillna({col: value})` cover most of the rest." ] }, { "cell_type": "code", "execution_count": 45, "id": "2b680d98-1337-455c-837d-aada8442fabe", "metadata": {}, "outputs": [], "source": [ "df = pd.DataFrame([[1., 6.5, 3.],\n", " [1., np.nan, np.nan],\n", " [np.nan, np.nan, np.nan],\n", " [np.nan, 6.5, 3.]])" ] }, { "cell_type": "code", "execution_count": 46, "id": "f1cf5fad-5dd6-46e7-bcd3-286c87f832ea", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
012
01.06.53.0
11.0NaNNaN
2NaNNaNNaN
3NaN6.53.0
\n", "
" ], "text/plain": [ " 0 1 2\n", "0 1.0 6.5 3.0\n", "1 1.0 NaN NaN\n", "2 NaN NaN NaN\n", "3 NaN 6.5 3.0" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head()" ] }, { "cell_type": "code", "execution_count": 47, "id": "d8f5d6f0-d6e3-4a55-baa0-79ce9972e098", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 3\n", "1 1\n", "2 0\n", "3 2\n", "dtype: int64" ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.notna().sum(axis=\"columns\") ## colapse by column, sum row-wise non-NA values" ] }, { "cell_type": "code", "execution_count": 48, "id": "ff6ad577-f4af-4f06-960b-24c844ed5fc3", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
012
01.06.53.0
\n", "
" ], "text/plain": [ " 0 1 2\n", "0 1.0 6.5 3.0" ] }, "execution_count": 48, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.dropna() " ] }, { "cell_type": "code", "execution_count": 49, "id": "7ffcd786-2a28-4ce1-9d50-05c262ed1aec", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
012
01.06.53.0
11.0NaNNaN
3NaN6.53.0
\n", "
" ], "text/plain": [ " 0 1 2\n", "0 1.0 6.5 3.0\n", "1 1.0 NaN NaN\n", "3 NaN 6.5 3.0" ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.dropna(how=\"all\")" ] }, { "cell_type": "code", "execution_count": 50, "id": "a9391e34-91bc-47be-8b87-1bbf4404dff6", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
012
01.06.53.0
3NaN6.53.0
\n", "
" ], "text/plain": [ " 0 1 2\n", "0 1.0 6.5 3.0\n", "3 NaN 6.5 3.0" ] }, "execution_count": 50, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.dropna(thresh=2)" ] }, { "cell_type": "markdown", "id": "e036", "metadata": {}, "source": [ "---\n", "# 6. Split–Apply–Combine\n", "\n", "**Split** the data by a key → **apply** a function to each piece → **combine** the\n", "results. This is `group_by() %>% summarize()` from dplyr, or `PROC MEANS BY` in SAS." ] }, { "cell_type": "code", "execution_count": 51, "id": "e037", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "day\n", "Fri 0.166\n", "Sat 0.172\n", "Sun 0.188\n", "Thur 0.179\n", "Name: tip_pct, dtype: float64" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips.groupby(\"day\")[\"tip_pct\"].mean().round(4)" ] }, { "cell_type": "markdown", "id": "e038", "metadata": {}, "source": [ "Two keys give a hierarchical index; `unstack()` turns it into a readable table." ] }, { "cell_type": "code", "execution_count": 53, "id": "55f09067-b17d-4119-a994-712e0027e6c7", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "day smoker\n", "Fri No 0.164\n", " Yes 0.169\n", "Sat No 0.174\n", " Yes 0.169\n", "Sun No 0.186\n", " Yes 0.191\n", "Thur No 0.181\n", " Yes 0.177\n", "Name: tip_pct, dtype: float64" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips.groupby([\"day\", \"smoker\"])[\"tip_pct\"].mean().round(3)" ] }, { "cell_type": "code", "execution_count": 52, "id": "e039", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
smokerNoYes
day
Fri0.1640.169
Sat0.1740.169
Sun0.1860.191
Thur0.1810.177
\n", "
" ], "text/plain": [ "smoker No Yes\n", "day \n", "Fri 0.164 0.169\n", "Sat 0.174 0.169\n", "Sun 0.186 0.191\n", "Thur 0.181 0.177" ] }, "execution_count": 52, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips.groupby([\"day\", \"smoker\"])[\"tip_pct\"].mean().unstack().round(3)" ] }, { "cell_type": "markdown", "id": "e040", "metadata": {}, "source": [ "### Named aggregation\n", "\n", "**Prefer this form.** It names each output column, so you never end up parsing a\n", "hierarchical column index afterwards." ] }, { "cell_type": "code", "execution_count": 55, "id": "8bda22d3-9046-405a-a632-7011b5db03a8", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
total_billtiptip_pctsmokerdaytimesizebill_pp
023.574.210.179NoSatLunch45.893
117.184.030.235YesSunDinner28.590
217.403.620.208YesThurDinner28.700
319.853.750.189NoSunDinner44.963
451.7913.050.252YesThurDinner225.895
\n", "
" ], "text/plain": [ " total_bill tip tip_pct smoker day time size bill_pp\n", "0 23.57 4.21 0.179 No Sat Lunch 4 5.893\n", "1 17.18 4.03 0.235 Yes Sun Dinner 2 8.590\n", "2 17.40 3.62 0.208 Yes Thur Dinner 2 8.700\n", "3 19.85 3.75 0.189 No Sun Dinner 4 4.963\n", "4 51.79 13.05 0.252 Yes Thur Dinner 2 25.895" ] }, "execution_count": 55, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips.head()" ] }, { "cell_type": "code", "execution_count": 57, "id": "6c4b39af-1bba-434c-acc6-075e9bdfc90f", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "244" ] }, "execution_count": 57, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips[\"tip\"].size" ] }, { "cell_type": "code", "execution_count": 54, "id": "e041", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
navg_pctmax_bill
smoker
No1430.17857.18
Yes1010.17751.79
\n", "
" ], "text/plain": [ " n avg_pct max_bill\n", "smoker \n", "No 143 0.178 57.18\n", "Yes 101 0.177 51.79" ] }, "execution_count": 54, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips.groupby(\"smoker\").agg(\n", " n = (\"tip\", \"size\"),\n", " avg_pct = (\"tip_pct\", \"mean\"),\n", " max_bill = (\"total_bill\", \"max\")).round(3)" ] }, { "cell_type": "markdown", "id": "e042", "metadata": {}, "source": [ "### `agg` vs `transform`\n", "\n", "`agg` returns **one row per group**. `transform` returns **one row per original row**,\n", "broadcasting the group result back — which is exactly what you need for group-centering\n", "and within-group standardization." ] }, { "cell_type": "code", "execution_count": 63, "id": "ac7321a2-33e1-4a56-88c0-69a12c6d8428", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
total_billtiptip_pctsmokerdaytimesizebill_ppday_meanvs_day
023.574.210.179NoSatLunch45.8930.1720.750
117.184.030.235YesSunDinner28.5900.1884.687
217.403.620.208YesThurDinner28.7000.1792.853
319.853.750.189NoSunDinner44.9630.1880.087
451.7913.050.252YesThurDinner225.8950.1797.253
\n", "
" ], "text/plain": [ " total_bill tip tip_pct smoker day time size bill_pp day_mean \\\n", "0 23.57 4.21 0.179 No Sat Lunch 4 5.893 0.172 \n", "1 17.18 4.03 0.235 Yes Sun Dinner 2 8.590 0.188 \n", "2 17.40 3.62 0.208 Yes Thur Dinner 2 8.700 0.179 \n", "3 19.85 3.75 0.189 No Sun Dinner 4 4.963 0.188 \n", "4 51.79 13.05 0.252 Yes Thur Dinner 2 25.895 0.179 \n", "\n", " vs_day \n", "0 0.750 \n", "1 4.687 \n", "2 2.853 \n", "3 0.087 \n", "4 7.253 " ] }, "execution_count": 63, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips.head()" ] }, { "cell_type": "code", "execution_count": 64, "id": "e043", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
daytip_pctday_meanvs_day
0Sat0.1790.1720.750
1Sun0.2350.1884.687
2Thur0.2080.1792.853
3Sun0.1890.1880.087
4Thur0.2520.1797.253
\n", "
" ], "text/plain": [ " day tip_pct day_mean vs_day\n", "0 Sat 0.179 0.172 0.750\n", "1 Sun 0.235 0.188 4.687\n", "2 Thur 0.208 0.179 2.853\n", "3 Sun 0.189 0.188 0.087\n", "4 Thur 0.252 0.179 7.253" ] }, "execution_count": 64, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips[\"day_mean\"] = tips.groupby(\"day\")[\"tip_pct\"].transform(\"mean\")\n", "tips[\"vs_day\"] = (tips[\"tip_pct\"] - tips[\"day_mean\"])*100\n", "tips[[\"day\", \"tip_pct\", \"day_mean\", \"vs_day\"]].head()" ] }, { "cell_type": "markdown", "id": "e044", "metadata": {}, "source": [ "### Always report the group sizes\n", "\n", "A mean over 18 observations is not the same kind of number as a mean over 87. Putting\n", "`size` in the `agg` call gets you `n` for free." ] }, { "cell_type": "code", "execution_count": 65, "id": "e045", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
navg_pct
day
Fri260.166
Sat840.172
Sun750.188
Thur590.179
\n", "
" ], "text/plain": [ " n avg_pct\n", "day \n", "Fri 26 0.166\n", "Sat 84 0.172\n", "Sun 75 0.188\n", "Thur 59 0.179" ] }, "execution_count": 65, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips.groupby(\"day\").agg(n=(\"tip\", \"size\"),\n", " avg_pct=(\"tip_pct\", \"mean\")).round(4)" ] }, { "cell_type": "markdown", "id": "e046", "metadata": {}, "source": [ "---\n", "# 7. Joining and Reshaping" ] }, { "cell_type": "code", "execution_count": 68, "id": "4b518edd-16c9-4b90-ad07-9fdc3c32b6ae", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
total_billtiptip_pctsmokerdaytimesizebill_ppday_meanvs_day
023.574.210.179NoSatLunch45.8930.1720.750
117.184.030.235YesSunDinner28.5900.1884.687
217.403.620.208YesThurDinner28.7000.1792.853
319.853.750.189NoSunDinner44.9630.1880.087
451.7913.050.252YesThurDinner225.8950.1797.253
\n", "
" ], "text/plain": [ " total_bill tip tip_pct smoker day time size bill_pp day_mean \\\n", "0 23.57 4.21 0.179 No Sat Lunch 4 5.893 0.172 \n", "1 17.18 4.03 0.235 Yes Sun Dinner 2 8.590 0.188 \n", "2 17.40 3.62 0.208 Yes Thur Dinner 2 8.700 0.179 \n", "3 19.85 3.75 0.189 No Sun Dinner 4 4.963 0.188 \n", "4 51.79 13.05 0.252 Yes Thur Dinner 2 25.895 0.179 \n", "\n", " vs_day \n", "0 0.750 \n", "1 4.687 \n", "2 2.853 \n", "3 0.087 \n", "4 7.253 " ] }, "execution_count": 68, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips.head()" ] }, { "cell_type": "code", "execution_count": 71, "id": "d93df951-5c90-43ff-b31f-6432b77b407b", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "day\n", "Sat 84\n", "Sun 75\n", "Thur 59\n", "Fri 26\n", "Name: count, dtype: int64" ] }, "execution_count": 71, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips[\"day\"].value_counts()" ] }, { "cell_type": "code", "execution_count": 73, "id": "e047", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rows before: 244 after: 244\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
daytipon_shift
0Sat4.219
1Sun4.038
2Thur3.624
3Sun3.758
4Thur13.054
\n", "
" ], "text/plain": [ " day tip on_shift\n", "0 Sat 4.21 9\n", "1 Sun 4.03 8\n", "2 Thur 3.62 4\n", "3 Sun 3.75 8\n", "4 Thur 13.05 4" ] }, "execution_count": 73, "metadata": {}, "output_type": "execute_result" } ], "source": [ "staff = pd.DataFrame({\"day\": [\"Thur\", \"Fri\", \"Sat\", \"Sun\"],\n", " \"on_shift\": [4, 5, 9, 8]})\n", "\n", "joined = tips.merge(staff, on=\"day\", how=\"left\")\n", "print(\"rows before:\", len(tips), \" after:\", len(joined))\n", "joined[[\"day\", \"tip\", \"on_shift\"]].head()" ] }, { "cell_type": "code", "execution_count": 75, "id": "3043a257-5d49-4d7c-ab8d-6a08d8175685", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "on_shift\n", "9 84\n", "8 75\n", "4 59\n", "5 26\n", "Name: count, dtype: int64" ] }, "execution_count": 75, "metadata": {}, "output_type": "execute_result" } ], "source": [ "joined[\"on_shift\"].value_counts()" ] }, { "cell_type": "markdown", "id": "e048", "metadata": {}, "source": [ "| `how=` | Keys kept | R |\n", "|---|---|---|\n", "| `\"inner\"` (default) | in both tables | `inner_join` |\n", "| `\"left\"` | all of the left table | `left_join` |\n", "| `\"outer\"` | union of both | `full_join` |\n", "\n", "### The merge that eats your data\n", "\n", "**If the right table has duplicate keys, a merge silently adds rows.** Nothing warns you." ] }, { "cell_type": "code", "execution_count": 76, "id": "e049", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rows before: 244\n", "rows after : 303\n" ] } ], "source": [ "dup_lookup = pd.DataFrame({\"day\": [\"Thur\", \"Thur\", \"Fri\", \"Sat\", \"Sun\"],\n", " \"on_shift\": [4, 7, 5, 9, 8]})\n", "print(\"rows before:\", len(tips))\n", "print(\"rows after :\", len(tips.merge(dup_lookup, on=\"day\", how=\"left\"))) # grew!" ] }, { "cell_type": "code", "execution_count": 77, "id": "e050", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "MergeError -> Merge keys are not unique in right dataset; not a many-to-one merge\n" ] } ], "source": [ "# make it fail loudly instead\n", "try:\n", " tips.merge(dup_lookup, on=\"day\", how=\"left\", validate=\"m:1\")\n", "except Exception as e:\n", " print(type(e).__name__, \"->\", e)" ] }, { "cell_type": "markdown", "id": "e051", "metadata": {}, "source": [ "Check `lookup[\"key\"].is_unique` before you join, or pass `validate=\"m:1\"` and let pandas\n", "refuse. Checking `shape` before and after every merge is the cheapest insurance in\n", "pandas.\n", "\n", "### `pivot_table` and `crosstab`\n", "\n", "`pivot_table` is `groupby` + `unstack` in one call. `crosstab` is the counting special\n", "case — R's `table()`." ] }, { "cell_type": "code", "execution_count": 78, "id": "e052", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "smoker No Yes\n", "day \n", "Fri 0.164 0.169\n", "Sat 0.174 0.169\n", "Sun 0.186 0.191\n", "Thur 0.181 0.177\n", "\n", "smoker No Yes All\n", "day \n", "Fri 15 11 26\n", "Sat 45 39 84\n", "Sun 45 30 75\n", "Thur 38 21 59\n", "All 143 101 244\n" ] } ], "source": [ "print(tips.pivot_table(index=\"day\", columns=\"smoker\",\n", " values=\"tip_pct\", aggfunc=\"mean\").round(3))\n", "print()\n", "print(pd.crosstab(tips[\"day\"], tips[\"smoker\"], margins=True))" ] }, { "cell_type": "markdown", "id": "e053", "metadata": {}, "source": [ "### Method chaining\n", "\n", "Wrap the chain in parentheses and put each step on its own line — pandas' answer to the\n", "`|>` pipe." ] }, { "cell_type": "code", "execution_count": 82, "id": "e23a72e5-f059-47fe-87c6-0b3f71db3753", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
total_billtiptip_pctsmokerdaytimesizebill_ppday_meanvs_day
023.574.210.179NoSatLunch45.8930.1720.750
319.853.750.189NoSunDinner44.9630.1880.087
825.583.150.123YesSatDinner46.3950.172-4.850
935.443.680.104YesSatLunch48.8600.172-6.750
1129.266.130.210YesSunDinner47.3150.1882.187
\n", "
" ], "text/plain": [ " total_bill tip tip_pct smoker day time size bill_pp day_mean \\\n", "0 23.57 4.21 0.179 No Sat Lunch 4 5.893 0.172 \n", "3 19.85 3.75 0.189 No Sun Dinner 4 4.963 0.188 \n", "8 25.58 3.15 0.123 Yes Sat Dinner 4 6.395 0.172 \n", "9 35.44 3.68 0.104 Yes Sat Lunch 4 8.860 0.172 \n", "11 29.26 6.13 0.210 Yes Sun Dinner 4 7.315 0.188 \n", "\n", " vs_day \n", "0 0.750 \n", "3 0.087 \n", "8 -4.850 \n", "9 -6.750 \n", "11 2.187 " ] }, "execution_count": 82, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test=tips.query(\"size >= 4\")\n", "test.head()" ] }, { "cell_type": "code", "execution_count": 83, "id": "e054", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
navg
day
Sun740.189
Thur590.179
Fri240.172
Sat840.172
\n", "
" ], "text/plain": [ " n avg\n", "day \n", "Sun 74 0.189\n", "Thur 59 0.179\n", "Fri 24 0.172\n", "Sat 84 0.172" ] }, "execution_count": 83, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(tips\n", " .query(\"size >= 2\")\n", " .groupby(\"day\")\n", " .agg(n=(\"tip\", \"size\"), avg=(\"tip_pct\", \"mean\"))\n", " .sort_values(\"avg\", ascending=False)\n", " .round(3))" ] }, { "cell_type": "markdown", "id": "e055", "metadata": {}, "source": [ "---\n", "# Wrapping Up\n", "\n", "## pandas and dplyr side by side\n", "\n", "| dplyr / tidyr | pandas |\n", "|---|---|\n", "| `filter(df, x > 1)` | `df[df[\"x\"] > 1]` |\n", "| `select(df, a, b)` | `df[[\"a\", \"b\"]]` |\n", "| `mutate(df, z = x + y)` | `df.assign(z=df[\"x\"] + df[\"y\"])` |\n", "| `arrange(df, desc(x))` | `df.sort_values(\"x\", ascending=False)` |\n", "| `group_by(g) %>% summarize(m = mean(x))` | `df.groupby(\"g\")[\"x\"].mean()` |\n", "| `left_join(a, b, by = \"k\")` | `pd.merge(a, b, on=\"k\", how=\"left\")` |\n", "| `distinct(df)` | `df.drop_duplicates()` |\n", "| `count(df, g)` | `df[\"g\"].value_counts()` |\n", "| `x %in% y` | `s.isin(y)` |\n", "\n", "## The Five Silent Failures\n", "\n", "1. **Alignment.** Operations match on *labels*, not position. Two Series with different\n", " indexes give you `NaN`, not an error.\n", "2. **Chained assignment.** `df[mask][\"col\"] = x` may do nothing. Write one `.loc` call.\n", "3. **`and` / `or`.** Use `&`, `|`, `~`, and parenthesize each condition.\n", "4. **Silent `NaN` from `map`.** A key missing from the dictionary becomes missing data\n", " with no warning. Count afterwards.\n", "5. **Merges that grow.** Duplicate keys multiply rows silently. Check `shape` before and\n", " after, every time." ] }, { "cell_type": "markdown", "id": "e056", "metadata": {}, "source": [ "---\n", "# Your Turn" ] }, { "cell_type": "markdown", "id": "e057", "metadata": {}, "source": [ "**Exercise 1.** Select the checks where `size` is at least 4 **and** `time` is `\"Dinner\"`.\n", "How many are there, and what is their mean `tip_pct`?" ] }, { "cell_type": "code", "execution_count": 90, "id": "e058", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
navg
time
Dinner340.187
Lunch160.159
\n", "
" ], "text/plain": [ " n avg\n", "time \n", "Dinner 34 0.187\n", "Lunch 16 0.159" ] }, "execution_count": 90, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(tips\n", " .query(\"size>=4\")\n", " .groupby(\"time\")\n", " .agg(n=(\"tip\", \"size\"), avg=(\"tip_pct\", \"mean\")))\n", "# your code here" ] }, { "cell_type": "markdown", "id": "e059", "metadata": {}, "source": [ "**Exercise 2.** Add a column `bill_pp = total_bill / size`. Then report, for each `day`,\n", "the number of checks and the mean `bill_pp` — in a single `groupby().agg()` call with\n", "named output columns." ] }, { "cell_type": "code", "execution_count": 92, "id": "65fa14c8-5238-4289-ad7b-870bb28aaf6d", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
total_billtiptip_pctsmokerdaytimesizebill_ppday_meanvs_day
023.574.210.179NoSatLunch45.8930.1720.750
117.184.030.235YesSunDinner28.5900.1884.687
217.403.620.208YesThurDinner28.7000.1792.853
319.853.750.189NoSunDinner44.9630.1880.087
451.7913.050.252YesThurDinner225.8950.1797.253
\n", "
" ], "text/plain": [ " total_bill tip tip_pct smoker day time size bill_pp day_mean \\\n", "0 23.57 4.21 0.179 No Sat Lunch 4 5.893 0.172 \n", "1 17.18 4.03 0.235 Yes Sun Dinner 2 8.590 0.188 \n", "2 17.40 3.62 0.208 Yes Thur Dinner 2 8.700 0.179 \n", "3 19.85 3.75 0.189 No Sun Dinner 4 4.963 0.188 \n", "4 51.79 13.05 0.252 Yes Thur Dinner 2 25.895 0.179 \n", "\n", " vs_day \n", "0 0.750 \n", "1 4.687 \n", "2 2.853 \n", "3 0.087 \n", "4 7.253 " ] }, "execution_count": 92, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips.head()" ] }, { "cell_type": "code", "execution_count": 100, "id": "e060", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
navg_pp
day
Fri269.772
Sat849.400
Sun7510.221
Thur5910.392
\n", "
" ], "text/plain": [ " n avg_pp\n", "day \n", "Fri 26 9.772\n", "Sat 84 9.400\n", "Sun 75 10.221\n", "Thur 59 10.392" ] }, "execution_count": 100, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(tips\n", " .assign(bill_pp=lambda d: d[\"total_bill\"]/d[\"size\"])\n", " .groupby(\"day\").agg(n=(\"tip\", \"size\"), avg_pp=(\"bill_pp\", \"mean\")))" ] }, { "cell_type": "markdown", "id": "e061", "metadata": {}, "source": [ "**Exercise 3.** Build a lookup table mapping each `day` to `\"Weekend\"` or `\"Weekday\"`.\n", "Attach it to `tips` **twice** — once with `map`, once with `merge` — and confirm that the\n", "row count did not change." ] }, { "cell_type": "code", "execution_count": 104, "id": "e58bd10e-2a01-4e59-b8b3-64e2fd47f7c4", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
dayisWeekend
0ThurFalse
1FriFalse
2SatTrue
3SunTrue
\n", "
" ], "text/plain": [ " day isWeekend\n", "0 Thur False\n", "1 Fri False\n", "2 Sat True\n", "3 Sun True" ] }, "execution_count": 104, "metadata": {}, "output_type": "execute_result" } ], "source": [ "look_up=pd.DataFrame({\"day\": [\"Thur\", \"Fri\", \"Sat\", \"Sun\"],\n", " \"isWeekend\": [False, False, True, True]})\n", "look_up.head()" ] }, { "cell_type": "code", "execution_count": 109, "id": "f33dc85f-4597-4d58-ba73-f90189a57f45", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
isWeekend
day
ThurFalse
FriFalse
SatTrue
SunTrue
\n", "
" ], "text/plain": [ " isWeekend\n", "day \n", "Thur False\n", "Fri False\n", "Sat True\n", "Sun True" ] }, "execution_count": 109, "metadata": {}, "output_type": "execute_result" } ], "source": [ "look_up.set_index(\"day\")" ] }, { "cell_type": "code", "execution_count": 120, "id": "e062", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "row before map 244\n", "row after map 244\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
total_billtiptip_pctsmokerdaytimesizebill_ppday_meanvs_dayisWeekend
023.574.210.179NoSatLunch45.8930.1720.750True
117.184.030.235YesSunDinner28.5900.1884.687True
217.403.620.208YesThurDinner28.7000.1792.853False
319.853.750.189NoSunDinner44.9630.1880.087True
451.7913.050.252YesThurDinner225.8950.1797.253False
\n", "
" ], "text/plain": [ " total_bill tip tip_pct smoker day time size bill_pp day_mean \\\n", "0 23.57 4.21 0.179 No Sat Lunch 4 5.893 0.172 \n", "1 17.18 4.03 0.235 Yes Sun Dinner 2 8.590 0.188 \n", "2 17.40 3.62 0.208 Yes Thur Dinner 2 8.700 0.179 \n", "3 19.85 3.75 0.189 No Sun Dinner 4 4.963 0.188 \n", "4 51.79 13.05 0.252 Yes Thur Dinner 2 25.895 0.179 \n", "\n", " vs_day isWeekend \n", "0 0.750 True \n", "1 4.687 True \n", "2 2.853 False \n", "3 0.087 True \n", "4 7.253 False " ] }, "execution_count": 120, "metadata": {}, "output_type": "execute_result" } ], "source": [ "look_up=pd.DataFrame({\"day\": [\"Thur\", \"Fri\", \"Sat\", \"Sun\"],\n", " \"isWeekend\": [False, False, True, True]})\n", "key=look_up.set_index(\"day\")[\"isWeekend\"]\n", "print(\"row before map\", len(tips))\n", "tips[\"isWeekend\"]=tips[\"day\"].map(key)\n", "print(\"row after map\", len(tips))\n", "tips.head()" ] }, { "cell_type": "code", "execution_count": 124, "id": "88f6d571-c591-4110-9305-eab6de20f3b9", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
total_billtiptip_pctsmokerdaytimesizebill_ppday_meanvs_dayisWeekend_xisWeekend_y
023.574.210.179NoSatLunch45.8930.1720.750TrueTrue
117.184.030.235YesSunDinner28.5900.1884.687TrueTrue
217.403.620.208YesThurDinner28.7000.1792.853FalseFalse
319.853.750.189NoSunDinner44.9630.1880.087TrueTrue
451.7913.050.252YesThurDinner225.8950.1797.253FalseFalse
\n", "
" ], "text/plain": [ " total_bill tip tip_pct smoker day time size bill_pp day_mean \\\n", "0 23.57 4.21 0.179 No Sat Lunch 4 5.893 0.172 \n", "1 17.18 4.03 0.235 Yes Sun Dinner 2 8.590 0.188 \n", "2 17.40 3.62 0.208 Yes Thur Dinner 2 8.700 0.179 \n", "3 19.85 3.75 0.189 No Sun Dinner 4 4.963 0.188 \n", "4 51.79 13.05 0.252 Yes Thur Dinner 2 25.895 0.179 \n", "\n", " vs_day isWeekend_x isWeekend_y \n", "0 0.750 True True \n", "1 4.687 True True \n", "2 2.853 False False \n", "3 0.087 True True \n", "4 7.253 False False " ] }, "execution_count": 124, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tips_merge=tips.merge(look_up, on=\"day\", how=\"left\")\n", "tips_merge.head()" ] }, { "cell_type": "markdown", "id": "e063", "metadata": {}, "source": [ "---\n", "## Where to go next\n", "\n", "- **McKinney, W.** *Python for Data Analysis*, 3rd ed. — \n", " — Ch. 5 getting started · Ch. 7 cleaning · Ch. 8 join/reshape · Ch. 10 group operations\n", "- **VanderPlas, J.** *Python Data Science Handbook*, Ch. 3 —\n", " \n", "- [pandas user guide](https://pandas.pydata.org/docs/user_guide/)\n", "- [Comparison with R](https://pandas.pydata.org/docs/getting_started/comparison/comparison_with_r.html)\n", "\n", "The **long version** of this lecture covers each of these ideas in depth, plus\n", "hierarchical indexing, `stack`/`unstack`, string methods, categoricals, and the full\n", "R-to-pandas translation tables." ] }, { "cell_type": "code", "execution_count": null, "id": "3381cdee-3b52-4684-bd06-39fd87c21d04", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "c43ff07d-b3f3-4b5e-a3ec-ff4ba3d7ce08", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "authors": [ { "name": "Yen-Yi Ho" } ], "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.3" }, "title": "pandas Essentials" }, "nbformat": 4, "nbformat_minor": 5 }