R to pandas¶

A Translation Reference¶

STAT 540 — Computing in Statistics Yen-Yi Ho · Department of Statistics, University of South Carolina


How to use this¶

Every table maps an R idiom to the pandas way of saying it. The right-hand column is what you type; the notes under each table flag the places where the translation is not one-to-one --- which is where the bugs live.

Three conventions throughout:

  • df is a DataFrame, s is a Series, and x/y are vectors or columns.
  • Where dplyr and base R differ, both are shown. dplyr is usually the closer analogy.
  • A Mind this note means the two languages genuinely disagree. Read those twice.

The companion notebook runs every pandas expression on a small worked example, so you can see the output rather than trust the table.

The worked example¶

Everything below runs against this five-row frame.

In [ ]:
import numpy as np
import pandas as pd

df = pd.DataFrame({
    "id":    ["a", "b", "c", "d", "e"],
    "grp":   ["x", "y", "x", "y", "x"],
    "score": [88.0, 92.5, np.nan, 71.0, 95.0],
    "n":     [3, 1, 4, 1, 5],
})
df

In R this would be

df <- data.frame(
  id    = c("a", "b", "c", "d", "e"),
  grp   = c("x", "y", "x", "y", "x"),
  score = c(88, 92.5, NA, 71, 95),
  n     = c(3, 1, 4, 1, 5),
  stringsAsFactors = FALSE
)

1. The mental map¶

Concept R pandas
Homogeneous array matrix, array numpy.ndarray
Labeled 1-D vector named vector pandas.Series
Table of columns data.frame, tibble pandas.DataFrame
Row labels rownames(df) df.index
Column labels names(df), colnames(df) df.columns
Missing value NA, NaN, NULL NaN, None, pd.NA
Factor factor category dtype
List list dict or list
NULL NULL None
Pipe %>%, \|> method chaining in ( ... )

Mind this. R's three missing-ish values collapse to roughly one in pandas: a float column stores None as NaN, so None and np.nan behave identically there. pd.NA is the newer sentinel used by the nullable dtypes (Int64, boolean, string).

2. Inspecting¶

Task R pandas
Dimensions dim(df) df.shape
Rows / columns nrow(df) / ncol(df) len(df) / df.shape[1]
Column names names(df) df.columns
Structure str(df) df.info()
First / last rows head(df) / tail(df) df.head() / df.tail()
Summary summary(df) df.describe()
Column types sapply(df, class) df.dtypes
Frequency table table(x) s.value_counts()
Two-way table table(x, y) pd.crosstab(x, y)
Distinct values unique(x) s.unique()
Count distinct length(unique(x)) s.nunique()
Any missing? any(is.na(df)) df.isna().any().any()
Missing per column colSums(is.na(df)) df.isna().sum()
Complete rows complete.cases(df) df.notna().all(axis="columns")

Mind this. dim, names and shape are attributes in pandas --- no parentheses. info() and describe() are methods --- parentheses required. Forgetting them on a method prints <bound method ...> with no error at all.

In [ ]:
print(df.shape)                       # dim(df)
print(df.dtypes.to_dict())            # sapply(df, class)
print(df["grp"].value_counts().to_dict())   # table(grp)
print(df.isna().sum().to_dict())      # colSums(is.na(df))
print(df.notna().all(axis="columns").tolist())   # complete.cases(df)

3. Selecting columns¶

Task R (base) R (dplyr) pandas
One column, as a vector df$score pull(df, score) df["score"]
One column, as a table df["score"] select(df, score) df[["score"]]
Several columns df[c("id", "n")] select(df, id, n) df[["id", "n"]]
Drop a column df$score <- NULL select(df, -score) df.drop(columns="score")
By position df[, 2] df.iloc[:, 1]
Rename names(df)[1] <- "ID" rename(df, ID = id) df.rename(columns={"id": "ID"})
Reorder df[c("n", "id")] relocate(df, n) df[["n", "id"]]

Mind this. df["score"] returns a Series; df[["score"]] returns a one-column DataFrame. Same distinction as pull() vs select().

Avoid attribute access (df.score). It fails on names with spaces, cannot create a new column, and silently loses to built-ins --- df.size is the element count, not a column called size.

4. Selecting rows and filtering¶

Task R (base) R (dplyr) pandas
By position df[1:3, ] slice(df, 1:3) df.iloc[0:3]
By row label df["r1", ] df.loc["r1"]
Filter on a condition df[df$score > 85, ] filter(df, score > 85) df[df["score"] > 85]
Two conditions df[df$n > 1 & df$grp == "x", ] filter(df, n > 1, grp == "x") df[(df["n"] > 1) & (df["grp"] == "x")]
Membership df[df$grp %in% c("x"), ] filter(df, grp %in% c("x")) df[df["grp"].isin(["x"])]
Negate df[!cond, ] filter(df, !cond) df[~cond]
Drop missing na.omit(df) drop_na(df) df.dropna()
Top n head(df[order(-df$n), ], 3) slice_max(df, n, n = 3) df.nlargest(3, "n")
Random rows df[sample(nrow(df), 3), ] slice_sample(df, n = 3) df.sample(3)
Query string df.query("n > 1 and grp == 'x'")

Mind this --- three separate traps.

  1. Use &, |, ~ --- never Python's and / or / not, which cannot work element-wise.
  2. Parenthesize every condition. & binds more tightly than >, so df["n"] > 1 & df["grp"] == "x" evaluates in the wrong order and raises.
  3. df.loc[0:2] includes the endpoint (3 rows); df.iloc[0:2] excludes it (2 rows). Label slices are the one place Python's half-open rule does not hold.
In [ ]:
print(df[df["score"] > 85]["id"].tolist())                       # filter
print(df[(df["n"] > 1) & (df["grp"] == "x")]["id"].tolist())     # two conditions
print(df.query("n > 1 and grp == 'x'")["id"].tolist())           # same, as a string
print(len(df.loc[0:2]), len(df.iloc[0:2]))   # 3 and 2 -- endpoint rule differs

5. which(), match() and set operations¶

R pandas Returns
which(x > 4) np.flatnonzero(s > 4) integer positions (0-based)
s.index[s > 4] index labels
x[which(x > 4)] s[s > 4] the values --- preferred
length(which(cond)) cond.sum() a count
which.max(x) s.idxmax() index label
s.argmax() integer position
which(x == max(x)) s.index[s == s.max()] all ties
which(is.na(x)) s.index[s.isna()] labels of the missing
which(duplicated(x)) np.flatnonzero(s.duplicated()) positions
which(m > 5, arr.ind = TRUE) np.argwhere((m > 5).to_numpy()) row/col positions
m.stack()[(m > 5).stack()] labels and values
match(x, table) pd.Index(table).get_indexer(x) positions, -1 if absent
(table has duplicates) pd.Index(table).get_indexer_for(x) every match
x %in% table s.isin(table) Boolean mask
intersect(x, y) pd.Index(x).intersection(y) in both
setdiff(x, y) pd.Index(x).difference(y) in x, not y
union(x, y) pd.Index(x).union(y) either

Mind this. Two differences bite everyone:

  • Positions are 0-based, so which() minus one.
  • "Not found" is -1, not NA --- and -1 is a legal Python position meaning "last element". Never feed get_indexer output to iloc without filtering on >= 0.

get_indexer also requires the lookup table to be unique, and the Index set operations sort and deduplicate their result.

In [ ]:
s = df["score"]
print(np.flatnonzero(s > 85))        # which(score > 85)
print(s.index[s > 85].tolist())      # the labels
print(s.idxmax(), s.argmax())        # which.max: label vs position

ids = pd.Index(df["id"])
print(ids.get_indexer(["c", "z"]))   # match(c("c","z"), id) -> 2, -1  (R: 3, NA)
print(ids.difference(["a", "b"]).tolist())   # setdiff

6. Sorting and ranking¶

Task R (base) R (dplyr) pandas
Sort a vector sort(x) s.sort_values()
Sort rows df[order(df$n), ] arrange(df, n) df.sort_values("n")
Descending df[order(-df$n), ] arrange(df, desc(n)) df.sort_values("n", ascending=False)
Several keys df[order(df$grp, -df$n), ] arrange(df, grp, desc(n)) df.sort_values(["grp", "n"], ascending=[True, False])
Sort by row label df.sort_index()
Ordering vector order(x) s.argsort()
Rank rank(x) min_rank(x) s.rank()
Reverse rev(x) s[::-1]

Tie-breaking in rank(method=): "average" (default), "min", "max", "first", "dense".

Mind this. Missing values sort last by default in both languages, but pandas lets you say na_position="first". And sort_values returns a new frame --- it never sorts in place unless you pass inplace=True, which is discouraged.

7. Creating and modifying columns¶

Task R (base) R (dplyr) pandas
New column df$z <- df$n * 2 mutate(df, z = n * 2) df["z"] = df["n"] * 2
Several at once mutate(df, a = ..., b = ...) df.assign(a=lambda d: ..., b=lambda d: ...)
Conditional ifelse(x > 2, "big", "small") if_else(...) np.where(x > 2, "big", "small")
Multi-way cut(x, breaks) case_when(...) np.select([c1, c2], [v1, v2], default=...)
Recode from a table lookup$v[match(df$k, lookup$k)] left_join(df, lookup) df["k"].map(lookup.set_index("k")["v"])
Replace values x[x == -99] <- NA na_if(x, -99) s.replace(-99, np.nan)
Bin, equal width cut(x, 4) pd.cut(x, 4)
Bin, equal count cut(x, quantile(x)) ntile(x, 4) pd.qcut(x, 4)
Cumulative cumsum(x) s.cumsum()
Lag / lead dplyr::lag(x) lag(x) / lead(x) s.shift(1) / s.shift(-1)
Row-wise difference diff(x) s.diff()

Mind this. Inside assign, pandas has no data masking --- you cannot write bare column names the way mutate lets you. Use lambda d: d["col"], which also keeps the expression correct when an earlier step in the chain changed the rows.

df.assign(z = lambda d: d["n"] * 2)     # right
df.assign(z = n * 2)                    # NameError: name 'n' is not defined

map returns NaN for any key missing from the lookup --- silently. Count the NaNs afterwards, every time.

In [ ]:
out = df.assign(
    z     = lambda d: d["n"] * 2,                        # mutate(z = n * 2)
    sizec = lambda d: np.where(d["n"] > 2, "big", "small"),   # ifelse
    lagn  = lambda d: d["n"].shift(1),                   # lag(n)
)
out[["id", "n", "z", "sizec", "lagn"]]
In [ ]:
# recode from a lookup table: lookup$v[match(df$k, lookup$k)]
lookup = pd.DataFrame({"grp": ["x", "y"], "label": ["Control", "Treated"]})
recoded = df["grp"].map(lookup.set_index("grp")["label"])
print(recoded.tolist())
print("NaNs introduced:", recoded.isna().sum())   # ALWAYS check

8. Missing data¶

Task R pandas
Test for missing is.na(x) s.isna()
Test for present !is.na(x) s.notna()
Count missing sum(is.na(x)) s.isna().sum()
Drop missing rows na.omit(df) df.dropna()
Drop if all missing df.dropna(how="all")
Keep rows with >= k present df.dropna(thresh=k)
Fill with a constant x[is.na(x)] <- 0 s.fillna(0)
Fill per column df.fillna({"a": 0, "b": -1})
Carry forward zoo::na.locf(x) s.ffill()
Carry backward s.bfill()
Mean imputation x[is.na(x)] <- mean(x, na.rm = TRUE) s.fillna(s.mean())
Ignore NA in a summary mean(x, na.rm = TRUE) s.mean() (default)
Include NA in a summary mean(x) s.mean(skipna=False)

Mind this --- the defaults are opposite. R's mean(x) returns NA if anything is missing and you must ask for na.rm = TRUE. pandas skips missing by default; you must ask for skipna=False to propagate. That one reversal silently changes results when translating code in either direction.

thresh=k counts the values that are present, not the ones that are missing.

In [ ]:
print("count missing :", df["score"].isna().sum())
print("pandas mean   :", round(df["score"].mean(), 2), "  (R: mean(score, na.rm=TRUE))")
print("skipna=False  :", df["score"].mean(skipna=False), "  (R: mean(score))")
print("ffill         :", df["score"].ffill().tolist())
print("dropna rows   :", len(df.dropna()), "of", len(df))

9. Summary statistics¶

Task R pandas
Mean / median mean(x) / median(x) s.mean() / s.median()
SD / variance sd(x) / var(x) s.std() / s.var()
Min / max min(x) / max(x) s.min() / s.max()
Range range(x) (s.min(), s.max())
Sum / product sum(x) / prod(x) s.sum() / s.prod()
Quantiles quantile(x, .25) s.quantile(.25)
Correlation cor(x, y) s.corr(t)
Correlation matrix cor(df) df.corr(numeric_only=True)
Covariance cov(x, y) s.cov(t)
Column means colMeans(df) df.mean(numeric_only=True)
Row means rowMeans(df) df.mean(axis="columns")
Count non-missing sum(!is.na(x)) s.count()
Length length(x) len(s) or s.size
All / any all(cond) / any(cond) cond.all() / cond.any()

Mind this. s.std() uses the sample standard deviation (ddof=1), matching R's sd(). NumPy's np.std() uses ddof=0 by default and will not match --- a classic source of "why are my numbers slightly off?"

Also: s.count() is the number of non-missing values (R's sum(!is.na(x))), while len(s) and s.size count every row.

In [ ]:
x = df["score"]
print("pandas  s.std():", round(x.std(), 4), "   <- ddof=1, matches R's sd()")
print("numpy np.std() :", round(np.std(x.dropna()), 4), "  <- ddof=0, does NOT match")
print("count vs len   :", x.count(), len(x))

10. Split--apply--combine¶

Task R (base) R (dplyr) pandas
Group means tapply(x, g, mean) group_by(g) %>% summarize(m = mean(x)) df.groupby("g")["x"].mean()
Group sizes table(g) count(df, g) df.groupby("g").size()
Several statistics aggregate(...) summarize(m = mean(x), s = sd(x)) df.groupby("g")["x"].agg(["mean", "std"])
Named outputs summarize(avg = mean(x)) df.groupby("g").agg(avg=("x", "mean"))
Two keys group_by(g1, g2) df.groupby(["g1", "g2"])
Keys as columns (default) df.groupby("g", as_index=False)
Broadcast back ave(x, g) group_by(g) %>% mutate(m = mean(x)) df.groupby("g")["x"].transform("mean")
Filter groups group_by(g) %>% filter(n() > 2) df.groupby("g").filter(lambda d: len(d) > 2)
Arbitrary function by(df, g, f) group_modify(f) df.groupby("g").apply(f)
Split into pieces split(df, g) group_split(df, g) dict(list(df.groupby("g")))

agg vs transform is the distinction to internalize:

  • agg returns one row per group --- it is summarize().
  • transform returns one row per original row --- it is mutate() after a group_by(), and it is what you need for group-centering.

Mind this. Always report the group size beside a group statistic. Put n=("col", "size") first in every agg call: a mean over 3 observations is not the same kind of number as a mean over 300.

In [ ]:
# summarize: one row per group
print(df.groupby("grp").agg(n=("id", "size"),
                            avg=("score", "mean")).round(2))
print()
# mutate-after-group_by: one row per original row
print(df.assign(grp_mean=lambda d: d.groupby("grp")["score"].transform("mean"))
        [["id", "grp", "score", "grp_mean"]].round(2))

11. Joining tables¶

Join R (base) R (dplyr) pandas
Inner merge(a, b) inner_join(a, b, by = "k") a.merge(b, on="k")
Left merge(a, b, all.x = TRUE) left_join(a, b, by = "k") a.merge(b, on="k", how="left")
Right merge(a, b, all.y = TRUE) right_join(a, b, by = "k") a.merge(b, on="k", how="right")
Full merge(a, b, all = TRUE) full_join(a, b, by = "k") a.merge(b, on="k", how="outer")
Different key names merge(a, b, by.x=, by.y=) join(a, b, by = c("k1" = "k2")) a.merge(b, left_on="k1", right_on="k2")
Several keys merge(a, b, by = c("k1","k2")) by = c("k1","k2") on=["k1", "k2"]
On the row index a.join(b)
Keep matches only semi_join(a, b) a[a["k"].isin(b["k"])]
Keep non-matches anti_join(a, b) a[~a["k"].isin(b["k"])]
Stack rows rbind(a, b) bind_rows(a, b) pd.concat([a, b])
Stack columns cbind(a, b) bind_cols(a, b) pd.concat([a, b], axis="columns")

Mind this --- the most expensive mistake in this document. If the right table has duplicate keys, a merge silently multiplies rows, and how="left" does not protect you: how= controls which keys survive, not the row count.

before = len(a)
out = a.merge(b, on="k", how="left")
assert len(out) == before                      # cheapest insurance in pandas
b["k"].is_unique                               # or check first
a.merge(b, on="k", how="left", validate="m:1") # or let pandas refuse

Overlapping non-key column names get _x / _y suffixes rather than an error. Control them with suffixes=("_a", "_b").

In [ ]:
b = pd.DataFrame({"grp": ["x", "y"], "boss": ["Ana", "Ben"]})
print("rows:", len(df), "->", len(df.merge(b, on="grp", how="left")))

dup = pd.DataFrame({"grp": ["x", "x", "y"], "boss": ["Ana", "Alt", "Ben"]})
print("with a duplicated key:", len(df), "->", len(df.merge(dup, on="grp", how="left")))
try:
    df.merge(dup, on="grp", how="left", validate="m:1")
except Exception as e:
    print(type(e).__name__, "->", e)

12. Reshaping¶

Task R (tidyr) R (base / reshape2) pandas
Wide to long pivot_longer(df, cols) melt(df, id.vars) pd.melt(df, id_vars=...)
Long to wide pivot_wider(df, names_from, values_from) dcast(df, ...) df.pivot(index=, columns=, values=)
Long to wide, aggregating dcast(df, ..., fun.aggregate = mean) df.pivot_table(index=, columns=, aggfunc=)
Columns into rows df.stack()
Index level into columns df.unstack()
Transpose t(df) df.T
Column to row index rownames(df) <- df$id df.set_index("id")
Row index to column df.reset_index()
Cross-tabulation table(a, b) xtabs(~ a + b) pd.crosstab(a, b)

Mind this. pivot fails if the index/column pair is not unique --- that is a feature, since it means you have duplicates you did not know about. pivot_table aggregates them instead (mean by default), which is convenient but can hide the same problem.

In [ ]:
long = pd.melt(df, id_vars=["id", "grp"], value_vars=["score", "n"])
print(long.head(4))
print()
print(long.pivot(index="id", columns="variable", values="value"))

13. Strings¶

pandas puts vectorized string methods behind the .str accessor. They skip missing values instead of raising.

Task R (base) R (stringr) pandas
Length nchar(x) str_length(x) s.str.len()
Upper / lower toupper(x) str_to_upper(x) s.str.upper()
Title case str_to_title(x) s.str.title()
Trim whitespace trimws(x) str_trim(x) s.str.strip()
Concatenate paste0(x, y) str_c(x, y) x + y
Collapse to one string paste(x, collapse=",") str_c(x, collapse=",") ",".join(s)
Substring substr(x, 1, 3) str_sub(x, 1, 3) s.str[:3]
Contains grepl("a", x) str_detect(x, "a") s.str.contains("a")
Replace first sub("a", "b", x) str_replace(x, "a", "b") s.str.replace("a", "b", n=1)
Replace all gsub("a", "b", x) str_replace_all(...) s.str.replace("a", "b")
Split strsplit(x, ",") str_split(x, ",") s.str.split(",")
Which match grep("a", x) str_which(x, "a") np.flatnonzero(s.str.contains("a"))
Extract groups regmatches(...) str_match(x, re) s.str.extract(r"(\w+)@(\w+)")
Pad formatC(x, width=5) str_pad(x, 5) s.str.pad(5)
Starts / ends with startsWith(x, "a") str_starts(x, "a") s.str.startswith("a")

Mind this. str.replace replaces every occurrence by default --- it is gsub, not sub. Pass n=1 for sub behaviour. And since pandas 2.0 it treats the pattern as a literal unless you pass regex=True.

14. Factors and categoricals¶

Task R pandas
Create factor(x) s.astype("category")
With an order factor(x, levels=, ordered=TRUE) pd.Categorical(s, categories=, ordered=True)
Levels levels(f) s.cat.categories
Integer codes as.integer(f) s.cat.codes
Rename levels levels(f) <- c(...) s.cat.rename_categories([...])
Add a level s.cat.add_categories([...])
Drop unused droplevels(f) s.cat.remove_unused_categories()
Back to text as.character(f) s.astype(str)
Counts incl. empty levels table(f) s.value_counts() (keeps empty categories)
Dummy variables model.matrix(~ f) pd.get_dummies(s, drop_first=True)

Mind this. R's as.integer(factor) is 1-based; pandas cat.codes is 0-based and uses -1 for missing. And groupby on a categorical keeps unused categories by default --- pass observed=True to drop them, or you get rows of NaN for groups that do not exist.

15. The apply family¶

R pandas / NumPy Note
sapply(x, f) / vapply s.map(f) element-wise over a Series
lapply(x, f) [f(i) for i in x] returns a list
apply(m, 2, f) df.apply(f) once per column (the default)
apply(m, 1, f) df.apply(f, axis="columns") once per row
mapply(f, x, y) np.vectorize(f)(x, y) or just write it vectorized
tapply(x, g, f) df.groupby(g)["x"].agg(f) split--apply--combine
Map(f, x, y) zip(x, y) in a comprehension
Reduce(f, x) functools.reduce(f, x)
do.call(rbind, lst) pd.concat(lst)
element-wise over a frame df.map(f) every cell

Mind this --- the axis numbering is the mirror image of R's.

What you want R pandas
one value per column apply(m, 2, f) df.apply(f) or axis=0
one value per row apply(m, 1, f) df.apply(f, axis=1)

Read axis= as "the axis that disappears." axis=0 collapses the rows, so you get one number per column.

Prefer vectorized operations over apply whenever one exists: df["x"] * 2 is orders of magnitude faster than df["x"].apply(lambda v: v * 2), and shorter.

In [ ]:
num = df[["score", "n"]]
print(num.apply(lambda c: c.max() - c.min()))            # apply(m, 2, f)
print()
print(num.apply(lambda r: r["n"] * 2, axis="columns").tolist())   # apply(m, 1, f)

16. Dates¶

Task R (base) R (lubridate) pandas
Parse as.Date(x) ymd(x) pd.to_datetime(x)
Today Sys.Date() today() pd.Timestamp.today()
Year / month / day format(d, "%Y") year(d) s.dt.year
Day of week weekdays(d) wday(d, label=TRUE) s.dt.day_name()
Format out format(d, "%Y-%m") s.dt.strftime("%Y-%m")
Difference d2 - d1 s2 - s1 (a Timedelta)
Add days d + 7 d + days(7) s + pd.Timedelta(days=7)
Sequence seq(d1, d2, "day") pd.date_range(d1, d2, freq="D")
Round down floor_date(d, "month") s.dt.to_period("M")

Like .str, the .dt accessor is where all the date methods live --- and it only works once the column is actually a datetime dtype, which pd.to_datetime gives you.

17. Reading and writing¶

Format R pandas
CSV read.csv(f) / readr::read_csv(f) pd.read_csv(f)
Tab-separated read.delim(f) pd.read_csv(f, sep="\t")
Excel readxl::read_excel(f) pd.read_excel(f, sheet_name=0)
SAS haven::read_sas(f) pd.read_sas(f)
Stata haven::read_dta(f) pd.read_stata(f)
Fixed width read.fwf(f) pd.read_fwf(f)
JSON jsonlite::fromJSON(f) pd.read_json(f)
From a database DBI::dbGetQuery(con, q) pd.read_sql(q, con)
Write CSV write.csv(df, f, row.names=FALSE) df.to_csv(f, index=False)
Save native saveRDS(df, f) df.to_parquet(f) or to_pickle(f)

Useful read_csv arguments: sep, header, names, index_col, na_values, skiprows, nrows, dtype, parse_dates.

Mind this. df.to_csv(f) writes the row index as an unnamed first column by default. Pass index=False --- otherwise every round trip through CSV grows a junk Unnamed: 0 column.

18. False friends¶

Same name, different behaviour. These are the ones that cost hours.

Looks like In R In pandas
df[1:3] rows 1 to 3 columns sliced, or rows for a slice --- ambiguous; use loc/iloc
x[0] error (1-based) the first element (0-based)
a:b slice includes b excludes b --- except in .loc, which includes it
mean(x) NA if anything is missing skips missing by default
sd(x) ddof = 1 s.std() is ddof=1, but np.std() is ddof=0
sub vs gsub distinct functions str.replace is gsub; pass n=1 for sub
&& / || scalar and/or no element-wise meaning --- use & / |
match() not found NA -1, which is a valid position
which.max ties first only idxmax first only --- same, but returns a label
merge(all.x=TRUE) keeps left keys how="left" keeps left keys but can add rows
df$col <- v always works df["col"] = v works; df.col = v silently makes an attribute
apply(m, 1, f) per row axis=1 per row --- the numbering is reversed
factor codes 1-based cat.codes 0-based, -1 for missing
rev(x) reverses s[::-1] reverses values and labels together

19. The five silent failures¶

None of these raises an error. Your code runs; your numbers are wrong.

  1. Alignment. Operations match on labels, not position. Two Series with different indexes give you NaN, not an error. Guard: check .isna().sum() after arithmetic between two Series.

  2. Chained assignment. df[mask]["col"] = x modifies a temporary copy and may do nothing at all. Guard: write one .loc[rows, cols] = x call, always.

  3. and / or. Python's keywords cannot work element-wise. Guard: &, |, ~, and parenthesize every condition.

  4. Silent NaN from map. A key missing from the dictionary becomes missing data with no warning, and value_counts() then hides it. Guard: recoded.isna().sum() after every map.

  5. Merges that grow. Duplicate keys multiply rows, and how="left" does not prevent it. Guard: assert len(out) == len(left), or pass validate="m:1".

Further reading¶

  • pandas, Comparison with R / R libraries --- https://pandas.pydata.org/docs/getting_started/comparison/comparison_with_r.html
  • McKinney, W. Python for Data Analysis, 3rd ed. --- https://wesmckinney.com/book/
  • VanderPlas, J. Python Data Science Handbook, Ch. 3 --- https://jakevdp.github.io/PythonDataScienceHandbook/
  • pandas user guide --- https://pandas.pydata.org/docs/user_guide/
  • Course lectures: PandasEssentials, TipsAnalysis, PandasDataManipulation