How to use this¶
Every table maps an R idiom to the pandas way of saying it. The right-hand column is what you type; the notes under each table flag the places where the translation is not one-to-one --- which is where the bugs live.
Three conventions throughout:
dfis a DataFrame,sis a Series, andx/yare vectors or columns.- Where dplyr and base R differ, both are shown. dplyr is usually the closer analogy.
- A Mind this note means the two languages genuinely disagree. Read those twice.
The companion notebook runs every pandas expression on a small worked example, so you can see the output rather than trust the table.
The worked example¶
Everything below runs against this five-row frame.
import numpy as np
import pandas as pd
df = pd.DataFrame({
"id": ["a", "b", "c", "d", "e"],
"grp": ["x", "y", "x", "y", "x"],
"score": [88.0, 92.5, np.nan, 71.0, 95.0],
"n": [3, 1, 4, 1, 5],
})
df
In R this would be
df <- data.frame(
id = c("a", "b", "c", "d", "e"),
grp = c("x", "y", "x", "y", "x"),
score = c(88, 92.5, NA, 71, 95),
n = c(3, 1, 4, 1, 5),
stringsAsFactors = FALSE
)
1. The mental map¶
| Concept | R | pandas |
|---|---|---|
| Homogeneous array | matrix, array |
numpy.ndarray |
| Labeled 1-D vector | named vector |
pandas.Series |
| Table of columns | data.frame, tibble |
pandas.DataFrame |
| Row labels | rownames(df) |
df.index |
| Column labels | names(df), colnames(df) |
df.columns |
| Missing value | NA, NaN, NULL |
NaN, None, pd.NA |
| Factor | factor |
category dtype |
| List | list |
dict or list |
NULL |
NULL |
None |
| Pipe | %>%, \|> |
method chaining in ( ... ) |
Mind this. R's three missing-ish values collapse to roughly one in pandas: a float
column stores None as NaN, so None and np.nan behave identically there. pd.NA
is the newer sentinel used by the nullable dtypes (Int64, boolean, string).
2. Inspecting¶
| Task | R | pandas |
|---|---|---|
| Dimensions | dim(df) |
df.shape |
| Rows / columns | nrow(df) / ncol(df) |
len(df) / df.shape[1] |
| Column names | names(df) |
df.columns |
| Structure | str(df) |
df.info() |
| First / last rows | head(df) / tail(df) |
df.head() / df.tail() |
| Summary | summary(df) |
df.describe() |
| Column types | sapply(df, class) |
df.dtypes |
| Frequency table | table(x) |
s.value_counts() |
| Two-way table | table(x, y) |
pd.crosstab(x, y) |
| Distinct values | unique(x) |
s.unique() |
| Count distinct | length(unique(x)) |
s.nunique() |
| Any missing? | any(is.na(df)) |
df.isna().any().any() |
| Missing per column | colSums(is.na(df)) |
df.isna().sum() |
| Complete rows | complete.cases(df) |
df.notna().all(axis="columns") |
Mind this. dim, names and shape are attributes in pandas --- no parentheses.
info() and describe() are methods --- parentheses required. Forgetting them on a
method prints <bound method ...> with no error at all.
print(df.shape) # dim(df)
print(df.dtypes.to_dict()) # sapply(df, class)
print(df["grp"].value_counts().to_dict()) # table(grp)
print(df.isna().sum().to_dict()) # colSums(is.na(df))
print(df.notna().all(axis="columns").tolist()) # complete.cases(df)
3. Selecting columns¶
| Task | R (base) | R (dplyr) | pandas |
|---|---|---|---|
| One column, as a vector | df$score |
pull(df, score) |
df["score"] |
| One column, as a table | df["score"] |
select(df, score) |
df[["score"]] |
| Several columns | df[c("id", "n")] |
select(df, id, n) |
df[["id", "n"]] |
| Drop a column | df$score <- NULL |
select(df, -score) |
df.drop(columns="score") |
| By position | df[, 2] |
df.iloc[:, 1] |
|
| Rename | names(df)[1] <- "ID" |
rename(df, ID = id) |
df.rename(columns={"id": "ID"}) |
| Reorder | df[c("n", "id")] |
relocate(df, n) |
df[["n", "id"]] |
Mind this. df["score"] returns a Series; df[["score"]] returns a one-column
DataFrame. Same distinction as pull() vs select().
Avoid attribute access (df.score). It fails on names with spaces, cannot create a new
column, and silently loses to built-ins --- df.size is the element count, not a column
called size.
4. Selecting rows and filtering¶
| Task | R (base) | R (dplyr) | pandas |
|---|---|---|---|
| By position | df[1:3, ] |
slice(df, 1:3) |
df.iloc[0:3] |
| By row label | df["r1", ] |
df.loc["r1"] |
|
| Filter on a condition | df[df$score > 85, ] |
filter(df, score > 85) |
df[df["score"] > 85] |
| Two conditions | df[df$n > 1 & df$grp == "x", ] |
filter(df, n > 1, grp == "x") |
df[(df["n"] > 1) & (df["grp"] == "x")] |
| Membership | df[df$grp %in% c("x"), ] |
filter(df, grp %in% c("x")) |
df[df["grp"].isin(["x"])] |
| Negate | df[!cond, ] |
filter(df, !cond) |
df[~cond] |
| Drop missing | na.omit(df) |
drop_na(df) |
df.dropna() |
| Top n | head(df[order(-df$n), ], 3) |
slice_max(df, n, n = 3) |
df.nlargest(3, "n") |
| Random rows | df[sample(nrow(df), 3), ] |
slice_sample(df, n = 3) |
df.sample(3) |
| Query string | df.query("n > 1 and grp == 'x'") |
Mind this --- three separate traps.
- Use
&,|,~--- never Python'sand/or/not, which cannot work element-wise. - Parenthesize every condition.
&binds more tightly than>, sodf["n"] > 1 & df["grp"] == "x"evaluates in the wrong order and raises. df.loc[0:2]includes the endpoint (3 rows);df.iloc[0:2]excludes it (2 rows). Label slices are the one place Python's half-open rule does not hold.
print(df[df["score"] > 85]["id"].tolist()) # filter
print(df[(df["n"] > 1) & (df["grp"] == "x")]["id"].tolist()) # two conditions
print(df.query("n > 1 and grp == 'x'")["id"].tolist()) # same, as a string
print(len(df.loc[0:2]), len(df.iloc[0:2])) # 3 and 2 -- endpoint rule differs
5. which(), match() and set operations¶
| R | pandas | Returns |
|---|---|---|
which(x > 4) |
np.flatnonzero(s > 4) |
integer positions (0-based) |
s.index[s > 4] |
index labels | |
x[which(x > 4)] |
s[s > 4] |
the values --- preferred |
length(which(cond)) |
cond.sum() |
a count |
which.max(x) |
s.idxmax() |
index label |
s.argmax() |
integer position | |
which(x == max(x)) |
s.index[s == s.max()] |
all ties |
which(is.na(x)) |
s.index[s.isna()] |
labels of the missing |
which(duplicated(x)) |
np.flatnonzero(s.duplicated()) |
positions |
which(m > 5, arr.ind = TRUE) |
np.argwhere((m > 5).to_numpy()) |
row/col positions |
m.stack()[(m > 5).stack()] |
labels and values | |
match(x, table) |
pd.Index(table).get_indexer(x) |
positions, -1 if absent |
| (table has duplicates) | pd.Index(table).get_indexer_for(x) |
every match |
x %in% table |
s.isin(table) |
Boolean mask |
intersect(x, y) |
pd.Index(x).intersection(y) |
in both |
setdiff(x, y) |
pd.Index(x).difference(y) |
in x, not y |
union(x, y) |
pd.Index(x).union(y) |
either |
Mind this. Two differences bite everyone:
- Positions are 0-based, so
which()minus one. - "Not found" is
-1, notNA--- and-1is a legal Python position meaning "last element". Never feedget_indexeroutput toilocwithout filtering on>= 0.
get_indexer also requires the lookup table to be unique, and the Index set
operations sort and deduplicate their result.
s = df["score"]
print(np.flatnonzero(s > 85)) # which(score > 85)
print(s.index[s > 85].tolist()) # the labels
print(s.idxmax(), s.argmax()) # which.max: label vs position
ids = pd.Index(df["id"])
print(ids.get_indexer(["c", "z"])) # match(c("c","z"), id) -> 2, -1 (R: 3, NA)
print(ids.difference(["a", "b"]).tolist()) # setdiff
6. Sorting and ranking¶
| Task | R (base) | R (dplyr) | pandas |
|---|---|---|---|
| Sort a vector | sort(x) |
s.sort_values() |
|
| Sort rows | df[order(df$n), ] |
arrange(df, n) |
df.sort_values("n") |
| Descending | df[order(-df$n), ] |
arrange(df, desc(n)) |
df.sort_values("n", ascending=False) |
| Several keys | df[order(df$grp, -df$n), ] |
arrange(df, grp, desc(n)) |
df.sort_values(["grp", "n"], ascending=[True, False]) |
| Sort by row label | df.sort_index() |
||
| Ordering vector | order(x) |
s.argsort() |
|
| Rank | rank(x) |
min_rank(x) |
s.rank() |
| Reverse | rev(x) |
s[::-1] |
Tie-breaking in rank(method=): "average" (default), "min", "max", "first",
"dense".
Mind this. Missing values sort last by default in both languages, but pandas
lets you say na_position="first". And sort_values returns a new frame --- it
never sorts in place unless you pass inplace=True, which is discouraged.
7. Creating and modifying columns¶
| Task | R (base) | R (dplyr) | pandas |
|---|---|---|---|
| New column | df$z <- df$n * 2 |
mutate(df, z = n * 2) |
df["z"] = df["n"] * 2 |
| Several at once | mutate(df, a = ..., b = ...) |
df.assign(a=lambda d: ..., b=lambda d: ...) |
|
| Conditional | ifelse(x > 2, "big", "small") |
if_else(...) |
np.where(x > 2, "big", "small") |
| Multi-way | cut(x, breaks) |
case_when(...) |
np.select([c1, c2], [v1, v2], default=...) |
| Recode from a table | lookup$v[match(df$k, lookup$k)] |
left_join(df, lookup) |
df["k"].map(lookup.set_index("k")["v"]) |
| Replace values | x[x == -99] <- NA |
na_if(x, -99) |
s.replace(-99, np.nan) |
| Bin, equal width | cut(x, 4) |
pd.cut(x, 4) |
|
| Bin, equal count | cut(x, quantile(x)) |
ntile(x, 4) |
pd.qcut(x, 4) |
| Cumulative | cumsum(x) |
s.cumsum() |
|
| Lag / lead | dplyr::lag(x) |
lag(x) / lead(x) |
s.shift(1) / s.shift(-1) |
| Row-wise difference | diff(x) |
s.diff() |
Mind this. Inside assign, pandas has no data masking --- you cannot write bare
column names the way mutate lets you. Use lambda d: d["col"], which also keeps the
expression correct when an earlier step in the chain changed the rows.
df.assign(z = lambda d: d["n"] * 2) # right
df.assign(z = n * 2) # NameError: name 'n' is not defined
map returns NaN for any key missing from the lookup --- silently. Count the
NaNs afterwards, every time.
out = df.assign(
z = lambda d: d["n"] * 2, # mutate(z = n * 2)
sizec = lambda d: np.where(d["n"] > 2, "big", "small"), # ifelse
lagn = lambda d: d["n"].shift(1), # lag(n)
)
out[["id", "n", "z", "sizec", "lagn"]]
# recode from a lookup table: lookup$v[match(df$k, lookup$k)]
lookup = pd.DataFrame({"grp": ["x", "y"], "label": ["Control", "Treated"]})
recoded = df["grp"].map(lookup.set_index("grp")["label"])
print(recoded.tolist())
print("NaNs introduced:", recoded.isna().sum()) # ALWAYS check
8. Missing data¶
| Task | R | pandas |
|---|---|---|
| Test for missing | is.na(x) |
s.isna() |
| Test for present | !is.na(x) |
s.notna() |
| Count missing | sum(is.na(x)) |
s.isna().sum() |
| Drop missing rows | na.omit(df) |
df.dropna() |
| Drop if all missing | df.dropna(how="all") |
|
| Keep rows with >= k present | df.dropna(thresh=k) |
|
| Fill with a constant | x[is.na(x)] <- 0 |
s.fillna(0) |
| Fill per column | df.fillna({"a": 0, "b": -1}) |
|
| Carry forward | zoo::na.locf(x) |
s.ffill() |
| Carry backward | s.bfill() |
|
| Mean imputation | x[is.na(x)] <- mean(x, na.rm = TRUE) |
s.fillna(s.mean()) |
| Ignore NA in a summary | mean(x, na.rm = TRUE) |
s.mean() (default) |
| Include NA in a summary | mean(x) |
s.mean(skipna=False) |
Mind this --- the defaults are opposite. R's mean(x) returns NA if anything is
missing and you must ask for na.rm = TRUE. pandas skips missing by default; you
must ask for skipna=False to propagate. That one reversal silently changes results
when translating code in either direction.
thresh=k counts the values that are present, not the ones that are missing.
print("count missing :", df["score"].isna().sum())
print("pandas mean :", round(df["score"].mean(), 2), " (R: mean(score, na.rm=TRUE))")
print("skipna=False :", df["score"].mean(skipna=False), " (R: mean(score))")
print("ffill :", df["score"].ffill().tolist())
print("dropna rows :", len(df.dropna()), "of", len(df))
9. Summary statistics¶
| Task | R | pandas |
|---|---|---|
| Mean / median | mean(x) / median(x) |
s.mean() / s.median() |
| SD / variance | sd(x) / var(x) |
s.std() / s.var() |
| Min / max | min(x) / max(x) |
s.min() / s.max() |
| Range | range(x) |
(s.min(), s.max()) |
| Sum / product | sum(x) / prod(x) |
s.sum() / s.prod() |
| Quantiles | quantile(x, .25) |
s.quantile(.25) |
| Correlation | cor(x, y) |
s.corr(t) |
| Correlation matrix | cor(df) |
df.corr(numeric_only=True) |
| Covariance | cov(x, y) |
s.cov(t) |
| Column means | colMeans(df) |
df.mean(numeric_only=True) |
| Row means | rowMeans(df) |
df.mean(axis="columns") |
| Count non-missing | sum(!is.na(x)) |
s.count() |
| Length | length(x) |
len(s) or s.size |
| All / any | all(cond) / any(cond) |
cond.all() / cond.any() |
Mind this. s.std() uses the sample standard deviation (ddof=1), matching R's
sd(). NumPy's np.std() uses ddof=0 by default and will not match --- a classic
source of "why are my numbers slightly off?"
Also: s.count() is the number of non-missing values (R's sum(!is.na(x))), while
len(s) and s.size count every row.
x = df["score"]
print("pandas s.std():", round(x.std(), 4), " <- ddof=1, matches R's sd()")
print("numpy np.std() :", round(np.std(x.dropna()), 4), " <- ddof=0, does NOT match")
print("count vs len :", x.count(), len(x))
10. Split--apply--combine¶
| Task | R (base) | R (dplyr) | pandas |
|---|---|---|---|
| Group means | tapply(x, g, mean) |
group_by(g) %>% summarize(m = mean(x)) |
df.groupby("g")["x"].mean() |
| Group sizes | table(g) |
count(df, g) |
df.groupby("g").size() |
| Several statistics | aggregate(...) |
summarize(m = mean(x), s = sd(x)) |
df.groupby("g")["x"].agg(["mean", "std"]) |
| Named outputs | summarize(avg = mean(x)) |
df.groupby("g").agg(avg=("x", "mean")) |
|
| Two keys | group_by(g1, g2) |
df.groupby(["g1", "g2"]) |
|
| Keys as columns | (default) | df.groupby("g", as_index=False) |
|
| Broadcast back | ave(x, g) |
group_by(g) %>% mutate(m = mean(x)) |
df.groupby("g")["x"].transform("mean") |
| Filter groups | group_by(g) %>% filter(n() > 2) |
df.groupby("g").filter(lambda d: len(d) > 2) |
|
| Arbitrary function | by(df, g, f) |
group_modify(f) |
df.groupby("g").apply(f) |
| Split into pieces | split(df, g) |
group_split(df, g) |
dict(list(df.groupby("g"))) |
agg vs transform is the distinction to internalize:
aggreturns one row per group --- it issummarize().transformreturns one row per original row --- it ismutate()after agroup_by(), and it is what you need for group-centering.
Mind this. Always report the group size beside a group statistic. Put
n=("col", "size") first in every agg call: a mean over 3 observations is not the
same kind of number as a mean over 300.
# summarize: one row per group
print(df.groupby("grp").agg(n=("id", "size"),
avg=("score", "mean")).round(2))
print()
# mutate-after-group_by: one row per original row
print(df.assign(grp_mean=lambda d: d.groupby("grp")["score"].transform("mean"))
[["id", "grp", "score", "grp_mean"]].round(2))
11. Joining tables¶
| Join | R (base) | R (dplyr) | pandas |
|---|---|---|---|
| Inner | merge(a, b) |
inner_join(a, b, by = "k") |
a.merge(b, on="k") |
| Left | merge(a, b, all.x = TRUE) |
left_join(a, b, by = "k") |
a.merge(b, on="k", how="left") |
| Right | merge(a, b, all.y = TRUE) |
right_join(a, b, by = "k") |
a.merge(b, on="k", how="right") |
| Full | merge(a, b, all = TRUE) |
full_join(a, b, by = "k") |
a.merge(b, on="k", how="outer") |
| Different key names | merge(a, b, by.x=, by.y=) |
join(a, b, by = c("k1" = "k2")) |
a.merge(b, left_on="k1", right_on="k2") |
| Several keys | merge(a, b, by = c("k1","k2")) |
by = c("k1","k2") |
on=["k1", "k2"] |
| On the row index | a.join(b) |
||
| Keep matches only | semi_join(a, b) |
a[a["k"].isin(b["k"])] |
|
| Keep non-matches | anti_join(a, b) |
a[~a["k"].isin(b["k"])] |
|
| Stack rows | rbind(a, b) |
bind_rows(a, b) |
pd.concat([a, b]) |
| Stack columns | cbind(a, b) |
bind_cols(a, b) |
pd.concat([a, b], axis="columns") |
Mind this --- the most expensive mistake in this document. If the right table has
duplicate keys, a merge silently multiplies rows, and how="left" does not
protect you: how= controls which keys survive, not the row count.
before = len(a)
out = a.merge(b, on="k", how="left")
assert len(out) == before # cheapest insurance in pandas
b["k"].is_unique # or check first
a.merge(b, on="k", how="left", validate="m:1") # or let pandas refuse
Overlapping non-key column names get _x / _y suffixes rather than an error. Control
them with suffixes=("_a", "_b").
b = pd.DataFrame({"grp": ["x", "y"], "boss": ["Ana", "Ben"]})
print("rows:", len(df), "->", len(df.merge(b, on="grp", how="left")))
dup = pd.DataFrame({"grp": ["x", "x", "y"], "boss": ["Ana", "Alt", "Ben"]})
print("with a duplicated key:", len(df), "->", len(df.merge(dup, on="grp", how="left")))
try:
df.merge(dup, on="grp", how="left", validate="m:1")
except Exception as e:
print(type(e).__name__, "->", e)
12. Reshaping¶
| Task | R (tidyr) | R (base / reshape2) | pandas |
|---|---|---|---|
| Wide to long | pivot_longer(df, cols) |
melt(df, id.vars) |
pd.melt(df, id_vars=...) |
| Long to wide | pivot_wider(df, names_from, values_from) |
dcast(df, ...) |
df.pivot(index=, columns=, values=) |
| Long to wide, aggregating | dcast(df, ..., fun.aggregate = mean) |
df.pivot_table(index=, columns=, aggfunc=) |
|
| Columns into rows | df.stack() |
||
| Index level into columns | df.unstack() |
||
| Transpose | t(df) |
df.T |
|
| Column to row index | rownames(df) <- df$id |
df.set_index("id") |
|
| Row index to column | df.reset_index() |
||
| Cross-tabulation | table(a, b) |
xtabs(~ a + b) |
pd.crosstab(a, b) |
Mind this. pivot fails if the index/column pair is not unique --- that is a feature,
since it means you have duplicates you did not know about. pivot_table aggregates them
instead (mean by default), which is convenient but can hide the same problem.
long = pd.melt(df, id_vars=["id", "grp"], value_vars=["score", "n"])
print(long.head(4))
print()
print(long.pivot(index="id", columns="variable", values="value"))
13. Strings¶
pandas puts vectorized string methods behind the .str accessor. They skip missing
values instead of raising.
| Task | R (base) | R (stringr) | pandas |
|---|---|---|---|
| Length | nchar(x) |
str_length(x) |
s.str.len() |
| Upper / lower | toupper(x) |
str_to_upper(x) |
s.str.upper() |
| Title case | str_to_title(x) |
s.str.title() |
|
| Trim whitespace | trimws(x) |
str_trim(x) |
s.str.strip() |
| Concatenate | paste0(x, y) |
str_c(x, y) |
x + y |
| Collapse to one string | paste(x, collapse=",") |
str_c(x, collapse=",") |
",".join(s) |
| Substring | substr(x, 1, 3) |
str_sub(x, 1, 3) |
s.str[:3] |
| Contains | grepl("a", x) |
str_detect(x, "a") |
s.str.contains("a") |
| Replace first | sub("a", "b", x) |
str_replace(x, "a", "b") |
s.str.replace("a", "b", n=1) |
| Replace all | gsub("a", "b", x) |
str_replace_all(...) |
s.str.replace("a", "b") |
| Split | strsplit(x, ",") |
str_split(x, ",") |
s.str.split(",") |
| Which match | grep("a", x) |
str_which(x, "a") |
np.flatnonzero(s.str.contains("a")) |
| Extract groups | regmatches(...) |
str_match(x, re) |
s.str.extract(r"(\w+)@(\w+)") |
| Pad | formatC(x, width=5) |
str_pad(x, 5) |
s.str.pad(5) |
| Starts / ends with | startsWith(x, "a") |
str_starts(x, "a") |
s.str.startswith("a") |
Mind this. str.replace replaces every occurrence by default --- it is gsub,
not sub. Pass n=1 for sub behaviour. And since pandas 2.0 it treats the pattern as
a literal unless you pass regex=True.
14. Factors and categoricals¶
| Task | R | pandas |
|---|---|---|
| Create | factor(x) |
s.astype("category") |
| With an order | factor(x, levels=, ordered=TRUE) |
pd.Categorical(s, categories=, ordered=True) |
| Levels | levels(f) |
s.cat.categories |
| Integer codes | as.integer(f) |
s.cat.codes |
| Rename levels | levels(f) <- c(...) |
s.cat.rename_categories([...]) |
| Add a level | s.cat.add_categories([...]) |
|
| Drop unused | droplevels(f) |
s.cat.remove_unused_categories() |
| Back to text | as.character(f) |
s.astype(str) |
| Counts incl. empty levels | table(f) |
s.value_counts() (keeps empty categories) |
| Dummy variables | model.matrix(~ f) |
pd.get_dummies(s, drop_first=True) |
Mind this. R's as.integer(factor) is 1-based; pandas cat.codes is 0-based
and uses -1 for missing. And groupby on a categorical keeps unused categories by
default --- pass observed=True to drop them, or you get rows of NaN for groups that
do not exist.
15. The apply family¶
| R | pandas / NumPy | Note |
|---|---|---|
sapply(x, f) / vapply |
s.map(f) |
element-wise over a Series |
lapply(x, f) |
[f(i) for i in x] |
returns a list |
apply(m, 2, f) |
df.apply(f) |
once per column (the default) |
apply(m, 1, f) |
df.apply(f, axis="columns") |
once per row |
mapply(f, x, y) |
np.vectorize(f)(x, y) |
or just write it vectorized |
tapply(x, g, f) |
df.groupby(g)["x"].agg(f) |
split--apply--combine |
Map(f, x, y) |
zip(x, y) in a comprehension |
|
Reduce(f, x) |
functools.reduce(f, x) |
|
do.call(rbind, lst) |
pd.concat(lst) |
|
| element-wise over a frame | df.map(f) |
every cell |
Mind this --- the axis numbering is the mirror image of R's.
| What you want | R | pandas |
|---|---|---|
| one value per column | apply(m, 2, f) |
df.apply(f) or axis=0 |
| one value per row | apply(m, 1, f) |
df.apply(f, axis=1) |
Read axis= as "the axis that disappears." axis=0 collapses the rows, so you get
one number per column.
Prefer vectorized operations over apply whenever one exists: df["x"] * 2 is orders of
magnitude faster than df["x"].apply(lambda v: v * 2), and shorter.
num = df[["score", "n"]]
print(num.apply(lambda c: c.max() - c.min())) # apply(m, 2, f)
print()
print(num.apply(lambda r: r["n"] * 2, axis="columns").tolist()) # apply(m, 1, f)
16. Dates¶
| Task | R (base) | R (lubridate) | pandas |
|---|---|---|---|
| Parse | as.Date(x) |
ymd(x) |
pd.to_datetime(x) |
| Today | Sys.Date() |
today() |
pd.Timestamp.today() |
| Year / month / day | format(d, "%Y") |
year(d) |
s.dt.year |
| Day of week | weekdays(d) |
wday(d, label=TRUE) |
s.dt.day_name() |
| Format out | format(d, "%Y-%m") |
s.dt.strftime("%Y-%m") |
|
| Difference | d2 - d1 |
s2 - s1 (a Timedelta) |
|
| Add days | d + 7 |
d + days(7) |
s + pd.Timedelta(days=7) |
| Sequence | seq(d1, d2, "day") |
pd.date_range(d1, d2, freq="D") |
|
| Round down | floor_date(d, "month") |
s.dt.to_period("M") |
Like .str, the .dt accessor is where all the date methods live --- and it only
works once the column is actually a datetime dtype, which pd.to_datetime gives you.
17. Reading and writing¶
| Format | R | pandas |
|---|---|---|
| CSV | read.csv(f) / readr::read_csv(f) |
pd.read_csv(f) |
| Tab-separated | read.delim(f) |
pd.read_csv(f, sep="\t") |
| Excel | readxl::read_excel(f) |
pd.read_excel(f, sheet_name=0) |
| SAS | haven::read_sas(f) |
pd.read_sas(f) |
| Stata | haven::read_dta(f) |
pd.read_stata(f) |
| Fixed width | read.fwf(f) |
pd.read_fwf(f) |
| JSON | jsonlite::fromJSON(f) |
pd.read_json(f) |
| From a database | DBI::dbGetQuery(con, q) |
pd.read_sql(q, con) |
| Write CSV | write.csv(df, f, row.names=FALSE) |
df.to_csv(f, index=False) |
| Save native | saveRDS(df, f) |
df.to_parquet(f) or to_pickle(f) |
Useful read_csv arguments: sep, header, names, index_col, na_values,
skiprows, nrows, dtype, parse_dates.
Mind this. df.to_csv(f) writes the row index as an unnamed first column by default.
Pass index=False --- otherwise every round trip through CSV grows a junk
Unnamed: 0 column.
18. False friends¶
Same name, different behaviour. These are the ones that cost hours.
| Looks like | In R | In pandas |
|---|---|---|
df[1:3] |
rows 1 to 3 | columns sliced, or rows for a slice --- ambiguous; use loc/iloc |
x[0] |
error (1-based) | the first element (0-based) |
a:b slice |
includes b |
excludes b --- except in .loc, which includes it |
mean(x) |
NA if anything is missing |
skips missing by default |
sd(x) |
ddof = 1 |
s.std() is ddof=1, but np.std() is ddof=0 |
sub vs gsub |
distinct functions | str.replace is gsub; pass n=1 for sub |
&& / || |
scalar and/or | no element-wise meaning --- use & / | |
match() not found |
NA |
-1, which is a valid position |
which.max ties |
first only | idxmax first only --- same, but returns a label |
merge(all.x=TRUE) |
keeps left keys | how="left" keeps left keys but can add rows |
df$col <- v |
always works | df["col"] = v works; df.col = v silently makes an attribute |
apply(m, 1, f) |
per row | axis=1 per row --- the numbering is reversed |
| factor codes | 1-based | cat.codes 0-based, -1 for missing |
rev(x) |
reverses | s[::-1] reverses values and labels together |
19. The five silent failures¶
None of these raises an error. Your code runs; your numbers are wrong.
Alignment. Operations match on labels, not position. Two Series with different indexes give you
NaN, not an error. Guard: check.isna().sum()after arithmetic between two Series.Chained assignment.
df[mask]["col"] = xmodifies a temporary copy and may do nothing at all. Guard: write one.loc[rows, cols] = xcall, always.and/or. Python's keywords cannot work element-wise. Guard:&,|,~, and parenthesize every condition.Silent
NaNfrommap. A key missing from the dictionary becomes missing data with no warning, andvalue_counts()then hides it. Guard:recoded.isna().sum()after everymap.Merges that grow. Duplicate keys multiply rows, and
how="left"does not prevent it. Guard:assert len(out) == len(left), or passvalidate="m:1".
Further reading¶
- pandas, Comparison with R / R libraries --- https://pandas.pydata.org/docs/getting_started/comparison/comparison_with_r.html
- McKinney, W. Python for Data Analysis, 3rd ed. --- https://wesmckinney.com/book/
- VanderPlas, J. Python Data Science Handbook, Ch. 3 --- https://jakevdp.github.io/PythonDataScienceHandbook/
- pandas user guide --- https://pandas.pydata.org/docs/user_guide/
- Course lectures:
PandasEssentials,TipsAnalysis,PandasDataManipulation