PYTHON / PANDAS
groupby: split, apply, combine
Split a DataFrame by column values, aggregate or transform each group, and control whether keys become the index or stay as columns.
What you will learn
- Build a GroupBy object and iterate it to see the actual row splits
- Aggregate with agg() and named outputs to get one row per group
- Use transform() to broadcast a group result back onto every original row
- Control key placement and NaN keys with as_index, sort, and dropna
Understanding groupby: split, apply, combine
groupby works in three stages: split the rows into pieces keyed by one or more column values, apply a function to each piece, and combine the pieces into a new object. Calling df.groupby("region") performs only the split bookkeeping - it records which row positions belong to which key and returns a DataFrameGroupBy, which is why printing it shows an object repr instead of a table. Nothing is computed until you name an operation, because pandas cannot know what shape the combined result should have until it knows what you do to each piece.
The shape of the output depends on which family of operations you apply. Aggregations such as sum, mean, or agg collapse each piece to a single row, so the result has one row per group and the group keys become the index. transform is different: it computes a per-group value and then broadcasts it back to every original row, returning something the same length and index as the input, which is exactly what makes it assignable as a new column. filter is a third family - it keeps or discards whole groups based on a function that returns True or False per group.
Group keys arrive sorted because pandas builds the result from the sorted unique keys; pass sort=False to keep first-appearance order and skip that sort. If you would rather have the keys as ordinary columns than as an index, use as_index=False or call reset_index() afterwards. Rows whose grouping key is NaN are dropped by default (dropna=True), and this is the usual reason a groupby total comes out smaller than the total of the whole column.
import pandas as pd
sales = pd.DataFrame({
"region": ["north", "south", "north", "west", "south", "north"],
"rep": ["ana", "bo", "cy", "dee", "eli", "fay"],
"units": [10, 4, 7, 12, 6, 3],
})
grouped = sales.groupby("region")
for name, part in grouped:
print(name, list(part["units"]))
print()
print(grouped["units"].sum())groupby splits rows by key and the operation you apply decides whether the result is one row per group or one value per original row.
Worked examples
Several aggregations at once
Produces one row per region with differently named columns computed from different source columns.
import pandas as pd
df = pd.DataFrame({
"region": ["north", "south", "north", "west", "south", "north"],
"rep": ["ana", "bo", "cy", "dee", "eli", "fay"],
"units": [10, 4, 7, 12, 6, 3],
})
summary = df.groupby("region").agg(
total_units=("units", "sum"),
reps=("rep", "nunique"),
)
print(summary)Example explained
Line 1Each keyword argument to agg names an output column, so total_units and reps come straight from the argument names.
Line 2The tuple ("units", "sum") says which column to read and which reduction to apply, letting one call mix columns.
Line 3nunique on the rep column counts distinct reps per region, which is why north shows 3 rather than the sum of anything.
Line 4region is the index of the result, not a column, because agg is an aggregation and the keys were used to build the index.
transform to compute each row's share of its group
Shows how transform returns a value per original row so it can be assigned back as a column.
import pandas as pd
df = pd.DataFrame({
"region": ["north", "south", "north", "west", "south", "north"],
"units": [10, 4, 7, 12, 6, 3],
})
df["region_total"] = df.groupby("region")["units"].transform("sum")
df["share"] = (df["units"] / df["region_total"]).round(2)
print(df)Example explained
Line 1transform("sum") computes 20, 10 and 12 once per region, then repeats each value on every row of that region.
Line 2The returned Series keeps the original 0-5 index, which is why the assignment to df["region_total"] lines up correctly.
Line 3Rows 0, 2 and 5 all show 20 because they are the three north rows sharing the same group total.
Line 4The share column is plain row-wise division afterwards; groupby is no longer involved once the totals are broadcast.
Important notes
Group keys are sorted by default; pass sort=False when you want first-appearance order or need the extra speed on many groups.
In pandas 2.x, calling mean() on a group of a DataFrame that still holds text columns raises a TypeError instead of skipping them, so select the numeric columns first or pass numeric_only=True.
Common mistakes
Printing df.groupby("region") and expecting a table; you get a DataFrameGroupBy repr because the split is lazy and no aggregation has been requested yet.
Using an aggregation such as sum() and then trying to assign it back with df["total"] = ...; the result is indexed by group key, not by the original row index, so you get NaN or a length error instead of the per-row value that transform would give.
Grouping on a column that contains NaN and trusting the totals; those rows are excluded by default, so the group sums add up to less than df["units"].sum() with no warning.
Try it yourself
Change, predict, then run
Using the sales frame from the main example, produce a table with one row per region showing the total units and the largest single order, then add a column to the original frame holding each row's units minus its region mean.
Open the Python workspaceCheck your understanding
You want a new column giving each row's units as a fraction of its region total. Why does grouped["units"].sum() fail for this while transform("sum") works?
- sum() returns one value per group indexed by the group key, so it cannot align to the original row index, while transform broadcasts each group's result back to every row of that group
- sum() only works on numeric columns, while transform works on any dtype
- sum() drops missing values and transform keeps them, so the two results have different lengths
- sum() returns a DataFrame and transform returns a Series, and only a Series can be assigned to a column
Show answer
Assignment to a column aligns on the index, and an aggregation's index is the group keys (north, south, west), which do not match the original 0-5 row labels; transform deliberately returns the input's index and length so alignment succeeds. Option 3 is tempting but wrong: called on the single selected column grouped["units"], both sum() and transform() return a Series - the difference is shape and index, not type.