pip, PyPI, and pinning dependencies

Understanding pip, PyPI, and pinning dependencies

PyPI is a package index: a catalogue of project names, each with a list of released versions and files (wheels, and sometimes source archives). pip is a client for that index. When you run `python -m pip install requests`, pip asks PyPI which versions of `requests` exist, picks one, downloads the matching wheel, and unpacks it into the `site-packages` directory of the interpreter that ran pip. That last detail is why `python -m pip` is safer than a bare `pip`: `pip` on your PATH may belong to a different Python than the one you use for `python script.py`, and then the install succeeds while the import still fails.

The important mental shift is that a requirement is not a version, it is a set of acceptable versions. `requests` means "anything", `requests>=2.28` means "anything at least that new", `requests~=2.31.0` means "2.31.x, at least .0", and only `requests==2.31.0` names one point. pip resolves all of these sets at once, including the requirements declared by your dependencies' dependencies, and prefers the newest version that satisfies everything. Because PyPI keeps gaining new releases, any unpinned set makes your install a function of the day you ran it, which is exactly how a project that "worked last month" breaks with no code change.

So real projects use two layers. The abstract layer describes what your code can tolerate — ranges in `pyproject.toml` or a `requirements.in` — and a library should stay here, because an exact pin in published metadata collides with every other package that wants a different version of the same dependency. The concrete layer is a fully pinned snapshot of one working environment: every package, direct and transitive, at an exact version, which is what `python -m pip freeze > requirements.txt` produces and what `pip install -r requirements.txt` reproduces. Deployments and CI install the concrete layer; humans edit the abstract one and regenerate the pins deliberately.

def split_req(line):
    for op in ("==", "~=", ">=", ">"):
        if op in line:
            name, version = line.split(op, 1)
            return name, op, version
    return line, None, None


for line in ["requests", "requests>=2.28", "requests~=2.31.0", "requests==2.31.0"]:
    name, op, version = split_req(line)
    if op is None:
        note = "newest version on PyPI at install time"
    elif op == "==":
        note = f"exactly {version}, every time"
    else:
        note = f"anything matching {op}{version}"
    print(f"{line:18} -> {name}: {note}")

Worked examples

def compatible(spec, candidate):
    base = [int(p) for p in spec.split(".")]
    cand = [int(p) for p in candidate.split(".")]
    return cand >= base and cand[:len(base) - 1] == base[:len(base) - 1]

for v in ["2.31.0", "2.31.4", "2.32.0", "3.0.0", "2.30.9"]:
    print(f"~=2.31.0 accepts {v}? {compatible('2.31.0', v)}")

versions = ["2.9.0", "2.10.0", "2.2.0"]
print("as text: ", sorted(versions))
print("as numbers:", sorted(versions, key=lambda v: tuple(int(p) for p in v.split("."))))
print("newest:   ", max(versions, key=lambda v: tuple(int(p) for p in v.split("."))))

installed = {"requests": "2.31.0", "urllib3": "2.2.1", "certifi": "2024.2.2"}

lock = "\n".join(f"{name}=={ver}" for name, ver in sorted(installed.items()))
print(lock)
print("--- reinstalling from the file ---")

restored = dict(line.split("==") for line in lock.splitlines())
print(len(restored), "packages pinned")
print("urllib3 ->", restored["urllib3"])

Important notes

Common mistakes

Try it yourself