CS Fundamentals Trainer · Guides

Python packaging, environments and imports

How an import finds and runs a module, sys.path and shadowing, relative and circular imports, what a virtual environment really is, pyproject.toml and editable installs, version specifiers, lock files and dependency security.

18 min read

"It works on my machine" is the most expensive sentence in software, and in Python it usually has one of three causes: a different interpreter, different installed package versions, or a different idea of where an import comes from. This guide explains the mechanisms behind all three: how Python finds and runs modules, why virtual environments exist and what they actually are, what a package is and how pyproject.toml describes one, how version specifiers and lock files make installs repeatable, and what the supply-chain risks are. Each part was run on CPython 3.13 (scripts in the appendix). It builds on how Python works and connects to the dependency advice in web application security.

One limit: the sandbox where I ran this had no access to the Python Package Index (PyPI), so nothing here was downloaded. I used only the interpreter, venv, pip and setuptools that were already installed, and everything about resolvers, lock files and tools such as uv, poetry or pip-tools is from their documentation, not from a run.

1. How an import works

import shop.prices does four things, in order:

  1. Look in sys.modules, the dictionary of modules already loaded. If the name is there, return that module and stop.
  2. Otherwise find the module: walk the folders in sys.path and look for shop/ (a package) or shop.py.
  3. Run the module's code from top to bottom, in a new namespace. The names it defines become the module's attributes.
  4. Store the module in sys.modules and bind the name in the importing code.

I tested step one. Importing shop.prices three times printed shop.prices runs once, and 'shop.prices' in sys.modules was True. A module's top-level code runs once per process, so keep side effects (opening connections, reading files, starting threads) out of import time; do them in functions called from main. The same module imported by two different names (for example by a relative and an absolute path) can be loaded twice as two objects, which is a classic source of confusing bugs.

Step three also explains the bytecode cache: after importing shop.prices, the folder shop/__pycache__/ held __init__.cpython-313.pyc and prices.cpython-313.pyc. Compiled bytecode is saved there so the next start can skip compiling; delete the folder freely, it is regenerated.

sys.path: where Python looks

sys.path is an ordinary list of folders. In order, it contains the folder of the script being run (or '' for the current folder with -c and the interactive prompt), the folders in the PYTHONPATH environment variable, the standard library, and the site-packages folder where installed packages live. I confirmed each: python whereami.py put the script's own folder first, python -c put '' first, and a folder named in PYTHONPATH appeared in the list.

The first match wins, which leads to the most baffling bug in this area. I created a file called random.py next to my script that defined a random() function returning 'not random'. import random loaded my file instead of the standard library, because the script's folder comes first. Never name your modules after standard-library modules (random.py, json.py, email.py, test.py, types.py), and if an import returns something strange, print module.__file__ to see where it came from.

Packages and relative imports

A folder with an __init__.py is a package (a folder without one can still work as a "namespace package", but add the file for regular code). Inside a package you can import siblings with a leading dot: from .prices import gross.

Relative imports only work when the module is imported as part of its package. I ran the same file two ways:

CommandResult
python shop/cli.pyImportError: attempted relative import with no known parent package
python -m shop.cligross(100) = 120.0

Running a file as a script makes it the top-level __main__ with no package around it; python -m package.module runs it as a member of the package. Use -m for anything inside a package, or better, install the project (section 3) and expose a console script. This is also what the if __name__ == "__main__": guard is for: __name__ is "__main__" only when the file is run directly, so the code under the guard does not run when the module is imported.

Circular imports

If a imports b and b imports a, one of them sees the other half-initialised, because the module is stored in sys.modules before its code finishes. Two variations that I ran:

ImportsResult
import b inside a.py, import a inside b.py, using a.fa() only inside a functionworks (a is fully loaded by the time the function is called)
from c import fc inside d.py, and from d import fd inside c.pyImportError: cannot import name 'fd' from 'd' (the error also suggests renaming the file, which is not the cause here)

import module style tolerates cycles because the attribute is looked up later; from module import name needs the name to exist at that moment. Fixes, best first: restructure so the shared part lives in a third module that both import; move the import inside the function that needs it; or import the module instead of the name. A cycle is usually a sign that two modules are really one, or that a dependency points the wrong way.

Import cost

Imports are not free. A bare interpreter start took about 14 ms on my machine and a start that imported numpy took about 92 ms; python -X importtime -c "import json" printed a per-module breakdown, with json costing roughly 9 ms cumulatively (mostly json.decoder and re) on a cold start. For command-line tools and serverless functions, startup time is user-visible. Import heavy libraries lazily inside the function that uses them, and use -X importtime to find what is slow.

2. Virtual environments

The problem

All the Python programs on a machine share one site-packages unless told otherwise. Project A needs library==1.4, project B needs library==2.0, and the operating system's own tools need yet another. Installing globally also risks breaking the system (many Linux distributions now refuse pip install into the system Python for this reason). A virtual environment gives each project its own site-packages.

What a venv is

It is not a virtual machine or a copy of Python. I created one with python3 -m venv .venv and looked inside:

So a venv is a folder with a launcher that points at a real interpreter and a private place to install packages. That has consequences: you do not move or copy a venv to another machine (paths are absolute); you recreate it from a requirements file. You do not need to "activate" it either: running .venv/bin/python directly uses it, and activate mostly edits PATH for convenience. Do not commit .venv/ to version control; do commit the files that describe it (section 4). Use one venv per project, and keep it out of Docker images' build context.

Interpreters

A venv is tied to the Python minor version that created it (3.13 here). Declare the supported versions in pyproject.toml (requires-python), test on each, and pin the one you deploy. Tools such as pyenv or uv python install and select interpreter versions; in containers, the base image does.

3. Packages and pyproject.toml

A distribution package is what you install (pip install shop-demo); it contains one or more import packages (import shop). The metadata describing it lives in pyproject.toml, the standard configuration file (PEP 518 and PEP 621):

[build-system]
requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta"

[project]
name = "shop-demo"
version = "0.1.0"
requires-python = ">=3.9"
dependencies = []

[project.scripts]
shop-hello = "shop.cli:main"

I installed that tiny project in editable mode into a fresh venv (pip install -e ., with --no-build-isolation because the sandbox could not download the build tools) and it worked: the shop-hello command printed hello from an installed console script, and importlib.metadata.version('shop-demo') returned 0.1.0. An editable install links the venv to your source folder, so edits take effect without reinstalling; this is the normal way to work on a project. A plain pip install . failed in the sandbox only because the default "build isolation" downloads the build backend from PyPI, which was not reachable; on a normal machine it works.

Distributions are published in two forms: a wheel (.whl, a ready-to-unpack archive, fast to install) and an sdist (source, built on install). Prefer wheels. Libraries and applications differ: a library should declare loose dependency ranges so it can coexist with others; an application should install exact pinned versions so its environment is reproducible.

4. Versions, requirements and lock files

Version specifiers

Dependencies carry a version constraint written in the PEP 440 syntax. I evaluated six specifiers against the available versions 1.9.0, 2.0.0, 2.1.0, 2.1.5, 2.2.0, 3.0.0 with the packaging library, the one pip itself uses:

SpecifierMatches
==2.1.02.1.0
==2.1.*2.1.0, 2.1.5
~=2.1 ("compatible release")2.1.0, 2.1.5, 2.2.0
~=2.1.02.1.0, 2.1.5
>=2.0,<32.0.0, 2.1.0, 2.1.5, 2.2.0
!=2.1.5,>=2.02.0.0, 2.1.0, 2.2.0, 3.0.0

Two details bite people. ~=2.1 allows any 2.x at or above 2.1, while ~=2.1.0 allows only 2.1.x: the number of components decides. And versions are not strings: "10.0" < "9.0" is True, but Version("10.0") < Version("9.0") is False. Never compare version numbers as text (and do not parse them with ad-hoc split); use packaging.version.

Reproducible installs

Constraints say which versions are acceptable; a lock file records the exact versions (and hashes) that were chosen, including indirect dependencies. Without one, two installs a week apart can get different versions and behave differently, and a broken release of a deep dependency breaks your build overnight. The ways to get one:

ApproachWhat it gives
pip freeze > requirements.txtthe exact versions installed right now (I listed a venv and got lines such as anyio==4.15.1); simple but mixes direct and indirect dependencies and has no hashes
pip-tools (pip-compile)a pinned requirements.txt generated from a short input file, with optional hashes
uv, poetry, pdma project tool with its own lock file (uv.lock, poetry.lock, pdm.lock) and an environment manager
A constraints file or hash-checking modepip install --require-hashes -r requirements.txt refuses any file whose hash differs

(From the tools' documentation; I could not run them offline.) Whichever you use, commit the lock file, install from it in CI and production, update it deliberately (a pull request that bumps versions, with the tests run), and keep development tools (test runner, linters) in a separate group from runtime dependencies. In a container, install dependencies in an earlier layer than your code, so editing a source file does not reinstall everything (see containers and Kubernetes).

5. Security of dependencies

An install runs other people's code, sometimes at install time. The risks and the controls:

6. Common mistakes

7. Questions interviewers like to ask

"What is a virtual environment and why use one?"

A directory with its own site-packages and a launcher pointing at a real interpreter, so each project has isolated dependencies. sys.prefix differs from sys.base_prefix inside it. It is not a VM and should be recreated from a lock file, not copied.

"What happens when you import a module?"

Check sys.modules; otherwise find it along sys.path, execute its code once in a new namespace, store it in sys.modules, and bind the name. That is why module-level code runs once and why a local file can shadow a standard module.

"Why do circular imports fail?"

A module is in sys.modules before its code finishes, so the other module can see a half-built one. from x import name fails if the name is not defined yet; import x often works. Fix by restructuring.

"What does if __name__ == '__main__': do?"

It runs the code only when the file is executed directly, not when it is imported.

"~=2.1 versus ==2.1.*?"

~=2.1 means >=2.1, ==2.* (so 2.2 is allowed); ==2.1.* means exactly the 2.1 series.

"How do you make a Python deployment reproducible?"

Pin the interpreter version, install from a lock file with hashes into a fresh environment or image layer, build once and promote the same artifact, and update dependencies through reviewed changes.

"What is the difference between a wheel and an sdist?"

A wheel is a prebuilt archive that is simply unpacked on install; an sdist is source that must be built, which is slower and can run arbitrary build code.

8. Things to mention in an interview

Quick recap

Appendix: the experiment code

Python 3.13 on Linux, using the interpreter, venv, pip and setuptools that came with it and the packaging library (installed with pip); nothing was downloaded. Paths and the sizes you see will differ. venv_demo.sh and install_demo.sh are bash scripts; run install_demo.sh from a folder that contains project_demo/ (the pyproject.toml above plus src/shop/__init__.py and src/shop/cli.py with def main(): print('hello from an installed console script')).

imports_demo.py: builds a throwaway project and runs the import experiments (caching, relative imports, cycles, shadowing, sys.path, import time, __pycache__):

"""Builds a throwaway project in a temp folder and runs small import experiments against it."""
import os, subprocess, sys, tempfile, textwrap, time

root = tempfile.mkdtemp()
def write(path, text):
    full = os.path.join(root, path); os.makedirs(os.path.dirname(full), exist_ok=True)
    open(full, "w").write(textwrap.dedent(text))
def run(*args, cwd=root, env=None):
    r = subprocess.run([sys.executable, *args], cwd=cwd, capture_output=True, text=True, env=env)
    return (r.stdout + r.stderr).strip()

write("shop/__init__.py", "print('  shop/__init__ runs')\n")
write("shop/prices.py", "print('  shop.prices runs')\nTAX = 0.2\ndef gross(p): return p * (1 + TAX)\n")
write("shop/cli.py", """
    from .prices import gross
    def main(): print('gross(100) =', gross(100))
    if __name__ == '__main__': main()
""")
write("a.py", "import b\ndef fa(): return 'a'\n")
write("b.py", "import a\ndef fb(): return a.fa() + 'b'\n")
write("c.py", "from d import fd\ndef fc(): return 'c'\n")
write("d.py", "from c import fc\ndef fd(): return fc()\n")

print("1. modules are executed once and cached in sys.modules")
print(run("-c", "import shop.prices, shop.prices, shop.prices; import sys; print('  cached:', 'shop.prices' in sys.modules)"))

print("2. relative imports need the package context")
print("  python shop/cli.py ->", run("shop/cli.py").splitlines()[-1])
print("  python -m shop.cli ->", run("-m", "shop.cli").splitlines()[-1])

print("3. circular imports")
print("  import a (module-style `import b`):", run("-c", "import a; print('  ok:', a.fa())"))
print("  import d (from-style `from c import fc`):", run("-c", "import d").splitlines()[-1])

print("4. a local file named like a standard module shadows it")
write("shadow/random.py", "def random(): return 'not random'\n")
print("  ", run("-c", "import random; print(random.__file__.split('/')[-2:], random.random())", cwd=os.path.join(root, "shadow")))

print("5. sys.path decides what `import x` finds")
write("whereami.py", "import sys; print(repr(sys.path[0]))\n")
print("  python whereami.py  -> sys.path[0] is the script's folder:", run("whereami.py") == repr(root))
print("  python -c ...       -> sys.path[0] is:", run("-c", "import sys; print(repr(sys.path[0]))"), "(the current folder)")
print("  PYTHONPATH adds a folder:", run("-c", "import sys; print(any(p.endswith('extra') for p in sys.path))", env={**os.environ, "PYTHONPATH": os.path.join(root, "extra")}))

print("6. import time")
r = subprocess.run([sys.executable, "-X", "importtime", "-c", "import json"], capture_output=True, text=True)
rows = [l for l in r.stderr.splitlines() if l.startswith("import time:") and "|" in l][1:]
cum = sorted(((int(l.split("|")[1]), l.split("|")[2].strip()) for l in rows), reverse=True)[:3]
print("  slowest cumulative imports for `import json` (microseconds):", cum)
t = time.perf_counter(); subprocess.run([sys.executable, "-c", "pass"]); base = time.perf_counter() - t
t = time.perf_counter(); subprocess.run([sys.executable, "-c", "import numpy"]); np_ = time.perf_counter() - t
print(f"  interpreter start {base*1e3:.0f} ms | with `import numpy` {np_*1e3:.0f} ms")

print("7. bytecode cache")
run("-c", "import shop.prices")
print("  __pycache__ files:", sorted(os.listdir(os.path.join(root, "shop", "__pycache__"))))

venv_demo.sh: creates a venv and inspects its layout, sys.prefix and what it can see:

#!/bin/bash
# Run from an empty scratch folder. Offline-friendly: no packages are downloaded.
set -e
W=$(mktemp -d); cd "$W"
python3 -m venv .venv
echo "--- venv layout"; ls .venv; cat .venv/pyvenv.cfg | sed 's/^/  /'
echo "--- inside the venv"
.venv/bin/python -c "import sys; print('prefix       :', sys.prefix); print('base_prefix  :', sys.base_prefix); print('in a venv    :', sys.prefix != sys.base_prefix)"
echo "--- venv size"; du -sh .venv | cut -f1
echo "--- packages visible"; echo -n "system: "; python3 -m pip list 2>/dev/null | tail -n +3 | wc -l; echo -n "venv  : "; .venv/bin/python -m pip list 2>/dev/null | tail -n +3 | wc -l
echo "--- sys.path in venv"; .venv/bin/python -c "import sys; print('\n'.join('  '+p for p in sys.path if p))"
rm -rf "$W"

install_demo.sh: an editable install of a tiny local project with a console script:

#!/bin/bash
# Needs only the Python install itself and setuptools (no network): editable install of a tiny local project.
set -e
W=$(mktemp -d); cp -r project_demo "$W/p"; cd "$W/p"
python3 -m venv --system-site-packages .venv
.venv/bin/pip install --no-build-isolation --no-deps -q -e .
.venv/bin/shop-hello
.venv/bin/python -c "import importlib.metadata as m; print(m.version('shop-demo'))"
.venv/bin/pip show shop-demo | sed -n 1,2p

specifiers.py: version specifiers and version comparison with packaging:

from packaging.specifiers import SpecifierSet
from packaging.version import Version

available = ["1.9.0", "2.0.0", "2.1.0", "2.1.5", "2.2.0", "3.0.0"]
for spec in ("==2.1.*", "~=2.1", "~=2.1.0", ">=2.0,<3", "!=2.1.5,>=2.0", "==2.1.0"):
    ok = SpecifierSet(spec)
    print(f"{spec:15s} ->", [v for v in available if Version(v) in ok])
print("string comparison '10.0' < '9.0':", "10.0" < "9.0", "| Version comparison:", Version("10.0") < Version("9.0"))

Sources

  1. Python documentation, The import system, Modules and sys.path initialisation.
  2. Python documentation, venv: creation of virtual environments.
  3. Python Packaging User Guide, packaging.python.org: Writing your pyproject.toml, Installing packages, src layout vs flat layout, and Version specifiers.
  4. Python Enhancement Proposals, PEP 440: Version Identification and Dependency Specification, PEP 518: Specifying Minimum Build System Requirements, PEP 621: Storing project metadata in pyproject.toml, PEP 427: The Wheel Binary Package Format, and PEP 668: Marking Python base environments as externally managed.
  5. pip documentation, Secure installs and hash-checking mode and Repeatable installs.
  6. Documentation of uv, pip-tools and Poetry for lock files and project management; pip-audit for vulnerability scanning.

Practise this topic: Python

More guides