CS Fundamentals Trainer · Guides
Python packaging, environments and imports
How an import finds and runs a module, sys.path and shadowing, relative and circular imports, what a virtual environment really is, pyproject.toml and editable installs, version specifiers, lock files and dependency security.
18 min read
"It works on my machine" is the most expensive sentence in software, and in Python it usually has one of three causes: a different interpreter, different installed package versions, or a different idea of where an import comes from. This guide explains the mechanisms behind all three: how Python finds and runs modules, why virtual environments exist and what they actually are, what a package is and how pyproject.toml describes one, how version specifiers and lock files make installs repeatable, and what the supply-chain risks are. Each part was run on CPython 3.13 (scripts in the appendix). It builds on how Python works and connects to the dependency advice in web application security.
One limit: the sandbox where I ran this had no access to the Python Package Index (PyPI), so nothing here was downloaded. I used only the interpreter, venv, pip and setuptools that were already installed, and everything about resolvers, lock files and tools such as uv, poetry or pip-tools is from their documentation, not from a run.
1. How an import works
import shop.prices does four things, in order:
- Look in
sys.modules, the dictionary of modules already loaded. If the name is there, return that module and stop. - Otherwise find the module: walk the folders in
sys.pathand look forshop/(a package) orshop.py. - Run the module's code from top to bottom, in a new namespace. The names it defines become the module's attributes.
- Store the module in
sys.modulesand bind the name in the importing code.
I tested step one. Importing shop.prices three times printed shop.prices runs once, and 'shop.prices' in sys.modules was True. A module's top-level code runs once per process, so keep side effects (opening connections, reading files, starting threads) out of import time; do them in functions called from main. The same module imported by two different names (for example by a relative and an absolute path) can be loaded twice as two objects, which is a classic source of confusing bugs.
Step three also explains the bytecode cache: after importing shop.prices, the folder shop/__pycache__/ held __init__.cpython-313.pyc and prices.cpython-313.pyc. Compiled bytecode is saved there so the next start can skip compiling; delete the folder freely, it is regenerated.
sys.path: where Python looks
sys.path is an ordinary list of folders. In order, it contains the folder of the script being run (or '' for the current folder with -c and the interactive prompt), the folders in the PYTHONPATH environment variable, the standard library, and the site-packages folder where installed packages live. I confirmed each: python whereami.py put the script's own folder first, python -c put '' first, and a folder named in PYTHONPATH appeared in the list.
The first match wins, which leads to the most baffling bug in this area. I created a file called random.py next to my script that defined a random() function returning 'not random'. import random loaded my file instead of the standard library, because the script's folder comes first. Never name your modules after standard-library modules (random.py, json.py, email.py, test.py, types.py), and if an import returns something strange, print module.__file__ to see where it came from.
Packages and relative imports
A folder with an __init__.py is a package (a folder without one can still work as a "namespace package", but add the file for regular code). Inside a package you can import siblings with a leading dot: from .prices import gross.
Relative imports only work when the module is imported as part of its package. I ran the same file two ways:
| Command | Result |
|---|---|
python shop/cli.py | ImportError: attempted relative import with no known parent package |
python -m shop.cli | gross(100) = 120.0 |
Running a file as a script makes it the top-level __main__ with no package around it; python -m package.module runs it as a member of the package. Use -m for anything inside a package, or better, install the project (section 3) and expose a console script. This is also what the if __name__ == "__main__": guard is for: __name__ is "__main__" only when the file is run directly, so the code under the guard does not run when the module is imported.
Circular imports
If a imports b and b imports a, one of them sees the other half-initialised, because the module is stored in sys.modules before its code finishes. Two variations that I ran:
| Imports | Result |
|---|---|
import b inside a.py, import a inside b.py, using a.fa() only inside a function | works (a is fully loaded by the time the function is called) |
from c import fc inside d.py, and from d import fd inside c.py | ImportError: cannot import name 'fd' from 'd' (the error also suggests renaming the file, which is not the cause here) |
import module style tolerates cycles because the attribute is looked up later; from module import name needs the name to exist at that moment. Fixes, best first: restructure so the shared part lives in a third module that both import; move the import inside the function that needs it; or import the module instead of the name. A cycle is usually a sign that two modules are really one, or that a dependency points the wrong way.
Import cost
Imports are not free. A bare interpreter start took about 14 ms on my machine and a start that imported numpy took about 92 ms; python -X importtime -c "import json" printed a per-module breakdown, with json costing roughly 9 ms cumulatively (mostly json.decoder and re) on a cold start. For command-line tools and serverless functions, startup time is user-visible. Import heavy libraries lazily inside the function that uses them, and use -X importtime to find what is slow.
2. Virtual environments
The problem
All the Python programs on a machine share one site-packages unless told otherwise. Project A needs library==1.4, project B needs library==2.0, and the operating system's own tools need yet another. Installing globally also risks breaking the system (many Linux distributions now refuse pip install into the system Python for this reason). A virtual environment gives each project its own site-packages.
What a venv is
It is not a virtual machine or a copy of Python. I created one with python3 -m venv .venv and looked inside:
- a small folder (about 13 MB, almost all of it
pip) withbin/(thepythonlauncher),lib/python3.13/site-packages, and a filepyvenv.cfg; pyvenv.cfgrecordshome = /usr/bin(the real interpreter),version = 3.13.17, andinclude-system-site-packages = false;- inside it,
sys.prefixpointed at the venv folder whilesys.base_prefixstill pointed at/usr, which is how Python knows it is in a venv (sys.prefix != sys.base_prefixwasTrue); sys.pathcontained the standard library and the venv's ownsite-packages, and not the system's. The system Python could see 156 installed packages; the new venv saw 1 (pip).
So a venv is a folder with a launcher that points at a real interpreter and a private place to install packages. That has consequences: you do not move or copy a venv to another machine (paths are absolute); you recreate it from a requirements file. You do not need to "activate" it either: running .venv/bin/python directly uses it, and activate mostly edits PATH for convenience. Do not commit .venv/ to version control; do commit the files that describe it (section 4). Use one venv per project, and keep it out of Docker images' build context.
Interpreters
A venv is tied to the Python minor version that created it (3.13 here). Declare the supported versions in pyproject.toml (requires-python), test on each, and pin the one you deploy. Tools such as pyenv or uv python install and select interpreter versions; in containers, the base image does.
3. Packages and pyproject.toml
A distribution package is what you install (pip install shop-demo); it contains one or more import packages (import shop). The metadata describing it lives in pyproject.toml, the standard configuration file (PEP 518 and PEP 621):
[build-system]
requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta"
[project]
name = "shop-demo"
version = "0.1.0"
requires-python = ">=3.9"
dependencies = []
[project.scripts]
shop-hello = "shop.cli:main"
[build-system]says how to build the project (setuptoolshere; others includehatchling,flitandpoetry-core). Because of this table,pipcan build any project without being tied to one tool.[project]holds the name, version, supported Python versions and dependencies (the libraries it needs when running).[project.scripts]declares console scripts: after installationshop-hellois a command that callsshop.cli:main.- A
src/layout (code insrc/shop/rather than next to the project root) keeps tests from accidentally importing the uninstalled copy.
I installed that tiny project in editable mode into a fresh venv (pip install -e ., with --no-build-isolation because the sandbox could not download the build tools) and it worked: the shop-hello command printed hello from an installed console script, and importlib.metadata.version('shop-demo') returned 0.1.0. An editable install links the venv to your source folder, so edits take effect without reinstalling; this is the normal way to work on a project. A plain pip install . failed in the sandbox only because the default "build isolation" downloads the build backend from PyPI, which was not reachable; on a normal machine it works.
Distributions are published in two forms: a wheel (.whl, a ready-to-unpack archive, fast to install) and an sdist (source, built on install). Prefer wheels. Libraries and applications differ: a library should declare loose dependency ranges so it can coexist with others; an application should install exact pinned versions so its environment is reproducible.
4. Versions, requirements and lock files
Version specifiers
Dependencies carry a version constraint written in the PEP 440 syntax. I evaluated six specifiers against the available versions 1.9.0, 2.0.0, 2.1.0, 2.1.5, 2.2.0, 3.0.0 with the packaging library, the one pip itself uses:
| Specifier | Matches |
|---|---|
==2.1.0 | 2.1.0 |
==2.1.* | 2.1.0, 2.1.5 |
~=2.1 ("compatible release") | 2.1.0, 2.1.5, 2.2.0 |
~=2.1.0 | 2.1.0, 2.1.5 |
>=2.0,<3 | 2.0.0, 2.1.0, 2.1.5, 2.2.0 |
!=2.1.5,>=2.0 | 2.0.0, 2.1.0, 2.2.0, 3.0.0 |
Two details bite people. ~=2.1 allows any 2.x at or above 2.1, while ~=2.1.0 allows only 2.1.x: the number of components decides. And versions are not strings: "10.0" < "9.0" is True, but Version("10.0") < Version("9.0") is False. Never compare version numbers as text (and do not parse them with ad-hoc split); use packaging.version.
Reproducible installs
Constraints say which versions are acceptable; a lock file records the exact versions (and hashes) that were chosen, including indirect dependencies. Without one, two installs a week apart can get different versions and behave differently, and a broken release of a deep dependency breaks your build overnight. The ways to get one:
| Approach | What it gives |
|---|---|
pip freeze > requirements.txt | the exact versions installed right now (I listed a venv and got lines such as anyio==4.15.1); simple but mixes direct and indirect dependencies and has no hashes |
pip-tools (pip-compile) | a pinned requirements.txt generated from a short input file, with optional hashes |
uv, poetry, pdm | a project tool with its own lock file (uv.lock, poetry.lock, pdm.lock) and an environment manager |
| A constraints file or hash-checking mode | pip install --require-hashes -r requirements.txt refuses any file whose hash differs |
(From the tools' documentation; I could not run them offline.) Whichever you use, commit the lock file, install from it in CI and production, update it deliberately (a pull request that bumps versions, with the tests run), and keep development tools (test runner, linters) in a separate group from runtime dependencies. In a container, install dependencies in an earlier layer than your code, so editing a source file does not reinstall everything (see containers and Kubernetes).
5. Security of dependencies
An install runs other people's code, sometimes at install time. The risks and the controls:
- Typosquatting and dependency confusion: a malicious package with a name close to a real one, or a public package that shadows the name of a private one. Check names, and configure private indexes so that internal names cannot be taken from the public one.
- Compromised or abandoned packages: pin versions and hashes, review what you add (age, maintainers, activity, how many things it pulls in), and prefer fewer dependencies.
- Known vulnerabilities: scan with
pip-auditor your platform's scanner in CI, and decide who triages the findings; an unread report protects nothing. - Install-time code: building an sdist can run arbitrary code, which is one reason to prefer wheels from a trusted index.
- Secrets: do not put index tokens in
requirements.txtor the repository; use environment variables or the CI's secret store. See web application security for the supply-chain checklist.
6. Common mistakes
- Installing into the system Python (or with
sudo pip), instead of a venv. - Naming a module like a standard-library module (
random.py) and being shadowed by it. - Running package files as scripts (
python shop/cli.py) and then fighting relative imports; usepython -m. - Side effects at import time (connecting to a database, reading config), which run once per process, in an unpredictable order.
- Circular imports patched with ad-hoc imports instead of fixing the structure.
- Committing
.venv/, or copying a venv between machines. - No lock file, or a
requirements.txtthat listspackagewith no version. - Comparing version strings as text.
- Mixing build tools or two environment managers in one project without a clear rule.
- Unpinned base images and unpinned Python versions in deployment.
- Adding a dependency for something the standard library already does.
7. Questions interviewers like to ask
"What is a virtual environment and why use one?"
A directory with its own site-packages and a launcher pointing at a real interpreter, so each project has isolated dependencies. sys.prefix differs from sys.base_prefix inside it. It is not a VM and should be recreated from a lock file, not copied.
"What happens when you import a module?"
Check sys.modules; otherwise find it along sys.path, execute its code once in a new namespace, store it in sys.modules, and bind the name. That is why module-level code runs once and why a local file can shadow a standard module.
"Why do circular imports fail?"
A module is in sys.modules before its code finishes, so the other module can see a half-built one. from x import name fails if the name is not defined yet; import x often works. Fix by restructuring.
"What does if __name__ == '__main__': do?"
It runs the code only when the file is executed directly, not when it is imported.
"~=2.1 versus ==2.1.*?"
~=2.1 means >=2.1, ==2.* (so 2.2 is allowed); ==2.1.* means exactly the 2.1 series.
"How do you make a Python deployment reproducible?"
Pin the interpreter version, install from a lock file with hashes into a fresh environment or image layer, build once and promote the same artifact, and update dependencies through reviewed changes.
"What is the difference between a wheel and an sdist?"
A wheel is a prebuilt archive that is simply unpacked on install; an sdist is source that must be built, which is slower and can run arbitrary build code.
8. Things to mention in an interview
- The import algorithm:
sys.modules,sys.path, execute once; shadowing;__pycache__. - Packages, relative imports and
python -m; the__main__guard; circular imports and how to break them. - A venv as a folder plus launcher;
sys.prefixagainstsys.base_prefix; recreate, never copy. pyproject.toml: build system, dependencies, scripts; editable installs; wheels and sdists; libraries (loose ranges) against applications (pinned).- PEP 440 specifiers and why versions are not strings; lock files and hashes; deliberate upgrades.
- Supply-chain risks and the controls: pinning, hashes, scanning, private index configuration.
Quick recap
- An import checks
sys.modules, searchessys.path, runs the module once, and caches it; module-level code ran once even after three imports. - A local
random.pybeat the standard library because the script's folder comes first insys.path. python shop/cli.pyfailed with "no known parent package";python -m shop.cliworked.from x import namecycles fail;import xcycles often work; restructure to remove the cycle.- A venv is about 13 MB: a launcher, a private
site-packagesand apyvenv.cfg. The system Python saw 156 packages, the new venv 1. Recreate it; do not copy it. pyproject.tomldescribes build, metadata, dependencies and console scripts; an editable install letshop-hellorun from the source tree.~=2.1matched 2.1.0, 2.1.5 and 2.2.0;~=2.1.0only 2.1.x;"10.0" < "9.0"is true as text.- Commit a lock file, install from it with hashes, scan and review dependencies, and keep secrets out of requirement files.
Appendix: the experiment code
Python 3.13 on Linux, using the interpreter, venv, pip and setuptools that came with it and the packaging library (installed with pip); nothing was downloaded. Paths and the sizes you see will differ. venv_demo.sh and install_demo.sh are bash scripts; run install_demo.sh from a folder that contains project_demo/ (the pyproject.toml above plus src/shop/__init__.py and src/shop/cli.py with def main(): print('hello from an installed console script')).
imports_demo.py: builds a throwaway project and runs the import experiments (caching, relative imports, cycles, shadowing, sys.path, import time, __pycache__):
"""Builds a throwaway project in a temp folder and runs small import experiments against it."""
import os, subprocess, sys, tempfile, textwrap, time
root = tempfile.mkdtemp()
def write(path, text):
full = os.path.join(root, path); os.makedirs(os.path.dirname(full), exist_ok=True)
open(full, "w").write(textwrap.dedent(text))
def run(*args, cwd=root, env=None):
r = subprocess.run([sys.executable, *args], cwd=cwd, capture_output=True, text=True, env=env)
return (r.stdout + r.stderr).strip()
write("shop/__init__.py", "print(' shop/__init__ runs')\n")
write("shop/prices.py", "print(' shop.prices runs')\nTAX = 0.2\ndef gross(p): return p * (1 + TAX)\n")
write("shop/cli.py", """
from .prices import gross
def main(): print('gross(100) =', gross(100))
if __name__ == '__main__': main()
""")
write("a.py", "import b\ndef fa(): return 'a'\n")
write("b.py", "import a\ndef fb(): return a.fa() + 'b'\n")
write("c.py", "from d import fd\ndef fc(): return 'c'\n")
write("d.py", "from c import fc\ndef fd(): return fc()\n")
print("1. modules are executed once and cached in sys.modules")
print(run("-c", "import shop.prices, shop.prices, shop.prices; import sys; print(' cached:', 'shop.prices' in sys.modules)"))
print("2. relative imports need the package context")
print(" python shop/cli.py ->", run("shop/cli.py").splitlines()[-1])
print(" python -m shop.cli ->", run("-m", "shop.cli").splitlines()[-1])
print("3. circular imports")
print(" import a (module-style `import b`):", run("-c", "import a; print(' ok:', a.fa())"))
print(" import d (from-style `from c import fc`):", run("-c", "import d").splitlines()[-1])
print("4. a local file named like a standard module shadows it")
write("shadow/random.py", "def random(): return 'not random'\n")
print(" ", run("-c", "import random; print(random.__file__.split('/')[-2:], random.random())", cwd=os.path.join(root, "shadow")))
print("5. sys.path decides what `import x` finds")
write("whereami.py", "import sys; print(repr(sys.path[0]))\n")
print(" python whereami.py -> sys.path[0] is the script's folder:", run("whereami.py") == repr(root))
print(" python -c ... -> sys.path[0] is:", run("-c", "import sys; print(repr(sys.path[0]))"), "(the current folder)")
print(" PYTHONPATH adds a folder:", run("-c", "import sys; print(any(p.endswith('extra') for p in sys.path))", env={**os.environ, "PYTHONPATH": os.path.join(root, "extra")}))
print("6. import time")
r = subprocess.run([sys.executable, "-X", "importtime", "-c", "import json"], capture_output=True, text=True)
rows = [l for l in r.stderr.splitlines() if l.startswith("import time:") and "|" in l][1:]
cum = sorted(((int(l.split("|")[1]), l.split("|")[2].strip()) for l in rows), reverse=True)[:3]
print(" slowest cumulative imports for `import json` (microseconds):", cum)
t = time.perf_counter(); subprocess.run([sys.executable, "-c", "pass"]); base = time.perf_counter() - t
t = time.perf_counter(); subprocess.run([sys.executable, "-c", "import numpy"]); np_ = time.perf_counter() - t
print(f" interpreter start {base*1e3:.0f} ms | with `import numpy` {np_*1e3:.0f} ms")
print("7. bytecode cache")
run("-c", "import shop.prices")
print(" __pycache__ files:", sorted(os.listdir(os.path.join(root, "shop", "__pycache__"))))
venv_demo.sh: creates a venv and inspects its layout, sys.prefix and what it can see:
#!/bin/bash
# Run from an empty scratch folder. Offline-friendly: no packages are downloaded.
set -e
W=$(mktemp -d); cd "$W"
python3 -m venv .venv
echo "--- venv layout"; ls .venv; cat .venv/pyvenv.cfg | sed 's/^/ /'
echo "--- inside the venv"
.venv/bin/python -c "import sys; print('prefix :', sys.prefix); print('base_prefix :', sys.base_prefix); print('in a venv :', sys.prefix != sys.base_prefix)"
echo "--- venv size"; du -sh .venv | cut -f1
echo "--- packages visible"; echo -n "system: "; python3 -m pip list 2>/dev/null | tail -n +3 | wc -l; echo -n "venv : "; .venv/bin/python -m pip list 2>/dev/null | tail -n +3 | wc -l
echo "--- sys.path in venv"; .venv/bin/python -c "import sys; print('\n'.join(' '+p for p in sys.path if p))"
rm -rf "$W"
install_demo.sh: an editable install of a tiny local project with a console script:
#!/bin/bash
# Needs only the Python install itself and setuptools (no network): editable install of a tiny local project.
set -e
W=$(mktemp -d); cp -r project_demo "$W/p"; cd "$W/p"
python3 -m venv --system-site-packages .venv
.venv/bin/pip install --no-build-isolation --no-deps -q -e .
.venv/bin/shop-hello
.venv/bin/python -c "import importlib.metadata as m; print(m.version('shop-demo'))"
.venv/bin/pip show shop-demo | sed -n 1,2p
specifiers.py: version specifiers and version comparison with packaging:
from packaging.specifiers import SpecifierSet
from packaging.version import Version
available = ["1.9.0", "2.0.0", "2.1.0", "2.1.5", "2.2.0", "3.0.0"]
for spec in ("==2.1.*", "~=2.1", "~=2.1.0", ">=2.0,<3", "!=2.1.5,>=2.0", "==2.1.0"):
ok = SpecifierSet(spec)
print(f"{spec:15s} ->", [v for v in available if Version(v) in ok])
print("string comparison '10.0' < '9.0':", "10.0" < "9.0", "| Version comparison:", Version("10.0") < Version("9.0"))
Sources
- Python documentation, The import system, Modules and
sys.pathinitialisation. - Python documentation,
venv: creation of virtual environments. - Python Packaging User Guide, packaging.python.org: Writing your
pyproject.toml, Installing packages,srclayout vs flat layout, and Version specifiers. - Python Enhancement Proposals, PEP 440: Version Identification and Dependency Specification, PEP 518: Specifying Minimum Build System Requirements, PEP 621: Storing project metadata in
pyproject.toml, PEP 427: The Wheel Binary Package Format, and PEP 668: Marking Python base environments as externally managed. - pip documentation, Secure installs and hash-checking mode and Repeatable installs.
- Documentation of uv, pip-tools and Poetry for lock files and project management; pip-audit for vulnerability scanning.
Practise this topic: Python