Python

Virtual environments

Every project gets its own environment. Installing packages into the system interpreter breaks distribution-managed tooling, and on Debian and Ubuntu it is blocked outright (error: externally-managed-environment).

venv

venv ships with Python, so it is the default choice unless something else is already in use.

# Debian / Ubuntu only: the module is packaged separately
sudo apt install python3-venv python3-pip

python3 -m venv .venv

# Activate
source .venv/bin/activate        # Linux, macOS
.venv\Scripts\Activate.ps1       # Windows PowerShell
.venv\Scripts\activate.bat       # Windows cmd.exe

deactivate

Once activated, python and pip resolve to the ones inside .venv. Calling them as python -m pip is still worth the habit — it guarantees the packages go to the interpreter you think they do.

python -m pip install requests
python -m pip install "requests==2.32.3"     # pin an exact version
python -m pip install --upgrade requests
python -m pip uninstall requests
python -m pip show requests                  # version, location, dependencies
python -m pip list
python -m pip list --outdated

Recording dependencies

python -m pip freeze > requirements.txt
python -m pip install -r requirements.txt

pip freeze captures the entire resolved environment, transitive dependencies included, which is what you want for a reproducible deployment. For a library, declare only the direct dependencies in pyproject.toml instead.

pipenv

Combines the environment and the lock file. Pipfile records what was asked for, Pipfile.lock records exactly what was installed, with hashes.

python -m pip install --user pipenv

pipenv --python 3.12          # create the environment
pipenv install requests chardet
pipenv install --dev pytest   # development-only dependency
pipenv uninstall requests
pipenv shell                  # spawn a shell inside the environment
pipenv run pytest             # run one command inside it, no shell
pipenv --venv                 # print the environment path
pipenv --rm                   # delete the environment

Other environment managers

• uv — resolver and installer written in Rust; drop-in pip/venv replacement and far faster on large dependency trees.
• Poetry — dependency management plus packaging, driven by pyproject.toml.
• conda / mamba — manages non-Python dependencies (BLAS, CUDA, compilers) as well, which matters for the scientific stack.
• pyenv — installs and switches between interpreter versions; complements rather than replaces the above.

Installation on Windows

The official installer offers "Add python.exe to PATH" — tick it. To fix PATH afterwards, add both of these entries, in this order:

C:\Users\USER\AppData\Local\Programs\Python\PythonVERSION\Scripts
C:\Users\USER\AppData\Local\Programs\Python\PythonVERSION

VERSION has no separator: Python 3.12 installs into Python312. The Scripts directory holds pip.exe and any console entry points that packages install.

Windows also ships the py launcher, which works without PATH changes and can select a version explicitly:

py -0p            # list installed interpreters and their paths
py -3.12 -m venv .venv

Common commands

# Run a module as a script
python -m http.server 8000

# Unit tests: a single file, then discovery over a package
python -m unittest tests/test_1.py
python -m unittest discover -s tests -p "test_*.py"

# pytest equivalents
pytest tests/test_1.py
pytest -k "keyword" -x       # filter by name, stop at the first failure

# Interactive debugger at the point of failure
python -m pdb script.py

unittest discover needs the directory it scans to be importable. Add an __init__.py, or pass -t to name the top-level directory, otherwise tests that import project modules fail with ModuleNotFoundError.

Useful built-ins

chr(65)        # 'A'      integer -> character
ord('A')       # 65       character -> integer
hex(255)       # '0xff'   integer -> hexadecimal string
bin(5)         # '0b101'
int('ff', 16)  # 255      parse from any base
format(255, '02X')   # 'FF'   no prefix, fixed width, upper case

Joining and splitting:

>>> a = ['a', 'b', 'c', 'd']
>>> ''.join(a)
'abcd'
>>> ', '.join(a)
'a, b, c, d'
>>> 'a,b,c'.split(',')
['a', 'b', 'c']

str.join only accepts strings. Convert first when the list holds other types:

>>> ''.join(str(n) for n in [1, 2, 3])
'123'

Bytes and hex, which comes up constantly with firmware images and protocols:

>>> bytes([0xDE, 0xAD]).hex()
'dead'
>>> bytes.fromhex('dead')
b'\xde\xad'
>>> int.from_bytes(b'\x01\x00', 'little')
1
>>> (1024).to_bytes(2, 'little')
b'\x00\x04'

Notable packages

A curated shortlist rather than an exhaustive index — every entry below is actively maintained and installable on current Python versions.

Formatting and linting

Package	Purpose
black	Opinionated formatter. Almost nothing is configurable beyond line length and string quoting, so any two Black-formatted files agree. Stricter than PEP 8 requires.
ruff	Extremely fast linter and formatter that reimplements most Flake8, isort and pyupgrade rules in one tool.
flake8	Long-standing linter, plug-in ecosystem.
pylint	Deeper static analysis than Flake8: detects unused code, naming issues, refactoring opportunities. Slower, noisier by default.
mypy / pyright	Static type checkers for annotated code.
isort	Sorts and groups imports.

Text and data processing

Package	Purpose
chardet / charset-normalizer	Detect the character encoding of text, HTML or XML. charset-normalizer is the one requests depends on today.
colorama	ANSI colour output that also works on Windows terminals.
rich	Colour, tables, progress bars, tracebacks and markup for terminal output.
prettytable / tabulate	Render tabular data as ASCII tables.
difflib	Standard library; computes differences between sequences.
Levenshtein / rapidfuzz	Fast edit distance and fuzzy string matching. rapidfuzz supersedes the unmaintained fuzzywuzzy.
regex	Drop-in re replacement with fuller Unicode support.
ftfy	Repairs text that was decoded with the wrong encoding ("mojibake").
Unidecode	Transliterates Unicode to ASCII.
python-slugify	Converts arbitrary text into URL-safe slugs.
shortuuid	Compact, URL-friendly UUIDs.
phonenumbers	Google libphonenumber port: parse, format and validate phone numbers.
sqlparse	Non-validating SQL parser and formatter.
pygments	Syntax highlighting for many languages.
pyparsing / lark / ply	Build parsers for custom grammars.
pyfiglet	ASCII-art banner text.

Dates and times

Package	Purpose
zoneinfo	Standard library since 3.9; IANA time zone support. Prefer it over pytz in new code.
python-dateutil	Flexible date-string parsing (parser) and recurrence rules (rrule).
arrow / pendulum	Friendlier datetime replacements with time-zone-aware arithmetic.

Documents, spreadsheets and files

Package	Purpose
openpyxl	Read and write .xlsx. The usual fix for ModuleNotFoundError: No module named 'openpyxl' is simply installing it into the active environment.
xlsxwriter	Write-only .xlsx with strong charting and formatting support.
xlrd	Legacy .xls only; it dropped .xlsx support in 2.0.
xlwings	Drives a running Excel instance from Python.
python-docx	Read, create and modify Word .docx files.
pypdf	Split, merge and transform PDFs. Successor to PyPDF2.
pdfplumber / pdfminer.six	Extract text and tables from PDFs.
tablib	One dataset object exported as XLSX, CSV, JSON or YAML.
markdown / mistune / markdown-it-py	Markdown parsers and renderers.
pathlib	Standard library; object-oriented filesystem paths. Prefer it over os.path.
watchdog	Filesystem event monitoring.
python-magic	File-type detection from content via libmagic.
pickle	Standard library object serialisation. Never unpickle untrusted data — it executes arbitrary code. Use JSON or a schema format for anything crossing a trust boundary.

Configuration and logging

Package	Purpose
configparser	Standard library INI parser.
tomllib	Standard library TOML reader since 3.11; tomli backports it, tomli-w writes.
PyYAML / ruamel.yaml	YAML. Always parse with yaml.safe_load.
python-dotenv	Load .env files into the environment.
pydantic-settings	Typed, validated configuration from environment and files.
logging	Standard library; configure once at start-up, then use module-level loggers.
loguru	Zero-configuration logging with sensible defaults.
sentry-sdk	Error reporting and traces to a Sentry server.

Documentation

Package	Purpose
sphinx	Documentation generator; reStructuredText, with Markdown via myst-parser.
mkdocs + mkdocs-material	Markdown-first documentation site generator.
pdoc	Generates API documentation straight from docstrings.

Images and plotting

Package	Purpose
Pillow	The imaging library. Maintained fork of the long-dead PIL — import it as PIL.
opencv-python	OpenCV bindings: image processing, computer vision, classic ML algorithms.
scikit-image	Image processing built on NumPy.
matplotlib	The reference plotting library, MATLAB-like API.
seaborn	Statistical plots on top of matplotlib, with better defaults.
plotly / bokeh	Interactive, browser-rendered charts.
pygal	SVG charts.
qrcode	QR code generation (python-qrcode on PyPI).
python-barcode	1D barcodes without requiring Pillow.
graphviz	Python interface to Graphviz layout.
fonttools	Inspect and manipulate TTF/OTF fonts.

Audio, video and games

Package	Purpose
pydub	High-level audio manipulation; needs ffmpeg for compressed formats.
librosa	Audio analysis and feature extraction.
mutagen / tinytag	Read and write audio metadata tags.
moviepy	Scripted video editing, including GIF export.
yt-dlp	Media downloader. Actively maintained successor to youtube-dl.
pygame	2D game and multimedia framework built on SDL.
pyglet	Windowing and multimedia, no external dependencies.
PyOpenGL	OpenGL bindings.
Panda3D	Full 3D engine with Python bindings, originally from Disney.

Scientific computing and data

Package	Purpose
numpy	N-dimensional arrays (ndarray) and vectorised operations (ufunc). The foundation nearly everything else builds on.
scipy	Optimisation, linear algebra, integration, interpolation, signal and image processing, ODE solvers, statistics.
pandas	Labelled tabular data (DataFrame), time-series handling, I/O for most tabular formats.
polars	DataFrame library with a Rust core; larger-than-memory and multi-threaded workloads.
sympy	Symbolic mathematics.
statsmodels	Statistical models, hypothesis tests, econometrics.
networkx	Graph and network algorithms.
numba	JIT-compiles numeric Python to machine code via LLVM.
cython	Compiles annotated Python to C extensions.
dask	Parallel and out-of-core execution of NumPy/pandas-style workloads.
pyspark	Python API for Apache Spark.
astropy / biopython / rdkit	Domain toolkits for astronomy, biology and cheminformatics.

Machine learning and NLP

Package	Purpose
scikit-learn	Classification, regression, clustering, model selection and preprocessing with a consistent estimator API. (Imported as sklearn; the old scikits.learn name is long gone.)
pytorch / tensorflow	Deep-learning frameworks with GPU support.
xgboost / lightgbm	Gradient-boosted trees; typically the strongest baselines on tabular data.
transformers	Pretrained transformer models and pipelines.
spacy	Industrial-strength NLP: tokenisation, tagging, parsing, NER.
nltk	Teaching- and research-oriented NLP toolkit with large corpora.
gensim	Topic modelling and document similarity.
jieba	Chinese word segmentation.
pymc	Bayesian modelling and probabilistic programming.

Databases

Package	Purpose
sqlite3	Standard library; zero-configuration embedded SQL database.
SQLAlchemy	Core query builder plus ORM; the de facto standard.
psycopg	PostgreSQL driver. Version 3 supersedes psycopg2.
mysqlclient	MySQL C-extension driver; the maintained successor to MySQL-python.
PyMySQL	Pure-Python MySQL driver, no build step.
redis	Redis client.
pymongo	Official MongoDB driver.
peewee	Small, expressive ORM.
alembic	Schema migrations for SQLAlchemy.

Networking and HTTP

Package	Purpose
requests	The everyday HTTP client. Simpler than urllib and supports sessions, TLS verification and streaming.
httpx	requests-compatible API with HTTP/2 and async support.
aiohttp	Async HTTP client and server.
urllib3	Connection pooling and retries; requests is built on it.
pycurl	libcurl bindings — faster than urllib, and speaks FTP, SFTP, LDAP and more. Harder to install and to use.
httpie	Command-line HTTP client, friendlier than curl.
scrapy	Full crawling and scraping framework.
beautifulsoup4	Forgiving HTML/XML parser; copes with malformed markup.
lxml	Fast XML/HTML processing with XPath; the recommended parser backend for BeautifulSoup.
pyquery	jQuery-style selectors over HTML.
xmltodict	Treat XML as nested dictionaries.
html2text	Convert HTML to Markdown.
paramiko	SSHv2 client and server implementation.
scapy	Packet crafting, sniffing and protocol analysis.
websockets	Async WebSocket client and server.
pyzmq	ZeroMQ bindings.
pika	RabbitMQ (AMQP 0-9-1) client.
boto3	AWS SDK.
paho-mqtt	MQTT client, widely used with embedded devices.

Web frameworks

Package	Purpose
django	Batteries-included MVT framework: ORM, admin, auth, migrations. Best fit for content-heavy applications.
flask	Minimal WSGI framework; you assemble the rest.
fastapi	Async framework with automatic validation and OpenAPI docs from type hints.
starlette	ASGI toolkit underneath FastAPI.
djangorestframework	REST APIs on top of Django.
jinja2	The standard template engine, used by Flask and many others.
pydantic	Data validation and settings from type annotations.
celery	Distributed task queue backed by Redis or RabbitMQ.
gunicorn / uvicorn	Production WSGI and ASGI servers.
elasticsearch	Official Elasticsearch client.
wagtail / django-cms	Django-based content management systems.
pelican	Static site generator from Markdown or reStructuredText.

Security and cryptography

Package	Purpose
cryptography	The recommended modern toolkit: hashes, symmetric and asymmetric ciphers, X.509 handling. Replaces the unmaintained PyCrypto.
hashlib	Standard library; MD5, SHA-1, SHA-2, SHA-3, BLAKE2 with one API, backed by OpenSSL. MD5 and SHA-1 are broken for signatures — use SHA-256 or better.
secrets	Standard library; cryptographically secure random values for tokens and keys. random is not suitable for this.
passlib / argon2-cffi / bcrypt	Password hashing with correct salting and work factors.
PyJWT	Encode and decode JSON Web Tokens.
authlib	OAuth 1/2 and OpenID Connect clients and servers.
ecdsa	Pure-Python ECDSA; convenient for verifying signatures offline, as in the ESP32 signing example.
pyOpenSSL	OpenSSL bindings for protocol-level work.

System and command line

Package	Purpose
argparse	Standard library argument parser.
click / typer	Declarative CLIs; typer derives them from type hints.
subprocess	Standard library; the supported way to run external commands.
psutil	Cross-platform process, CPU, memory, disk and network statistics.
threading / asyncio / multiprocessing	Standard library concurrency. asyncio for I/O-bound work, multiprocessing to escape the GIL for CPU-bound work.
gevent	Coroutine-based networking with greenlets.
schedule / APScheduler	In-process job scheduling.
fabric	Run commands over SSH on remote hosts.
ansible	Agentless configuration management and deployment.
cookiecutter	Generate projects from templates.
ctypes / cffi	Call C libraries from Python. cffi reads C declarations directly and needs no wrapper module; ctypes needs no build step at all.
pybind11	Write C++ extension modules with a modern API.
keyboard / pyautogui	Keyboard, mouse and screen automation.

Testing and debugging

Package	Purpose
unittest	Standard library xUnit framework.
pytest	The usual choice: plain assert, fixtures, parametrisation, large plug-in ecosystem.
unittest.mock	Standard library test doubles (also on PyPI as mock).
pytest-cov / coverage	Line and branch coverage measurement.
hypothesis	Property-based testing; generates inputs that break your assumptions.
responses / respx	Stub HTTP calls made with requests / httpx.
freezegun	Freeze or move datetime.now() in tests.
faker	Generate realistic fake data.
factory-boy	Build test objects for Django, SQLAlchemy and others.
selenium / playwright	Drive real browsers for end-to-end tests.
locust	Load testing with scenarios written in Python.
pdb / ipdb / pudb	Interactive debuggers: standard, IPython-flavoured, full-screen.
memory_profiler / scalene	Memory and combined CPU/memory profiling.
py-spy	Sampling profiler that attaches to a running process without modifying it.
django-debug-toolbar	Per-request SQL, cache and timing panels for Django.

Packaging and distribution

Package	Purpose
pip	The package installer.
build + twine	Build wheels and sdists, then upload them to PyPI.
hatchling / setuptools / flit	Build backends configured through pyproject.toml.
wheel	The binary distribution format; replaced eggs.
pyinstaller	Bundle an application and its interpreter into a standalone executable, on all major platforms.
cx_Freeze	Same idea, different trade-offs.
py2app	macOS application bundles.
pynsist	Windows installers that ship a normal Python installation.
devpi	Private PyPI server, mirroring and staging.

Industrial and hardware

Package	Purpose
pyserial	Serial port access on Windows, macOS, Linux and BSD. Import it as serial.
pymodbus	Modbus RTU and TCP client and server.
python-can	CAN bus interface layer across many adapters.
esptool	Espressif flashing tool for ESP8266 and ESP32.
RPi.GPIO / gpiozero	Raspberry Pi GPIO access.
pyvisa	Control lab instruments over GPIB, USB, serial and Ethernet.

Troubleshooting

VS Code does not see the pipenv environment

The interpreter selected in VS Code is not the pipenv one. Get the path, then select it:

pipenv --venv

Press Ctrl+Shift+P, run Python: Select Interpreter, and either pick the entry matching that path or choose Enter interpreter path… and point at <venv>/bin/python (<venv>\Scripts\python.exe on Windows).

Pin it per project so the choice survives a reload — .vscode/settings.json:

{
  "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python"
}

Setting PIPENV_VENV_IN_PROJECT=1 before pipenv install places the environment in .venv inside the project, which VS Code detects on its own.

Reload the window after switching interpreters — the language server keeps the old environment cached until it restarts.

ModuleNotFoundError for a package you just installed

Almost always a wrong-environment problem. Confirm which interpreter is running and where its packages come from:

python -c "import sys; print(sys.executable)"
python -m pip -V        # prints the pip path and the interpreter it serves

If those two disagree, the environment is not active. Reactivate it and install with python -m pip install PACKAGE.

error: externally-managed-environment

Debian, Ubuntu and Homebrew mark the system interpreter as managed by the OS package manager. Create a virtual environment instead. --break-system-packages exists but does what its name says.