Python Packaging

Brief History 

Python builds were traditionally done using:

• distutils - Python built-in, uses requirements.txt to list dependencies, setup.py for configuration
• ISSUES: split of deps and config could cause conflicts, and load order confusion
  
  
• setuptools - extended distutils, added automatic install of dependencies

Other packaging tools:

• wheel - PEP 427 - replaced eggs - build tool with better support for C-library integrations
• creates .pyc files during installation, to match local interpreter
• supported in ip >= 1.4 and setuptools >= 0.8
• PyPI - Python Package Index - online registry of built py libs

among others.

Poetry

Adds main project descritor pyproject.toml to contain project metadata, config, and dependencies.

Some features:

• can declare different dependency sets, i.e. for prod, dev, and test
• creates file poetry.lock to define frozen dependencies

Useful Commands

> Create a New Project

poetry new my-package

> Installing existing Project - exclude some dependencies

poetry install --without test,docs

> Installing existing Project - only include some dependencies

poetry install --only dev

> Install or Fix existing Project - sync local to canonical

poetry install --sync

> Get Info about an Installed Lib

poetry show <package-name>

TOML Example

From https://towardsdatascience.com/ditch-requirements-use-poetry-00a936fe9b6d:

[tool.poetry]
name = "my_project"
version = "0.1.0"
description = "Description of my project"
package-mode = true
authors = [
"My Team <team@my_company.com>",
]

[tool.poetry.dependencies]
python = "~3.10"
fastapi = "^0.111.1"
pydantic = "^2.8.2"
Jinja2 = "^3.1.2"
uvicorn = {extras = ["standard"], version = "^0.30.3"}

[tool.poetry.group.dev.dependencies]
pytest = "^8.2.2"
black = "^24.3.0"
pre-commit = "^3.8.0"
mypy = "^1.11"
flake8 = "^7.1.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.flake8]
max-line-length = 99
extend-ignore = "E203"

Include From Source

Add the following to include a project in development as a dependency.

[[tool.poetry.source]]
name = "my-artifact-registry-repo"
url = "https://europe-west2-python.pkg.dev/my-gcpproject/my-af-repo/simple"
priority = "explicit"