Best Practices for Python Project

A good python project should prove resilient under tesam collaboration and in production environments, excelling in four key areas:

  • maintainability
  • extensibility
  • observability
  • deliverability

1. Project Layout

Testing & Packaging

Best Practices for Python Project Orginazation

myproject/
├── pyproject.toml
├── README.md
├── .gitignore
├── .env.example # Environment template
├── config/ # YAML/TOML/JSON default configuration
│ ├── default.toml
│ └── logging.yml
├── src/
│ ├── common/ # public package
│ | ├── init.py
│ | ├── utils.py
│ | ├── retry.py
│ | ├── validators.py
│ | ├── converters.py
│ | ├── time.py
│ ├── module1/ # API model
│ | ├── init.py
│ | ├── core.py
│ | ├── api/
│ | │ ├──init.py
│ | │ └── router.py
│ | └── utils/
│ ├── module2/ # data pipline module
│ | ├── init.py
│ | ├── pipline.py
│ └── core/ #
│ ├── init.py
| ├── types.py
│ ├── logging.py
│ ├── exceptions.py
│ ├── security.py
│ └── settings.py # universal configuration center
├── tests/ #pytest scripts
│ ├── conftest.py
│ └── test_core.py
└── scripts/ # Helper script (unpackaged)

  1. Use some naming convention to seperate public and private variables, like using _ as prefix of private variables.
  2. Design public api via well config the init.py
  3. Using pyproject.toml + uv etc. package management tool
  4. Using pydantic.BaseSettings to read and manage configurations in confg/
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    # src/mypkg/settings.py
    from pathlib import Path
    from pydantic import BaseSettings

    class Settings(BaseSettings):
        debug: bool = False
        db_url: str
        log_cfg: Path = Path(__file__).with_name("logging.yml")

        class Config:
            env_file = Path(__file__).parent.parent.parent / ".env"

    settings = Settings()

  5. Using python_dotenv to load secret from .env (it should be git ignored.)
  6. Run project
    linux:
    1
    2
    3
    export PYTHONPATH=./src
    uvicorn mypkg1.main:app --reload
    python -m mypkg2.main
    windows:
    1
    2
    3
    $env:PYTHONPATH = ".\src"
    uvicorn mypkg1.main:app --reload
    python -m mypkg2.main

2. Package and Building

A typical pyproject.toml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[build-system]            # PEP 518 —— tell pip how to build
requires = ["hatchling>=1.24", "hatch-vcs"]
build-backend = "hatchling.build"

[project]                 # PEP 621 —— meta data of package
name = "demo"
dynamic = ["version"]     # version was managed by VCS plugin, read from git tag
dependencies = ["requests>=2.32"]
optional-dependencies.test = ["pytest"]
classifiers = ["Private :: Do Not Upload"]  #pypi will reject publishing

[tool.black]              # user setting, start with tool
line-length = 88

[tool.hatch.version]       # hatch-vcs setting
source = "vcs"
tag-pattern = "v{version}"   # read Git tag v1.2.3

[tool.hatch.build.targets.wheel]
packages = ["src/pkg_a", "src/pkg_b", "src/acme"]

  1. VCS driven versioning (uv+hatching.build)
  • package version read from “git tab v1.2.3 && git push –tags”
  • “uv build” will write version=”1.2.3” into the package level “init.py” automatically.
  • dev branch will have special suffix, implemented by hatch-vcs.
  1. uv build / uv publish to build whl locally or publish to pypi.org( or self host repo)

3. Type Hints and Documentation

1
2
3
4
5
6
7
8
9
10
11
12
def add(x: int, y: int) -> int:
    """
    Add two integers.

    Args:
        x (int): First number.
        y (int): Second number.

    Returns:
        int: The sum of x and y.
    """
    return x + y
  1. Using type hints for all function/class/module/variable, following PEP 484 including:
    • function: parameters and return value
    • class: all fields
    • variables
    • generic container: generic type
    • 3-rd part module’s function/return
  2. Using mypy check type hints before integration.
    mypy could work at
    • github action prebuilding
    • local terminal
    • local prebuilding
  3. Using standard docstring format (google-style)
  4. Using Sphinx/MkDocs generate documentation webpage through type hints and docstring automatically.

4. Testing

Construct test pyramid with pytest, based on unit test, supplimented with integration test and contract test.
Force test coverage > 90% with pytest-cov in CI.
Adopting hypothesis(property-based testing) to includ massive random test case input.

1. Testing Pyramid

1
2
3
▲ UI Test (end to end) (emulate real business process)
▲ Integration Test (components interaction) (verify the cooperation between mupltiple components)
▲ Unit Test (function, class) (low cost, responsive)

2. Unit Test + Integration Test + Contract Test (Example-Based Test)

pytest

  1. Unit Test

    1
    2
    3
    4
    5
    def add(x: int, y: int) -> int:
        return x + y

    def test_add():
        assert add(2, 3) == 5

  2. Integration Test

    1
    2
    3
    def test_user_login_and_access_db():
        token = login("alice", "pass")
        assert db.get_user_by_token(token).name == "alice"

  3. Contract Test

    1
    2
    3
    4
    def test_api_contract(requests_mock):
        requests_mock.get("https://api.example.com/user", json={"name": "Elvin"})
        resp = fetch_user("https://api.example.com/user")
        assert resp["name"] == "Elvin"

3. Force 90% Test Coverage

pytest-cov

1
pytest --cov=src/ --cov-report=term
1
2
- name: Run tests with coverage
  run: pytest --cov=src --cov-fail-under=90

4. Auto Test (Property-Based Testing)

hypothesis

1
2
3
4
5
6
from hypothesis import given
from hypothesis.strategies import integers

@given(integers(), integers())
def test_add_commutative(x, y):
    assert add(x, y) == add(y, x)

5. CI/CD

github action

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
name: CI/CD Pipeline

on:
  push:   # trigger by push
    branches: [ main ]  
  pull_request:         # trigger by pull_request creating or updating
    branches: [ main ]
  workflow_dispatch:  # manually trigger

jobs:
  build-test-publish:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install uv & dependencies
        run: |
          curl -LsSf https://astral.sh/uv/install.sh | sh
          uv pip install -r requirements.txt  # compatible fallback
          uv pip install -r requirements-dev.txt  # dev dependencies

      - name: Lint (black)
        run: uv pip install black && black . --check

      - name: Type check (mypy)
        run: |
          uv pip install mypy
          mypy src/

      - name: Run tests with coverage
        run: |
          uv pip install pytest pytest-cov
          pytest --cov=src --cov-report=xml --cov-fail-under=90

      - name: Build package (PEP 517)
        run: uv build

      - name: Publish (optional)
        if: startsWith(github.ref, 'refs/tags/v')
        run: |
          uv pip install uv
          uv publish --yes  #  PyPI Token need to be configed ahead

    env:
      UV_SYSTEM_PYTHON: "true"
      UV_CACHE_DIR: "${{ github.workspace }}/.uv-cache"
      # this is for uv publish
      UV_PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}

6. Logging, Monitoring and Tracing

  1. Structured Logging
    logging + pythonjsonlogger
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import logging
import sys
from pythonjsonlogger import jsonlogger

#base: LOG_LEVEL=DEBUG python app.py
log_level = os.getenv("LOG_LEVEL", "INFO").upper()
logging.basicConfig(level=log_level)

logger = logging.getLogger()
#logger.setLevel(logging.INFO)

logHandler = logging.StreamHandler(sys.stdout)
formatter = jsonlogger.JsonFormatter('%(asctime)s %(levelname)s %(message)s %(request_id)s')
logHandler.setFormatter(formatter)
logger.addHandler(logHandler)

logger.info("User logged in", extra={"request_id": "abc123"})
1
2
3
4
5
6
{
  "asctime": "2025-06-22 12:34:56",
  "levelname": "INFO",
  "message": "User logged in",
  "request_id": "abc123"
}
  1. Metrics Monitoring

Prometheus 

1
2
3
4
5
6
7
8
9
from prometheus_client import Counter, start_http_server

request_counter = Counter('http_requests_total', 'Total HTTP Requests', ['method', 'endpoint'])

def handle_request(method, endpoint):
    request_counter.labels(method=method, endpoint=endpoint).inc()
    ...

start_http_server(8000)  # Prometheus 会抓取 localhost:8000/metrics
  1. Distributed Tracing
    opentelemetry
1
2
3
4
5
6
7
8
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("my-function"):
    ...  # record execution time, exception, error, context

7. Ohter Advance Practices

1. Observability

Purpose: Make runtime behavior visible and measurable for debugging, monitoring, and alerting.

Key Practices:

  • Health Checks: Expose /health, /ready, or /live endpoints to indicate service status.
  • Metrics Exposure: Expose Prometheus-compatible /metrics endpoint.
  • Tracing and Exception Monitoring: Integrate tools to trace requests and capture errors.

Tools:

  • FastAPI, Flask with custom middleware for health/ready endpoints
  • Prometheus, OpenTelemetry, Grafana
  • Sentry for exception reporting

Example:

1
2
3
@app.get("/health")
def health_check():
    return {"status": "ok"}

2. Security Baseline

Purpose: Prevent security vulnerabilities during development and deployment.

Key Practices:

  • Dependency Vulnerability Scanning
  • Static Code Analysis
  • Secrets Detection
  • Least Privilege Execution

Tools:

  • pip-audit: detect vulnerable packages
  • bandit: static security analysis
  • gitleaks: find hardcoded secrets
  • Docker USER directive: run containers as non-root

CI Integration Example:

1
2
3
4
5
- name: Security Audit
  run: |
    pip install pip-audit bandit
    pip-audit
    bandit -r src/

3. Performance Strategy

Purpose: Optimize for critical paths, resource efficiency, and responsiveness.

Key Practices:

  • Profiling Critical Code Paths
  • Accelerate with Native Extensions
  • Caching Expensive Computations
  • Separate Async vs Parallel Work

Tools:

  • cProfile, py-spy, line_profiler
  • Cython, Rust + PyO3
  • functools.lru_cache, Redis, memcached
  • asyncio, concurrent.futures, multiprocessing

Example:

1
2
3
4
5
from functools import lru_cache

@lru_cache(maxsize=128)
def slow_query(x):
    ...  # expensive computation

4. Delivery and Operations

Purpose: Enable containerized deployment, reproducibility, and safe infrastructure automation.

Key Practices:

  • Multi-stage Docker Build
  • Non-root Containers
  • Kubernetes Deployment with Helm
  • Infrastructure as Code
  • Safe Database Migrations

Tools:

  • Docker, docker-compose
  • Helm, Terraform
  • alembic, flyway, django-migrations

Example: Minimal Dockerfile with best practices

1
2
3
4
5
6
7
FROM python:3.11-slim AS base
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
RUN useradd -m appuser
USER appuser
CMD ["python", "main.py"]