Skip to content

local-development.md

Latest commit

 

History

History
802 lines (578 loc) · 18.3 KB

local-development.md

File metadata and controls

802 lines (578 loc) · 18.3 KB

Local Development Guide

Complete guide for using pycontainer-build locally during development, including installation, building images, testing, and troubleshooting.

📦 Installation

From Source (Development)

The recommended way for local development:

# Clone the repository
git clone https://github.com/spboyer/pycontainer-build.git
cd pycontainer-build

# Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install in editable mode
pip install -e .

This installs the pycontainer command globally in your virtual environment, and any code changes you make are immediately reflected.

Verify Installation

# Check that the command is available
pycontainer --help

# Verify the Python API works
python -c "from pycontainer.builder import ImageBuilder; print('✓ Import successful')"

🏗️ Building Container Images Locally

Basic Build

# Build from the current directory
pycontainer build --tag myapp:latest

# Build from a specific directory
pycontainer build --tag myapp:latest --context /path/to/project

# Build with verbose output
pycontainer build --tag myapp:latest --verbose

Output Location

By default, images are created at:

<context-path>/dist/image/
├── index.json                  # OCI index (manifest list)
├── oci-layout                  # Version marker
├── blobs/sha256/
│   ├── <manifest-digest>       # Manifest blob
│   ├── <config-digest>         # Config blob
│   ├── <base-layer-digests>    # Base image layers (if using --base-image)
│   └── <app-layer-digest>      # Application layer (tar)
└── refs/tags/
    └── myapp:latest            # Tag reference

With Base Image & Dependencies

# Build with auto-detected Python base image
pycontainer build \
  --tag myapp:latest \
  --include-deps

# Or explicitly specify a base image
pycontainer build \
  --tag myapp:latest \
  --base-image python:3.12-slim \
  --include-deps

# Build for different platform (e.g., amd64 from ARM Mac)
pycontainer build \
  --tag myapp:amd64 \
  --base-image python:3.11-slim \
  --platform linux/amd64 \
  --include-deps

# Build for ARM64 (e.g., for AWS Graviton instances)
pycontainer build \
  --tag myapp:arm64 \
  --base-image python:3.11-slim \
  --platform linux/arm64 \
  --include-deps

# This will:
# 1. Pull python:3.11-slim layers from registry
# 2. Package your app files
# 3. Include pip-installed dependencies
# 4. Combine everything into a complete image

Preview Before Building

# Dry-run mode - see what will be built without creating files
pycontainer build --tag myapp:latest --dry-run --verbose

This shows:

  • Detected entry point
  • Files that will be included
  • Base image layers (if applicable)
  • Final image configuration

🧪 Testing Your Builds

Important: pycontainer creates OCI-compliant image layouts in dist/image/. These are not directly runnable by Docker without importing. See the options below for testing your built images.

Option 1: Use Skopeo (Recommended for OCI Layouts)

Skopeo is the proper tool for working with OCI image layouts and can copy them directly to Docker:

# Install skopeo
# macOS:
brew install skopeo

# Ubuntu/Debian:
sudo apt-get install skopeo

# Fedora/RHEL:
sudo dnf install skopeo

# 1. Build the image with pycontainer
pycontainer build --tag myapp:latest --base-image python:3.11-slim --include-deps

# 2. Copy the OCI layout to Docker daemon
skopeo copy oci:dist/image docker-daemon:myapp:latest

# 3. Run the container
docker run -p 8000:8000 myapp:latest

# Alternative: Copy to a registry
skopeo copy oci:dist/image docker://localhost:5000/myapp:latest

Option 2: Use Podman (Native OCI Support)

Podman natively supports OCI layouts without conversion:

# Install Podman
# macOS:
brew install podman
podman machine init
podman machine start

# Ubuntu/Debian:
sudo apt-get install podman

# Fedora/RHEL (included by default):
sudo dnf install podman

# 1. Build with pycontainer
pycontainer build --tag myapp:latest --base-image python:3.11-slim --include-deps

# 2. Run directly from the OCI layout
podman run --rm -p 8000:8000 oci:dist/image:myapp

# Or import into Podman's local storage
skopeo copy oci:dist/image containers-storage:localhost/myapp:latest
podman run -p 8000:8000 localhost/myapp:latest

Option 3: Use Docker with Manual Import

If you only have Docker and can't install skopeo:

# 1. Build the image with pycontainer
pycontainer build --tag myapp:latest --base-image python:3.11-slim --include-deps

# 2. Create a tar archive and import (less efficient)
tar -C dist/image -czf - . | docker import - myapp:latest

# Note: This creates a single-layer image and loses metadata
# For proper testing, use skopeo (Option 1) instead

# 3. Run the container
docker run -p 8000:8000 myapp:latest

Option 2: Use Podman

# Build with pycontainer
pycontainer build --tag myapp:latest --base-image python:3.11-slim --include-deps

# Import into Podman
podman load -i <(tar -C dist/image -cf - .)

# Run with Podman
podman run -p 8000:8000 myapp:latest

Option 4: Push to Local Registry

Run a local registry for testing across tools:

# Start a local registry (Docker or Podman required)
docker run -d -p 5000:5000 --name registry registry:2
# OR
podman run -d -p 5000:5000 --name registry docker.io/library/registry:2

# Build and push to local registry
pycontainer build \
  --tag localhost:5000/myapp:latest \
  --base-image python:3.11-slim \
  --include-deps \
  --push

# Pull and run from local registry
docker pull localhost:5000/myapp:latest
docker run -p 8000:8000 localhost:5000/myapp:latest

# Or with Podman:
podman pull localhost:5000/myapp:latest
podman run -p 8000:8000 localhost:5000/myapp:latest

# Clean up when done
docker stop registry && docker rm registry
# OR
podman stop registry && podman rm registry

Option 5: Test Without Containers

For pure Python testing (no container runtime needed):

# Just run your app directly with Python
cd /path/to/your/app
pip install -r requirements.txt
python -m app  # or uvicorn app.main:app, etc.

🐳 Understanding OCI vs Docker Images

Key Concept: pycontainer creates OCI image layouts, not Docker-specific images.

Format What is it? Tools that understand it
OCI Layout Standard directory structure (dist/image/) Podman, skopeo, buildah, containerd
Docker Image Docker daemon's internal format Docker only
Docker Archive Tar file with Docker-specific metadata Docker only
OCI Tarball Tar file with OCI layout Most tools

Why this matters:

  • Docker requires importing/converting OCI layouts before use
  • Podman can use OCI layouts directly
  • Skopeo is the bridge between OCI and Docker formats

Best practices:

  • Local testing: Use Podman or skopeo for native OCI support
  • CI/CD: Push to a registry; most tools pull from registries seamlessly
  • Production: Always use registries (GHCR, ACR, Docker Hub, etc.)

🔧 Development Workflow

Typical Local Development Flow

# 1. Make code changes to pycontainer-build
vim src/pycontainer/builder.py

# 2. Test with a sample project
cd examples/fastapi-app
pycontainer build --tag test:latest --verbose --dry-run

# 3. Build a real image
pycontainer build --tag test:latest --base-image python:3.11-slim --include-deps

# 4. Test the image
docker load -i <(tar -C dist/image -cf - .)
docker run -p 8000:8000 test:latest
curl http://localhost:8000

# 5. Iterate - changes to pycontainer source are live (editable install)

Quick Iteration with Examples

The examples/ directory contains ready-to-use test projects:

# FastAPI example
cd examples/fastapi-app
pycontainer build --tag fastapi-test:latest --include-deps --verbose

# Test the output structure
tree dist/image

# Inspect the manifest
cat dist/image/index.json | jq .

🎯 Common Local Use Cases

Use Case 1: Testing Config Changes

# Create a test config file
cat > pycontainer.toml << EOF
[build]
base_image = "python:3.11-slim"
workdir = "/app"
include_deps = true

[build.env]
DEBUG = "true"
PORT = "8000"

[build.labels]
maintainer = "[email protected]"
version = "dev"
EOF

# Build with the config
pycontainer build --tag myapp:dev --config pycontainer.toml --verbose

# Verify the config was applied
jq '.config' dist/image/blobs/sha256/<config-digest>

Use Case 2: Testing Entry Point Detection

# See what entry point is detected
pycontainer build --tag myapp:latest --dry-run --verbose | grep -i "entry"

# Override with explicit entry point
pycontainer build \
  --tag myapp:latest \
  --entrypoint '["python", "-m", "myapp.cli"]'

Use Case 3: Testing SBOM Generation

# Generate SPDX SBOM
pycontainer build --tag myapp:latest --sbom spdx --verbose

# Check the SBOM was created
ls -lh dist/image/sbom.spdx.json

# View the SBOM
cat dist/image/sbom.spdx.json | jq '.packages[].name'

Use Case 4: Testing Framework Auto-Detection

# FastAPI project
cd examples/fastapi-app
pycontainer build --tag test:latest --dry-run --verbose
# Should show: "Detected framework: fastapi"

# Flask project (create test)
mkdir test-flask && cd test-flask
cat > app.py << 'EOF'
from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello():
    return 'Hello from Flask!'
EOF

echo '[project]\nname="flask-test"\nversion="0.1.0"' > pyproject.toml

pycontainer build --tag flask-test:latest --dry-run --verbose
# Should show: "Detected framework: flask"

Use Case 5: Testing Cache Behavior

# First build (cold cache)
time pycontainer build --tag myapp:v1 --base-image python:3.11-slim --include-deps

# Second build (warm cache - should be faster)
time pycontainer build --tag myapp:v2 --base-image python:3.11-slim --include-deps

# Check cache usage
ls -lh ~/.pycontainer/cache/layers/

# Force rebuild without cache
pycontainer build --tag myapp:v3 --base-image python:3.11-slim --include-deps --no-cache

Use Case 6: Cross-Platform Builds

# Build for AMD64 from your ARM Mac
pycontainer build \
  --tag myapp:amd64 \
  --platform linux/amd64 \
  --base-image python:3.11-slim \
  --include-deps \
  --verbose

# Verify the platform in the image config
MANIFEST_DIGEST=$(jq -r '.manifests[0].digest' dist/image/index.json | cut -d: -f2)
CONFIG_DIGEST=$(jq -r '.config.digest' dist/image/blobs/sha256/$MANIFEST_DIGEST | cut -d: -f2)
jq '.architecture, .os' dist/image/blobs/sha256/$CONFIG_DIGEST
# Output: "amd64" "linux"

# Build for ARM64 (AWS Graviton, Raspberry Pi, etc.)
pycontainer build \
  --tag myapp:arm64 \
  --platform linux/arm64 \
  --base-image python:3.11-slim \
  --include-deps

# Build for both platforms and push
for PLATFORM in linux/amd64 linux/arm64; do
  ARCH=$(echo $PLATFORM | cut -d/ -f2)
  pycontainer build \
    --tag ghcr.io/user/myapp:${ARCH} \
    --platform $PLATFORM \
    --push
done

🔍 Inspecting Built Images

View Image Manifest

# Pretty-print the index
jq '.' dist/image/index.json

# Find the manifest digest
MANIFEST_DIGEST=$(jq -r '.manifests[0].digest' dist/image/index.json | cut -d: -f2)

# View the manifest
jq '.' dist/image/blobs/sha256/$MANIFEST_DIGEST

View Image Config

# Get config digest from manifest
CONFIG_DIGEST=$(jq -r '.config.digest' dist/image/blobs/sha256/$MANIFEST_DIGEST | cut -d: -f2)

# View the config
jq '.' dist/image/blobs/sha256/$CONFIG_DIGEST

Extract a Layer

# Get layer digest
LAYER_DIGEST=$(jq -r '.layers[0].digest' dist/image/blobs/sha256/$MANIFEST_DIGEST | cut -d: -f2)

# Extract to temporary directory
mkdir -p /tmp/layer
tar -xzf dist/image/blobs/sha256/$LAYER_DIGEST -C /tmp/layer

# View contents
tree /tmp/layer

Compare With Docker Images

# Build with pycontainer
pycontainer build --tag myapp:pycontainer --base-image python:3.11-slim --include-deps

# Build equivalent with Docker
cat > Dockerfile << EOF
FROM python:3.11-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["python", "-m", "app"]
EOF
docker build -t myapp:docker .

# Compare sizes
du -sh dist/image
docker images myapp:docker --format "{{.Size}}"

# Compare manifests
docker save myapp:docker -o docker-image.tar
tar -xf docker-image.tar
cat manifest.json | jq .

🛠️ Debugging & Troubleshooting

Enable Verbose Output

# See detailed build progress
pycontainer build --tag myapp:latest --verbose

# Outputs:
# - Project detection results
# - File paths being included
# - Layer creation progress
# - Registry push details (if --push)
# - Cache hits/misses

Common Issues & Solutions

Issue: "No entry point detected"

Solution: Explicitly specify entry point or add to pyproject.toml:

# Option 1: CLI flag
pycontainer build --tag myapp:latest --entrypoint '["python", "-m", "myapp"]'

# Option 2: Add to pyproject.toml
[project.scripts]
myapp = "myapp.cli:main"

Issue: "Base image pull failed"

Solution: Check registry authentication:

# For Docker Hub
docker login

# For GHCR
echo $GITHUB_TOKEN | docker login ghcr.io -u USERNAME --password-stdin

# For ACR
az acr login --name myregistry

# Then build
pycontainer build --tag myapp:latest --base-image python:3.11-slim

Issue: "Dependencies not found in image"

Solution: Ensure --include-deps and dependencies are installed:

# Install dependencies first
pip install -r requirements.txt

# Then build with --include-deps
pycontainer build --tag myapp:latest --base-image python:3.11-slim --include-deps

Issue: "Permission denied writing to dist/"

Solution: Check write permissions:

# Check permissions
ls -ld dist/

# Fix permissions
chmod -R u+w dist/

# Or use a different output directory
pycontainer build --tag myapp:latest --output /tmp/myimage

Issue: "Cache is stale or corrupted"

Solution: Clear cache and rebuild:

# Remove cache
rm -rf ~/.pycontainer/cache

# Rebuild
pycontainer build --tag myapp:latest --no-cache

Debug with Python API

For deeper debugging, use the Python API:

from pycontainer.config import BuildConfig
from pycontainer.builder import ImageBuilder
import logging

# Enable debug logging
logging.basicConfig(level=logging.DEBUG)

# Create config
config = BuildConfig(
    tag="myapp:debug",
    context_path=".",
    base_image="python:3.11-slim",
    include_deps=True,
    verbose=True
)

# Build and catch exceptions
try:
    builder = ImageBuilder(config)
    builder.build()
except Exception as e:
    print(f"Build failed: {e}")
    import traceback
    traceback.print_exc()

📊 Performance Tips

Optimize Build Speed

# 1. Use cache (enabled by default)
pycontainer build --tag myapp:latest --base-image python:3.11-slim --include-deps

# 2. Use smaller base images
pycontainer build --tag myapp:latest --base-image python:3.11-alpine --include-deps

# 3. Include only necessary files
pycontainer build \
  --tag myapp:latest \
  --include src/ \
  --include pyproject.toml \
  --include requirements.txt

Reduce Image Size

# Use distroless for smaller images
pycontainer build --tag myapp:latest --base-image gcr.io/distroless/python3-debian11

# Or use Alpine
pycontainer build --tag myapp:latest --base-image python:3.11-alpine

# Don't include unnecessary dependencies
pip install --no-dev -r requirements.txt
pycontainer build --tag myapp:latest --include-deps

🔗 Integration with Local Tools

Using with VS Code

# 1. Install the VS Code extension
code --install-extension ms-python.pycontainer

# 2. Open your project in VS Code
code /path/to/project

# 3. Use Command Palette (Cmd+Shift+P):
#    - "pycontainer: Build Container Image"
#    - "pycontainer: Build and Push Container Image"

Using with Poetry

# Add Poetry plugin
poetry self add poetry-pycontainer

# Build container
cd /path/to/poetry/project
poetry build-container --tag myapp:latest

# Configuration via pyproject.toml
[tool.pycontainer]
base_image = "python:3.11-slim"
include_deps = true
push = false

Using with Hatch

# Install Hatch plugin
pip install hatch-pycontainer

# Build (creates both wheel and container)
cd /path/to/hatch/project
hatch build

# Configuration via pyproject.toml
[tool.hatch.build.hooks.pycontainer]
base_image = "python:3.11-slim"
include_deps = true

📝 Example Projects

Try these ready-made examples:

FastAPI Application

cd examples/fastapi-app

# Quick build
pycontainer build --tag fastapi-demo:latest --include-deps --verbose

# Test locally
docker load -i <(tar -C dist/image -cf - .)
docker run -p 8000:8000 fastapi-demo:latest
curl http://localhost:8000

Minimal Test App

# Create a minimal test
mkdir minimal-test && cd minimal-test

cat > app.py << 'EOF'
print("Hello from pycontainer!")
EOF

cat > pyproject.toml << 'EOF'
[project]
name = "minimal-test"
version = "0.1.0"
EOF

# Build
pycontainer build --tag minimal:latest --verbose

# Inspect
tree dist/image

🚀 Next Steps

💡 Tips & Best Practices

  1. Always use --verbose during development to understand what's happening
  2. Use --dry-run to preview builds before creating artifacts
  3. Test with multiple base images (slim, alpine, distroless) to find the best fit
  4. Keep pycontainer.toml in your project for consistent builds
  5. Use cache for faster iterations (don't disable unless debugging cache issues)
  6. Version your images with meaningful tags (not just latest)
  7. Generate SBOMs for security compliance (--sbom spdx)
  8. Use reproducible builds for consistent outputs (--reproducible, enabled by default)

🆘 Getting Help

Happy building! 🐍🐳