Skip to content

TESTING.md

Latest commit

 

History

History
341 lines (245 loc) · 7.93 KB

TESTING.md

File metadata and controls

341 lines (245 loc) · 7.93 KB
Raw

Testing Guide

This document describes the testing infrastructure for Claude How To.

Overview

The project uses GitHub Actions to automatically run tests on every push and pull request. Tests cover:

  • Unit Tests: Python tests using pytest
  • Code Quality: Linting and formatting with Ruff
  • Security: Vulnerability scanning with Bandit
  • Type Checking: Static type analysis with mypy
  • Build Verification: EPUB generation test

Running Tests Locally

Prerequisites

# Install uv (fast Python package manager)
pip install uv

# Or on macOS with Homebrew
brew install uv

Setup Environment

# Clone the repository
git clone https://github.com/luongnv89/claude-howto.git
cd claude-howto

# Create virtual environment
uv venv

# Activate it
source .venv/bin/activate  # macOS/Linux
# or
.venv\Scripts\activate     # Windows

# Install development dependencies
uv pip install -r requirements-dev.txt

Run Tests

# Run all unit tests
pytest scripts/tests/ -v

# Run tests with coverage
pytest scripts/tests/ -v --cov=scripts --cov-report=html

# Run specific test file
pytest scripts/tests/test_build_epub.py -v

# Run specific test function
pytest scripts/tests/test_build_epub.py::test_function_name -v

# Run tests in watch mode (requires pytest-watch)
ptw scripts/tests/

Run Linting

# Check code formatting
ruff format --check scripts/

# Auto-fix formatting issues
ruff format scripts/

# Run linter
ruff check scripts/

# Auto-fix linter issues
ruff check --fix scripts/

Run Security Scan

# Run Bandit security scan
bandit -c pyproject.toml -r scripts/ --exclude scripts/tests/

# Generate JSON report
bandit -c pyproject.toml -r scripts/ --exclude scripts/tests/ -f json -o bandit-report.json

Run Type Checking

# Check types with mypy
mypy scripts/ --ignore-missing-imports --no-implicit-optional

GitHub Actions Workflow

Triggered On

  • Push to main or develop branches (when scripts change)
  • Pull Request to main (when scripts change)
  • Manual workflow dispatch

Jobs

1. Unit Tests (pytest)

  • Runs on: Ubuntu latest
  • Python versions: 3.10, 3.11, 3.12
  • What it does:
    • Installs dependencies from requirements-dev.txt
    • Runs pytest with coverage reporting
    • Uploads coverage to Codecov
    • Archives test results and coverage HTML

Outcome: If any test fails, the workflow fails (critical)

2. Code Quality (Ruff)

  • Runs on: Ubuntu latest
  • Python version: 3.11
  • What it does:
    • Checks code formatting with ruff format
    • Runs linter with ruff check
    • Reports issues but doesn't fail the workflow

Outcome: Non-blocking (warning only)

3. Security Scan (Bandit)

  • Runs on: Ubuntu latest
  • Python version: 3.11
  • What it does:
    • Scans for security vulnerabilities
    • Generates JSON report
    • Uploads report as artifact

Outcome: Non-blocking (warning only)

4. Type Checking (mypy)

  • Runs on: Ubuntu latest
  • Python version: 3.11
  • What it does:
    • Performs static type analysis
    • Reports type mismatches
    • Helps catch bugs early

Outcome: Non-blocking (warning only)

5. Build EPUB

  • Runs on: Ubuntu latest
  • Depends on: pytest, lint, security (all must pass)
  • What it does:
    • Builds the EPUB file using scripts/build_epub.py
    • Verifies the EPUB was created successfully
    • Uploads EPUB as artifact

Outcome: If build fails, the workflow fails (critical)

6. Summary

  • Runs on: Ubuntu latest
  • Depends on: All other jobs
  • What it does:
    • Generates workflow summary
    • Lists all artifacts
    • Reports overall status

Writing Tests

Test Structure

Tests should be placed in scripts/tests/ with names like test_*.py:

# scripts/tests/test_example.py
import pytest
from scripts.example_module import some_function

def test_basic_functionality():
    """Test that some_function works correctly."""
    result = some_function("input")
    assert result == "expected_output"

def test_error_handling():
    """Test that some_function handles errors gracefully."""
    with pytest.raises(ValueError):
        some_function("invalid_input")

@pytest.mark.asyncio
async def test_async_function():
    """Test async functions."""
    result = await async_function()
    assert result is not None

Test Best Practices

  • Use descriptive names: test_function_returns_correct_value()
  • One assertion per test (when possible): Easier to debug failures
  • Use fixtures for reusable setup: See scripts/tests/conftest.py
  • Mock external services: Use unittest.mock or pytest-mock
  • Test edge cases: Empty inputs, None values, errors
  • Keep tests fast: Avoid sleep() and external I/O
  • Use pytest markers: @pytest.mark.slow for slow tests

Fixtures

Common fixtures are defined in scripts/tests/conftest.py:

# Use fixtures in your tests
def test_something(tmp_path):
    """tmp_path fixture provides temporary directory."""
    test_file = tmp_path / "test.txt"
    test_file.write_text("content")
    assert test_file.read_text() == "content"

Coverage Reports

Local Coverage

# Generate coverage report
pytest scripts/tests/ --cov=scripts --cov-report=html

# Open the coverage report in your browser
open htmlcov/index.html

Coverage Goals

  • Minimum coverage: 80%
  • Branch coverage: Enabled
  • Focus areas: Core functionality and error paths

Pre-commit Hooks

The project uses pre-commit hooks to run checks automatically before commits:

# Install pre-commit hooks
pre-commit install

# Run hooks manually
pre-commit run --all-files

# Skip hooks for a commit (not recommended)
git commit --no-verify

Configured hooks in .pre-commit-config.yaml:

  • Ruff formatter
  • Ruff linter
  • Bandit security scanner
  • YAML validation
  • File size checks
  • Merge conflict detection

Troubleshooting

Tests Pass Locally but Fail in CI

Common causes:

  1. Python version difference: CI uses 3.10, 3.11, 3.12
  2. Missing dependencies: Update requirements-dev.txt
  3. Platform differences: Path separators, environment variables
  4. Flaky tests: Tests that depend on timing or order

Solution:

# Test with the same Python versions
uv python install 3.10 3.11 3.12

# Test with clean environment
rm -rf .venv
uv venv
uv pip install -r requirements-dev.txt
pytest scripts/tests/

Bandit Reports False Positives

Some security warnings may be false positives. Configure in pyproject.toml:

[tool.bandit]
exclude_dirs = ["scripts/tests"]
skips = ["B101"]  # Skip assert_used warning

Type Checking Too Strict

Relax type checking for specific files:

# Add at the top of file
# type: ignore

# Or for specific lines
some_dynamic_code()  # type: ignore

Continuous Integration Best Practices

  1. Keep tests fast: Each test should complete in <1 second
  2. Don't test external APIs: Mock external services
  3. Test in isolation: Each test should be independent
  4. Use clear assertions: assert x == 5 not assert x
  5. Handle async tests: Use @pytest.mark.asyncio
  6. Generate reports: Coverage, security, type checking

Resources

Contributing Tests

When submitting a PR:

  1. Write tests for new functionality
  2. Run tests locally: pytest scripts/tests/ -v
  3. Check coverage: pytest scripts/tests/ --cov=scripts
  4. Run linting: ruff check scripts/
  5. Security scan: bandit -r scripts/ --exclude scripts/tests/
  6. Update documentation if tests change

Tests are required for all PRs! 🧪

For questions or issues with testing, open a GitHub issue or discussion.