Your changes

.dockerignore

@@ -0,0 +1,70 @@
+# Python
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+.Python
+env/
+venv/
+.venv/
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Virtual environments
+bin/
+include/
+lib/
+lib64/
+share/
+pyvenv.cfg
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Git
+.git/
+.gitignore
+
+# Docker
+Dockerfile
+.dockerignore
+
+# Temporary files
+*.tmp
+*.log
+
+# Output files
+output.csv
+annotated.pdf
+modified.pdf
+data.json
+data.txt
+
+# Test files (optional, if you want to exclude)
+# test_*.py
+
+# Environment files
+.env
+.env.local
+
+# Development files
+dev-requirements.txt
+pytest.ini
+.pre-commit-config.yaml
+.coverage
+htmlcov/
+.mypy_cache/
+.pytest_cache/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo

.env.example

@@ -0,0 +1,3 @@
+# Environment variables for Docker container
+DEFAULT_MAX_FILES=10
+DEFAULT_OUTPUT=/app/output/result.csv

.pre-commit-config.yaml

@@ -0,0 +1,23 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.4.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-added-large-files
+
+  - repo: https://github.com/psf/black
+    rev: 22.10.0
+    hooks:
+      - id: black
+
+  - repo: https://github.com/pycqa/isort
+    rev: 5.10.1
+    hooks:
+      - id: isort
+
+  - repo: https://github.com/pycqa/flake8
+    rev: 5.0.4
+    hooks:
+      - id: flake8

Dockerfile

@@ -0,0 +1,31 @@
+# Use Python 3.9 slim image
+FROM python:3.9-slim
+
+# Set working directory
+WORKDIR /app
+
+# Install system dependencies for PyMuPDF
+RUN apt-get update && apt-get install -y \
+    build-essential \
+    libmupdf-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy requirements and install Python dependencies
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy the application code
+COPY . .
+
+# Create directories for input/output if needed
+RUN mkdir -p /app/input /app/output
+
+# Set environment variables
+ENV DEFAULT_MAX_FILES=10
+ENV DEFAULT_OUTPUT=/app/output/result.csv
+
+# Set the default command
+ENTRYPOINT ["python", "main.py"]
+
+# Default arguments (can be overridden)
+CMD ["--help"]

README.md

@@ -29,29 +29,188 @@ We hope you find this code helpful for your invoice data extraction needs!
 
 ## Usage
 
-To use the code in this repository, follow the instructions below:
+### Local Setup
+
+To use the code locally:
 
 1. Activate the virtual environment:
    ```
-   source venv/bin/activate
+   source bin/activate
    ```
 
 2. Install the required dependencies:
    ```
    pip install -r requirements.txt
    ```
 
-3. Run the `main.py` file:
+   For development, also install:
+   ```
+   pip install -r dev-requirements.txt
+   ```
+
+3. Run the `main.py` file with a PDF file or directory:
+   ```
+   python main.py <pdf_file_or_directory> [--max-files N] [--output filename.csv]
+   ```
+
+   - For a single PDF: `python main.py path/to/invoice.pdf --pages 1,3`
+   - Skip specific pages: `python main.py path/to/invoice.pdf --skip-pages 2,4`
+   - Password-protected PDF: `python main.py path/to/invoice.pdf --password mypassword`
+   - Auto-rotate pages: `python main.py path/to/invoice.pdf --auto-rotate`
+   - Combined: `python main.py path/to/invoice.pdf --password mypassword --auto-rotate --pages 1,2`
+   - For batch processing: `python main.py path/to/directory/ --max-files 5 --output my_output.csv --auto-rotate --password batchpassword`
+
+   Options:
+   - `--max-files N`: Process up to N files (batch only).
+   - `--output filename.csv`: Specify output CSV filename.
+   - `--pages PAGES`: Process only specified pages (e.g., '1,3,5-7').
+   - `--skip-pages PAGES`: Skip specified pages (e.g., '2,4'). Overrides --pages if both specified.
+   - `--password PASSWORD`: Password for password-protected PDFs.
+   - `--auto-rotate`: Automatically detect and correct inverted/rotated pages.
+
+4. After the extraction process, a CSV file will be generated containing the extracted data. For batch processing, a detailed summary including processed, total, and error counts is displayed.
+
+### Docker Setup
+
+The project includes Docker support for easy deployment:
+
+#### Using Docker CLI
+
+1. **Build the Docker image:**
+   ```
+   docker build -t pdf-extractor .
+   ```
+
+2. **Run a single PDF extraction:**
+   ```
+   docker run --rm -v $(pwd)/input:/app/input -v $(pwd)/output:/app/output pdf-extractor /app/input/invoice.pdf --output /app/output/result.csv
+   ```
+
+3. **Run batch processing:**
+   ```
+   docker run --rm -v $(pwd)/input:/app/input -v $(pwd)/output:/app/output pdf-extractor /app/input/ --max-files 10 --output /app/output/batch_results.csv
+   ```
+
+   - Mount your input directory to `/app/input`
+   - Mount your output directory to `/app/output`
+   - Replace `$(pwd)` with the full path if on Windows
+
+4. **View help:**
+   ```
+   docker run --rm pdf-extractor --help
+   ```
+
+#### Using Docker Compose
+
+For convenience, a `docker-compose.yml` is provided:
+
+1. **Build and run:**
+   ```bash
+   docker-compose up --build
    ```
-   python main.py
+
+2. **Run with custom command:**
+   ```bash
+   docker-compose run --rm pdf-extractor /app/input/sample.pdf --output /app/output/result.csv
    ```
 
-   The program will prompt you to provide the location of the PDF file to extract data from.
+**Note:** Ensure your PDF files are in a directory mounted to `/app/input`, and output will be written to a directory mounted to `/app/output`.
+
+**Security Note:** When using `--password`, the password will be visible in command history and logs. For sensitive operations, consider using environment variables or secure input methods.
+
+## Docker
+
+This project is containerized using Docker for consistent deployment across environments. The following files are included:
+
+- `Dockerfile`: Multi-stage build with Python 3.9, installs dependencies, and sets up the application.
+- `docker-compose.yml`: Convenient orchestration for running the container.
+- `.dockerignore`: Excludes unnecessary files from the build context.
+
+### Building the Image
 
-4. After the extraction process, a CSV file named `output.csv` will be generated containing the extracted data.
+```bash
+docker build -t pdf-extractor .
+```
+
+### Running Containers
+
+- **Single File Processing:**
+  ```bash
+  docker run --rm -v /host/path/to/input:/app/input -v /host/path/to/output:/app/output pdf-extractor /app/input/sample.pdf --output /app/output/result.csv
+  ```
+
+- **Batch Processing:**
+  ```bash
+  docker run --rm -v /host/path/to/input:/app/input -v /host/path/to/output:/app/output pdf-extractor /app/input/ --max-files 5 --output /app/output/batch.csv
+  ```
+
+### Environment Variables (Optional)
+
+You can set environment variables for default behaviors:
+- `DEFAULT_MAX_FILES`: Default maximum files for batch processing
+- `DEFAULT_OUTPUT`: Default output filename
+
+Example:
+```bash
+docker run --rm -e DEFAULT_MAX_FILES=10 -v $(pwd)/data:/app/input pdf-extractor /app/input/
+```
 
 All the necessary functions are implemented in `qnot.py` and are imported into `main.py`.
 
+## Development
+
+### Requirements Files
+
+- `requirements.txt`: Production dependencies
+- `dev-requirements.txt`: Development dependencies including linting, testing, and documentation tools
+
+### Testing
+
+Run tests with:
+```bash
+python -m pytest
+```
+
+### Code Quality
+
+Use pre-commit hooks for code quality:
+```bash
+pip install pre-commit
+pre-commit install
+```
+
+Available tools: black, isort, flake8, mypy.
+
+## New Features
+
+- **Regex-based Extraction**: Added functions in `qnot.py` for extracting specific data using regex:
+  - `extract_prices(text)`: Extracts monetary values.
+  - `extract_quantities(text)`: Extracts integer quantities.
+  - `extract_sizes(text)`: Extracts size information (XS, S, M, etc.).
+  - `extract_dates(text)`: Extracts dates in common formats.
+  - `extract_emails(text)`: Extracts email addresses.
+  - `extract_phone_numbers(text)`: Extracts phone numbers.
+  - `regex_based_extraction(text)`: Combines all above into a dictionary.
+
+- **Batch Processing**:
+  - `process_batch_pdfs(input_dir, output_csv, max_files)`: Processes PDFs in a directory, optionally limiting to max_files, and returns processing statistics.
+
+- **Pydantic Validation**: Data validation using Pydantic models (`models.py`):
+  - `Product`: Validates product data with field-specific rules.
+  - `PageData`: Groups products per page.
+  - `LineItem`: Validates raw line items from tables.
+  - `InvoiceData`: Aggregates data from all pages with validation reports.
+  - Text validation for emails, phones, dates, and prices extracted from each page.
+
+- **Table Parsing**: Advanced table detection and parsing:
+  - Detects tables from text spans, handling merged cells and irregular structures.
+  - Extracts line items from tables as separate entities.
+  - Validates line items using Pydantic models.
+  - Generates CSV with both structured products and raw line items.
+  - Supports complex table layouts with row/column detection.
+
+- **Tests**: Unit tests for edge cases in `test_regex.py`. Run with `python -m pytest` or `pytest`.
+
 # Contribution
 
 Contributions to the "Extracting Data from PDF Invoices" project are welcome and encouraged! If you have any ideas, improvements, or bug fixes, please feel free to contribute.

dev-requirements.txt

@@ -0,0 +1,20 @@
+# Development dependencies
+-r requirements.txt
+
+# Linting and formatting
+flake8>=4.0.0
+black>=22.0.0
+isort>=5.10.0
+
+# Type checking
+mypy>=0.950
+
+# Testing
+pytest>=7.0.0
+pytest-cov>=3.0.0
+
+# Documentation
+sphinx>=4.0.0
+
+# Development tools
+pre-commit>=2.17.0

docker-compose.yml

@@ -0,0 +1,14 @@
+version: '3.8'
+
+services:
+  pdf-extractor:
+    build: .
+    volumes:
+      - ./input:/app/input
+      - ./output:/app/output
+    env_file:
+      - .env
+    environment:
+      - DEFAULT_MAX_FILES=${DEFAULT_MAX_FILES:-10}
+      - DEFAULT_OUTPUT=${DEFAULT_OUTPUT:-/app/output/result.csv}
+    command: ["--help"]