Skip to content

Commit 611d76d

Browse files
committed
- Removing the Convenient Functions to unify the Class instaniation approach
- Warnings Suppression fix - Lazy imports for faster loading time of PDFstract CLI - Adding Logs into Embedding factory - Docs Update
1 parent 887faa5 commit 611d76d

14 files changed

Lines changed: 12362 additions & 951 deletions

File tree

.gitignore

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,6 @@ ultralytics-cache/*
1515
results/
1616
*chunks.json
1717
.env.example
18-
uv.lock
18+
uv.lock
19+
test.ipynb
20+
notebooks/

.project_doc_record/meta-info.json

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"doc_version": "",
3+
"in_generation_process": true,
4+
"fake_file_reflection": {},
5+
"jump_files": [],
6+
"deleted_items_from_older_meta": []
7+
}

.project_doc_record/project_hierarchy.json

Lines changed: 12035 additions & 0 deletions
Large diffs are not rendered by default.

CLI_README.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -29,10 +29,10 @@ pip install pdfstract[all]
2929
### Convert a PDF
3030

3131
```python
32-
from pdfstract import convert_pdf
32+
from pdfstract import PDFStract
3333

34-
# One-liner conversion
35-
text = convert_pdf('document.pdf', library='marker')
34+
pdfstract = PDFStract()
35+
text = pdfstract.convert('document.pdf', library='marker')
3636
print(text)
3737
```
3838

@@ -141,7 +141,7 @@ pdfstract compare sample.pdf -l marker -l docling -l pymupdf4llm
141141

142142
## Embeddings
143143

144-
PDFStract can generate embeddings using multiple providers (OpenAI, Azure OpenAI, Google Generative, Ollama, Sentence-Transformers, Model2Vec). Use the Python API `embed_text` / `embed_texts` or the CLI `pdfstract embeddings-list` and `pdfstract embed-text` commands. Hosted providers require API keys; local providers require installed models or running services.
144+
PDFStract can generate embeddings using multiple providers (OpenAI, Azure OpenAI, Google Generative, Ollama, Sentence-Transformers, Model2Vec). Use a `PDFStract()` instance and call `embed_text` / `embed_texts`, or use the CLI `pdfstract embeddings-list` and `pdfstract embed-text` commands. Hosted providers require API keys; local providers require installed models or running services.
145145

146146
## Web UI & Docker
147147

README.md

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ chunks=ps.convert_chunk("path/to/pdf", library="auto", chunker="auto")
6262

6363
## Embeddings
6464

65-
PDFStract can also generate vector embeddings from text using pluggable providers (OpenAI, Azure OpenAI, Google Generative, Ollama, Sentence-Transformers, Model2Vec). Use `PDFStract.embed_text(s)` or the convenience `embed_text(s)` functions. Hosted providers require API keys (see docs), while local providers like Sentence-Transformers and Ollama run locally.
65+
PDFStract can also generate vector embeddings from text using pluggable providers (OpenAI, Azure OpenAI, Google Generative, Ollama, Sentence-Transformers, Model2Vec). Use `pdfstract.embed_text(s)` / `pdfstract.embed_texts(...)` on a `PDFStract()` instance. Hosted providers require API keys (see docs), while local providers like Sentence-Transformers and Ollama run locally.
6666

6767
# or do it in two steps
6868
# convert first with your library of choice
@@ -134,13 +134,13 @@ pdfstract download marker
134134

135135
You don't need to use the CLI! PDFStract can be easily integrated into your Python applications as a library.
136136

137-
#### Convert a PDF (One-liner)
137+
#### Convert a PDF
138138

139139
```python
140-
from pdfstract import convert_pdf
140+
from pdfstract import PDFStract
141141

142-
# Quick conversion with default settings
143-
result = convert_pdf('sample.pdf', library='marker')
142+
pdfstract = PDFStract()
143+
result = pdfstract.convert('sample.pdf', library='marker')
144144
print(result) # Markdown content
145145
```
146146

@@ -150,8 +150,6 @@ print(result) # Markdown content
150150
from pdfstract import PDFStract
151151

152152
pdfstract = PDFStract()
153-
154-
# Get list of available libraries
155153
available = pdfstract.list_available_libraries()
156154
print(available) # ['pymupdf4llm', 'marker', 'docling', ...]
157155
```

cli.py

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,17 @@
44
Provides: single conversions, multi-library comparisons, batch processing
55
"""
66

7+
# Suppress noisy warnings from third-party libraries during CLI runs
8+
import warnings
9+
warnings.filterwarnings(
10+
"ignore",
11+
message=".*urllib3.*or chardet.*doesn't match a supported version.*",
12+
module="requests",
13+
)
14+
warnings.filterwarnings("ignore", category=UserWarning)
15+
warnings.filterwarnings("ignore", category=DeprecationWarning)
16+
warnings.filterwarnings("ignore", category=SyntaxWarning)
17+
718
import click
819
import json
920
import os
@@ -12,7 +23,6 @@
1223
from typing import List, Dict, Optional
1324
from datetime import datetime
1425
from concurrent.futures import ThreadPoolExecutor
15-
import csv
1626
import sys
1727

1828
from rich.console import Console
@@ -24,7 +34,7 @@
2434
from services.cli_factory import CLILazyFactory
2535
from services.base import OutputFormat
2636
from services.logger import logger
27-
from services.chunker_factory import get_chunker_factory
37+
# get_chunker_factory imported lazily inside chunk/convert-chunk/chunkers to avoid loading chonkie on every CLI run
2838

2939
# Import version for --version option (reads from api module which reads from pyproject.toml)
3040
try:
@@ -789,6 +799,7 @@ def batch_compare(input_dir: Path, libraries: tuple, format: str, output: str, m
789799
@pdfstract.command()
790800
def chunkers():
791801
"""List all available text chunkers and their parameters"""
802+
from services.chunker_factory import get_chunker_factory
792803
cli_app.print_banner()
793804

794805
factory = get_chunker_factory()
@@ -880,6 +891,7 @@ def chunk(input_file: Path, chunker: str, chunk_size: int, chunk_overlap: int, o
880891
cli_app.print_info(f"Chunker: {chunker} | Size: {chunk_size} | Overlap: {chunk_overlap}")
881892

882893
try:
894+
from services.chunker_factory import get_chunker_factory
883895
factory = get_chunker_factory()
884896

885897
with Progress(
@@ -1042,6 +1054,7 @@ def convert_chunk(
10421054
# Step 2: Chunk the converted text
10431055
progress.add_task("Chunking text...", total=None)
10441056

1057+
from services.chunker_factory import get_chunker_factory
10451058
factory = get_chunker_factory()
10461059
result = asyncio.run(
10471060
factory.chunk_with_result(chunker, converted_text, **chunker_params)

docs/docs/api/overview.md

Lines changed: 6 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -183,14 +183,14 @@ print(f"Semantic chunker: {info}")
183183
Generate vector embeddings for text using pluggable providers.
184184

185185
```python
186-
# Embed multiple texts (synchronous)
187-
from pdfstract import embed_texts
188-
vectors = embed_texts(["First sentence.", "Second sentence."], model='auto')
186+
from pdfstract import PDFStract
189187

190-
# Using the PDFStract object (async + sync)
191188
pdf = PDFStract()
192-
vecs = pdf.embed_texts(["Hello world"], model='sentence-transformers')
189+
# Sync
190+
vecs = pdf.embed_texts(["First sentence.", "Second sentence."], model='auto')
193191
print(len(vecs[0])) # embedding dimension
192+
# Or single text
193+
e = pdf.embed_text("Hello world", model='sentence-transformers')
194194
```
195195

196196
Parameters:
@@ -201,39 +201,7 @@ Notes:
201201
- Credentials are validated internally and a clear error is raised if required environment variables are missing.
202202
- For hosted providers set `OPENAI_API_KEY`, `AZURE_OPENAI_KEY`, `GOOGLE_API_KEY`, etc. For Ollama, ensure a local Ollama daemon is running.
203203

204-
## Convenience Functions
205-
206-
PDFStract also provides standalone functions for quick operations:
207-
208-
### PDF Conversion Functions
209-
210-
```python
211-
from pdfstract import (
212-
convert_with_docling,
213-
convert_with_marker,
214-
convert_with_pymupdf4llm,
215-
convert_with_unstructured
216-
)
217-
218-
# Use specific converters directly
219-
text = convert_with_marker('document.pdf')
220-
text = convert_with_docling('document.pdf', extract_images=True)
221-
```
222-
223-
### Chunking Functions
224-
225-
```python
226-
from pdfstract import (
227-
chunk_by_tokens,
228-
chunk_semantically,
229-
chunk_recursively,
230-
chunk_by_sentences
231-
)
232-
233-
# Use specific chunkers directly
234-
chunks = chunk_semantically(text, chunk_size=512)
235-
chunks = chunk_by_tokens(text, chunk_size=1024, overlap=100)
236-
```
204+
Use a single `PDFStract()` instance for all operations: `pdfstract.convert(...)`, `pdfstract.chunk_text(...)`, `pdfstract.embed_texts(...)`, etc. Instantiate once and reuse.
237205

238206
## Error Handling
239207

docs/docs/quick-start.md

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -87,14 +87,12 @@ Set `chunker='auto'` to automatically select the best available method (typicall
8787
PDFStract can produce vector embeddings using multiple providers. Embeddings are useful for indexing, semantic search, and RAG pipelines.
8888

8989
```python
90-
# Embed a list of texts using auto-selected provider
91-
from pdfstract import embed_texts
92-
vecs = embed_texts(["First sentence", "Second sentence"], model='auto')
93-
print(len(vecs[0]))
90+
from pdfstract import PDFStract
9491

95-
# Embed a single text
96-
from pdfstract import embed_text
97-
e = embed_text("Hello world", model='sentence-transformers')
92+
pdf = PDFStract()
93+
vecs = pdf.embed_texts(["First sentence", "Second sentence"], model='auto')
94+
print(len(vecs[0]))
95+
e = pdf.embed_text("Hello world", model='sentence-transformers')
9896
```
9997

10098
Notes:

pdfstract/__init__.py

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
See services/ for implementation details
2929
"""
3030

31-
from .api import PDFStract, convert_pdf, list_available_libraries, chunk_text, list_available_chunkers, list_available_embeddings, embed_text, embed_texts
31+
from .api import PDFStract
3232

3333
# Version is managed in pyproject.toml
3434
# Read it dynamically to keep a single source of truth
@@ -51,13 +51,6 @@
5151

5252
__all__ = [
5353
"PDFStract",
54-
"convert_pdf",
55-
"list_available_libraries",
56-
"chunk_text",
57-
"list_available_chunkers",
58-
"list_available_embeddings",
59-
"embed_text",
60-
"embed_texts",
6154
"__version__",
6255
]
6356

0 commit comments

Comments
 (0)