Add Playwright-backed row extraction and local LLM provider support#145
Closed
AdamEXu wants to merge 18 commits into
Closed
Add Playwright-backed row extraction and local LLM provider support#145AdamEXu wants to merge 18 commits into
AdamEXu wants to merge 18 commits into
Conversation
# Conflicts: # backend/src/mastra/tools/investigate-tool.ts
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (20)
📒 Files selected for processing (48)
Disabled knowledge base sources:
📝 WalkthroughWalkthroughThis change adds configurable LLM providers and provider-scoped model settings, including local setup, model discovery, schema finalization, and extractor configuration. Schema contracts now include codification profiles, richer column metadata, and array-based primary keys. Populate and refresh workflows gain queued investigations, browser extractors, structured sources, abort handling, and runtime controls. Convex persistence stores codification, extractor, provider, and cell-source data. Frontend setup, model selection, dataset creation, and source display are updated accordingly. Sequence Diagram(s)sequenceDiagram
participant User
participant SetupUI
participant Backend
participant Provider
User->>SetupUI: Select LLM provider and credentials
SetupUI->>Backend: Verify and save provider configuration
Backend->>Provider: Verify provider access
Provider-->>Backend: Verification response
Backend-->>SetupUI: Setup status and available models
Possibly related PRs
Suggested reviewers: ✨ Finishing Touches🧪 Generate unit tests (beta)
⚔️ Resolve merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Author
|
WIP: Still need to add sandboxing and other important features like that to make sure it's good and stuff |
Contributor
Author
|
have not tested the latest commit at all yet btw |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a Playwright-backed row extraction path that can build, validate, cache, run, and repair reusable extractors through TinyFish Browser. It also carries the supporting fixes needed to make that reliable in practice: richer schema contracts, provider-selectable LLM setup, row extractor settings, better refresh/populate cancellation, and UI/backend wiring for the new configuration.
What changed
Validation
git diff --check upstream/main...HEADcd backend && npm run buildcd frontend && npm run lintpassed with warnings only: existing/new<img>warnings, React Compiler/TanStack warning, generated Convex eslint-disable warnings, and an existing analytics hook dependency warning.cd frontend && npm run buildNotes for reviewers
The biggest behavioral change is that row collection can now move from one-off agent investigation to reusable browser extractors when the dataset has a stable page family. The provider/model and schema-contract changes are included because the extractor builder needs explicit model selection, validation rules, and typed normalization to avoid producing brittle scripts.