Set mock test parameter to save resources - #49
Merged
Merged
Conversation
- Introduce `SubRequirementReport` model in `RequirementReport` to expose article-level audit details. - Update `evaluate_requirement` to populate sub-requirement results. - Improve `get_sub_prompt` for stricter and more structured evaluation. - Improve `get_aggregate_prompt` to generate a summary that explicitly mentions sub-requirement coverage. - Optimize `_query_qdrant_for_requirement` to use cleaned query text (removing law names) for better retrieval precision. Co-authored-by: davidedm26 <[email protected]>
- Ensure `Auditor_Notes` in `RequirementReport` is always a string by serializing dict/list outputs from LLM. - Introduce `SubRequirementReport` model to expose granular audit details. - Update `evaluate_requirement` to populate `SubRequirements` list. - Enhance `_query_qdrant_for_requirement` to accept `embedding_model` for re-embedding cleaned queries. - Refine prompt templates for better structured output. Co-authored-by: davidedm26 <[email protected]>
- Extract `RetrievalEngine` to `backend/core/retrieval.py` for query cleaning and embedding. - Extract `EvaluationEngine` to `backend/core/evaluation.py` for prompt management and LLM interaction. - Update `rag_engine.py` to orchestrate these components. - Fix Pydantic `ValidationError` by ensuring `Auditor_Notes` is always a string. - Update `evaluate_rag.py` to use the new modular structure. Co-authored-by: davidedm26 <[email protected]>
- Explicitly formulate `ragas_question` in `EvaluationEngine` to include regulatory content, improving Answer Relevancy. - Combine LLM `rationale` and `auditor_notes` as the `answer` for RAGAS evaluation, improving Groundedness and Faithfulness. - Update `_get_sub_prompt` to force the LLM to use direct quotes from document chunks, grounding its reasoning in the text. Co-authored-by: davidedm26 <[email protected]>
- Add `main_req_name` to `_get_sub_prompt` for better LLM context.
- Soften prompt instructions to encourage referencing specific text parts rather than strict verbatim quotes, reducing false negatives ("no evidence").
- Simplify `ragas_question` to avoid duplicating the regulatory text already present in the context, improving Answer Relevancy scoring.
- Ensure all caller sites in `rag_engine.py` and `evaluate_rag.py` pass the new arguments correctly.
Co-authored-by: davidedm26 <[email protected]>
- Explicitly pass `auditor_notes` in `sub_results` dict within `rag_engine.py`. - Update `EvaluationEngine._get_aggregate_prompt` to read the concise `auditor_notes` instead of the verbose `answer` (which includes the full rationale). Co-authored-by: davidedm26 <[email protected]>
- Remove the verbose `Context` array from the `RequirementReport` Pydantic model to prevent massive JSON bloat in the final output. The structured `SubRequirements` list is retained. - Update `EvaluationEngine._get_sub_prompt` to include the `source` (e.g., EU AI Act, ISO 42001) alongside the reference, providing the LLM with explicit knowledge of the regulatory framework being evaluated. - Update `evaluate_rag.py` and `rag_engine.py` to pass the `source` argument. Co-authored-by: davidedm26 <[email protected]>
- Update `evaluate_single_case` to read data from `SubRequirements` instead of the legacy, removed `Context` field. - Reconstruct `ragas_question` and `combined_answer` dynamically from the Pydantic properties (`Rationale`, `Auditor_Notes`, `Reference`, `Source`). Co-authored-by: davidedm26 <[email protected]>
This reverts commit 9ef2c43.
…ify-articles-logic
…ify-articles-logic
This reverts commit e15d94c.
…ebug script to obtain audit json responses
adding_references_notes
…ify-articles-logic
Feat/modify-articles-logic
- Removed train-of-thought comments from `backend/rag_engine.py` - Removed first-person and conversational notes from `evaluation/case_evaluation.py`, `frontend/app.py`, `vectorize_data.py`, and `evaluate_rag.py` - Reformatted excessively long single-line comments into standard professional docstrings and block comments - Removed "AI engine" phrasing from frontend UI notes to sound more professional Co-authored-by: davidedm26 <[email protected]>
Updates the backend API endpoints to properly expose `SubRequirements` by including `SubRequirementReportAPI` in the lightweight response model `RequirementReportAPI`. Modifies the frontend `Audit_Compliance.py` logic to parse the `sub_requirements` and renders them interactively using an `st.expander` toggle list under each top-level requirement to improve granular visibility of regulatory analysis. Co-authored-by: davidedm26 <[email protected]>
…galAIze into feat/general-debug
Feat/general-debug
Debug container
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.