Competencies repeat within framework and throughout extract

[ccalixto1]

Issue:
Some competencies frameworks repeat competencies multiple times throughout the framework. For example, the framework below is entirely comprised of multiple repetitions of the same 4 competencies: "Critical Thinking," "Information Competency & Technology Literacy," "Self-Development & Global Awareness," and "Communication Skills."

This happens again with the framework below.

It seems that repetitions of competencies all happen to be the same 4 competencies mentioned above. Could it be that these competencies are somehow causing errors?

Details:
Found in Moreno Valley College Course Competency Extract: Agentic Extraction

Anticipated results:
Frameworks do not have multiple repetitions of competencies throughout the framework. Each competency only appears once throughout the framework.

@jillianscholten tagging for reference on CA MAP.

[alexculealt]

Indeed I'm able to see the same repetition and it occured after a newer extraction. The reason is because the model sees some of them in the course content:

This image is from this url.

@ccalixto1, seeing how the subpoints could be interpreted as competency frameworks, could we confirm what is the desired outcome? Should these subpoints be dropped resulting in CF called "Medical Social Work" for this specific URL, or is it a different CF we consider to be correct?

[ccalixto1]

@alexculealt thank you for catching this! @jillianscholten and I will bring this question back to the CA MAP team for feedback, and will follow up on next steps once we've received direction from them. We can hold on this issue for now. In the meantime, please let me know if you need any further information from our end for this issue.

[ccalixto1]

Hi @alexculealt, we've heard back from the CA MAP team. They would like to move forward with pulling only the top level competency (i.e. "Define and analyze concepts and theories related to medical social work practice"). At this time, we do not need to worry about pulling any sub-competencies at all (i.e. "Critical Thinking").

I do see that some frameworks in this extract contain top level competencies, but some -- like the "Administration of Justice" framework seen in the first screenshot on this issue -- contain no top level competencies at all, only contain sub-competencies. Is it possible to rerun this extract? Thank you again for your help with this issue!

[alexculealt]

The original issue has been resolved - the competencies are no longer repeating and no longer extracted from the sub-points, the issue no longer being observed in any recent extractions.

Some different issues arise separately:

1. Sonnet 4, the model we have used throughout became highly inconsistent with how it processes this catalogue, with it not following the full catalogue pagination, or it is submitting invalid data to xTRA (either random scripts it uses or text that lost its formatting).
2. GPT 5.4 is inconsistent in identifying using the presence prompt, therefore skipping some pages from extraction as it believes there are no competencies on the page.

Prompt adjustments have improved the extractions coverage and quality but more work is required. On the Anthropic side, a quick win will be to update to Sonnet 5 release recently and hope it works as good as Sonnet 4 used to work a month ago. As for the detection prompt, we will have to improve its content and potentially fire multiple indentical requests and compute an average as a response.

Two of the better resulting (but still incomplete) extractions are here:

• https://drive.google.com/file/d/1GQQnO1JCisNh9WkSDM1z7mFU9E-h0dKa/view?usp=drive_link
• https://drive.google.com/file/d/1YHn7GxNnRDrDqJTUD6fSj7-vwf7Zemfp/view?usp=sharing

[alexculealt]

@ccalixto1, following the latest extraction using our agentic MVP for Moreno Course Competencies we have the resulting CSV at here. The agentic MVP works well for many cases but it shows clear limitations when extracting catalogues of this size requiring more manual work to fix certain model misbehaviors that in xTRA are either already mitigated or there are better foundations to improve the model harness (guiding tips). I mention this as we might be at the point where manual adjustments on top of this extraction will be more efficient than repeating the extraction excercise.

Please let me know if there is anything else left to address.