Misinterpreted "Cap lists at 5 items"

[nbali]

Based on my experience the limitation can be followed too literally (literal 5 limit) and/or applied on lists it shouldn't be applied on (lists that influence further results and shouldn't be really limited, grouped, etc at that point as they might be relevant later on).

In multiple occasions it didn't do what IMO the original intention was. It doesn't just split a list - that is bigger than 5 items - into "do now" vs "later," or "must" vs "nice to have.". It just literally "capped the list at 5 items". It should be called more like "split up", "create sections", "group", etc.

Also on top of this not every question has 5 (or less) valid answer, and there are lists that must have all the actually relevant items, even if it means having more than 5. I totally agree that the endless lists some newer models produce are bad, but intentionally dropping very relevant findings aren't really better either - and with the current wording it did happen.

For my environment I completely removed that section, and it still brings most of the benefit - with better accuracy - so that is an option as well, but there might be less-nuclear options than completely removing that section:

• the wording should be tweaked to drop anything barely resembling "limiting", and probably explicitly scoped to influence only the final output
• the item count limit should be made dynamic to be adjusted by the LLM based on the context/task itself - or at least increasesed to 10 so cuts less by accident

My point is that - if I get it right - this skill should be about influencing the output the user gets, not the internal thought process. Sure burning tokens on something that won't be shown/used later on is a waste, but I doubt we can influence the internal workings of a thought process without affecting result/finding validity/completeness, so instructions that might be interpreted as such should be made less ambigous and more focused about the scope.

I wanted to talk about this before I do any PR - or you might take it yourself.

i-have-adhd/skills/i-have-adhd/SKILL.md

Lines 105 to 108 in d05af1e

	### 9. Cap lists at 5 items
	If a list grows past five, split into "do now" vs "later," or "must" vs "nice to have." Five items ranked beats ten unranked.

[ayghri]

That's a good point. I think the best approach would be to formulate the rule in a way that avoids setting a specific cap number.
Let's try to come up with a clear solution in this direction.

[nbali]

@ayghri What was your original intention: to ignore/hide the "later"/"nice to have" section of the list or to have an emphasis difference between the two group?

[ayghri]

@nbali
What do you think about this? it keeps the size task-specific and adds a backlog to prevent the model from interpreting 'cap at 5' as 'discard the rest':

9. Cap active lists; backlog the rest

Put the most important and immediate items in the active list. Choose the active-list size based on the task while minimizing the user’s working set. Retain other relevant items in a hidden backlog instead of discarding them. As active items are completed and more work remains in scope, promote the next relevant backlog item. You can expose the backlog when the user requests an exhaustive inventory and group the output for readability.

[nbali]

NGL I have no clue how the LLM will understand "active list". Anyway this feels like it's strictly for a TODO list, but it influences a lof ot lists, not only "next action" lists.

This is what I'm testing right now

9. Split up lists over 5-10 items

If a list grows past 5-10 items, split into "do now" vs "later," or "must" vs "nice to have." Fewer items ranked beats more unranked.

Not perfect, but no loss so far.

[ayghri]

@nbali the main point is to avoid the rule being interpreted as "keep at most 5 and discard the rest", so suggesting the use of a backlog/"scratch pad" for the trimmed items should prevent that

[nbali]

What you suggested means is that:

• we work with top X
• the rest goes into some vaguely described background storage
• the only way they will surface if we explicitly ask for them, or the top ones has been "completed"

The problem is not every list contains completable items.

[ayghri]

not necessarily "completed", but more about keeping the top ones based on relevance. So if an item turned out to be "un-completable" we move to the next relevant one:

9. Cap task lists at 5
   List up to 5 task items most relevant to the current request. Retain all other relevant items in reserve. As context changes, surface whichever reserved items become relevant. If the reader requests an exhaustive list, show the items grouped for readability.

There was also a modification to "when to break the rules", rule 5:

" A rule fights the task. When a rule would delete the answer itself, the task wins; the shape stays. Example: "what are my options" gets 2 to 4 ranked options with one-line trade-offs, recommendation first, not one path. The options are the answer."

So if the task requires exhaustive list, the skill does not override the task (if the model is "smart" enough).

[nbali]

Well using "task list" instead of "list" should definitely narrow it down, as it most likely wouldn't consider most lists as "task lists", but other than that IMO it's still a rephrase and conceptually the same. One of the original 5 has to be considered less relevant as it was before for some reason. Apart from some explicit action - may it be called completion or not doesn't matter - I can't really see how would that happen. If the UI would support having a "X more" button that would reveal them in place it would be nice, but during my tests sometimes I didn't even get the info that there are more.

Meanwhile this rule should be occasionally applied on non-task lists as well. For example I have received non-task lists, where the over-eagerness of Fable/Opus was unbearable, so some kind of categorization/limiting/grouping would have been nice there as well.