Skip to content

fix context routing when sentry telemetry tasks are misread as junior-local work #243

@sentry-junior

Description

@sentry-junior

this came up in a slack thread where junior was asked to investigate april 21 signup API errors in sentry.

what happened:

  • junior asked which sentry org/project to search instead of using the obvious thread/workspace context.
  • after the user clarified sentry org / python project and gave additional code-context hints, junior still answered as if the runtime lacked the necessary tooling.
  • junior later acknowledged that the failure was routing/context selection, not actual lack of sentry capability.

problems observed:

  • over-anchored on junior's own product/repo context instead of the active slack/workspace context.
  • failed to prefer the explicit task operation (look for sentry errors) and load/use the sentry skill.
  • treated missing explicit provider target as blocking even though the thread strongly implied the default target.
  • produced a misleading explanation about missing CLIs/tooling rather than executing the available sentry path.
  • did not recover correctly after the user supplied stronger context in-thread.

hypothesis:
there are two related failures in model behavior:

  1. context-priority bug: repo/app identity context (junior, junior project, junior repo) is overpowering workspace/channel/task context, so the model chooses the wrong default target when the task is clearly about sentry product telemetry.
  2. skill-routing bug: the model is reasoning about generic tools/CLI availability before routing to the operation-specific skill. once it frames the task as "tooling missing," it sticks to that frame instead of re-evaluating when the sentry skill is available and the user provides stronger context.

likely contributing factors:

  • too much high-salience identity/config text near the top of the prompt about junior's own repo/project.
  • not enough explicit instruction to prefer thread-local/task-local context over assistant identity context.
  • insufficient bias toward loading the domain skill that matches the requested operation before discussing limitations.
  • weak recovery behavior after user correction.

suggested fixes:

  • make task/operation routing happen before assistant-self context is considered for defaults.
  • if the user asks to inspect live sentry errors/issues/events, strongly prefer the sentry skill unless they explicitly ask for repo/code work.
  • add a guardrail: don't claim missing tooling for sentry lookups if the sentry skill is available but unused.
  • when a user clarifies org/project/repo in-thread, force a re-route/re-evaluation instead of continuing the prior frame.
  • consider install/workspace-wide defaults for providers like sentry.org=sentry to reduce avoidable clarification churn.

thread evidence:

  • initial miss: asked for org/project instead of assuming sentry context.
  • after clarification: user explicitly said sentry org / python project and suggested getsentry/sentry for code context.
  • later self-assessment: junior stated it did have the sentry skill and should have used it, and that the real failure was routing/context selection.

this issue is about the misrouting/model behavior. install-wide defaults may help, but they do not fully explain why the available sentry skill was not selected after clear user correction.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions