Jira changelog items collected but not converted to domain layer (issue_changelogs)

[ciaramulligan]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

Jira status/resolution changelog items are collected and stored in _tool_jira_issue_changelog_items but a significant percentage are never promoted to the domain-level issue_changelogs table. This happens silently — no errors, no warnings.

The affected changelogs are from the same sync run, same board, same connection as the ones that ARE converted. The converter processed approximately 79% and silently skipped approximately 21% from the same sync run.

Across multiple Jira projects on the same instance, the percentage of resolved issues with no status changelogs in the domain table ranges from 4% to 99%.

For a specific issue, the tool layer has the data:

  SELECT ci.field, ci.from_string, ci.to_string             
  FROM _tool_jira_issue_changelog_items ci                                                                                                                                                                      
  JOIN _tool_jira_issue_changelogs cl
    ON ci.connection_id = cl.connection_id AND ci.changelog_id = cl.changelog_id                                                                                                                                
  WHERE cl.issue_id = <ISSUE_ID> AND cl.connection_id = <CONN_ID> AND ci.field = 'status';  

-- Returns: e.g. New → In Progress, In Progress → Closed

But the domain layer is empty:

  SELECT field_name, original_from_value, original_to_value                                                                                                                                                     
  FROM issue_changelogs                                                                                                                                                                                         
  WHERE issue_id = 'jira:JiraIssue:<CONN_ID>:<ISSUE_ID>' AND field_name = 'status'; 

-- Returns: (empty)

Jira UI (History tab) confirms the status transitions exist.

What we've ruled out:

• Not board scoping — all affected issues exist in _tool_jira_board_issues for the same board as the working issues.
• Not incremental since filter — _devlake_collector_latest_state shows time_after far in the past, meaning all changelogs should be converted.
• Not a collection gap — 64% of the affected issues have tool-level changelogs that were synced on the same date as the working ones.
• Not a specific field — the affected issues have changelog items for status, resolution, labels, Story Points, and other fields. None were converted.

Related issues: #7911, #7826, #8196, #7394

What do you expect to happen

Expected behavior:
All changelog items in _tool_jira_issue_changelog_items that have a matching _tool_jira_board_issues entry should be converted to issue_changelogs. The converter should not silently skip records — if a changelog item cannot be converted, it should log a warning.

Actual behavior:
The converter silently skips a subset of changelogs. No error is logged. The skipped items remain in the tool layer but never appear in the domain layer.

How to reproduce

1. Configure a Jira Cloud connection with a board
2. Run a full sync (not incremental)
3. Compare counts:

-- Tool layer count

  SELECT COUNT(*) FROM _tool_jira_issue_changelog_items ci                                                                                                                                                      
  JOIN _tool_jira_issue_changelogs cl                                                                                                                                                                           
    ON ci.connection_id = cl.connection_id AND ci.changelog_id = cl.changelog_id
  WHERE cl.connection_id = <CONN_ID> AND ci.field = 'status';   

-- Domain layer count

  SELECT COUNT(*) FROM issue_changelogs                     
  WHERE issue_id LIKE 'jira:JiraIssue:<CONN_ID>:%' AND field_name = 'status';

4. If domain count is significantly lower than tool count, the bug is present.
5. Verify affected issues have status transitions in Jira UI (History tab).

Anything else

Related issues

• [Bug][jira] field_id is empty in issue_changelogs #7911 — field_id empty in changelog items (fixed, PR fix: fill field_id of changelog item of Jira in tool layer (#7911) #7912)
• [Bug][Jira] issues disappearing from dataset, progressive load issue? #7826 — Issues disappearing from dataset (raw has data, domain doesn't — same symptom pattern)
• [Question][Jira Connector] Is there a way to allow Apache DevLake to connect on Jira Projects instead of a Jira Boards ? #8196 — Request to use Jira Projects instead of Boards
• feat: jira issue / changelog converter supports incremental mode #7394 — Incremental mode for changelog converter

Version

Latest as of March 2026. Managed instance — exact version/commit SHA not accessible.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

Hi @ciaramulligan! I'm Dosu and I’m helping the incubator-devlake team.

Good investigation — I was able to trace this to the converter's query logic.

The root cause is in issue_changelog_convertor.go. The query uses a LEFT JOIN to _tool_jira_board_issues, but then the WHERE clause filters on _tool_jira_board_issues.board_id = ?:

LEFT JOIN _tool_jira_board_issues ON (
    _tool_jira_board_issues.connection_id = _tool_jira_issue_changelogs.connection_id
    AND _tool_jira_board_issues.issue_id = _tool_jira_issue_changelogs.issue_id
)
WHERE _tool_jira_issue_changelog_items.connection_id = ?
  AND _tool_jira_board_issues.board_id = ?

This effectively turns the LEFT JOIN into an INNER JOIN — any changelog item whose issue is not present in _tool_jira_board_issues for the specific board being processed is silently dropped. No error, no warning.

So the ~21% of changelog items being skipped likely belong to issues that were collected (their changelogs exist in the tool layer) but are not associated with the board in _tool_jira_board_issues at conversion time. This can happen when:

1. The board's JQL filter doesn't match all issues that have changelogs in the tool layer
2. Issues moved between boards or lost board membership between collection runs
3. Changelogs were collected via a different path (e.g., epic-sourced issues) than the board-scoped issue list

This is not isolated to changelogs — the same LEFT JOIN + WHERE anti-pattern appears in at least 8 other converter files across the Jira, DORA, and Linker plugins [1] [2], meaning the same class of silent data loss can affect other domain tables too.

A minimal fix for changelogs would be to either:

• Change the LEFT JOIN to an explicit INNER JOIN (making the filtering intentional and obvious), or
• Move the board_id condition into the JOIN's ON clause so the LEFT JOIN semantics are preserved and non-board issues are still converted

And adding a log/warning when items are filtered by board scope would address the "silent" part of the problem.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}
   

[ciaramulligan]

Thanks @dosubot — that's a great analysis. The LEFT JOIN + WHERE anti-pattern explains exactly what we're seeing.

To confirm from our data: we verified that all affected issues exist in _tool_jira_board_issues for the board, but their changelogs may have been collected via epic-sourced or cross-referenced paths and lack the board association at conversion time. This matches your explanation.

The impact for us is significant — projects range from 4% to 99% missing changelogs in the domain layer, which makes any metric relying on issue_changelogs unreliable at scale.

We'd prefer option 2 (move board_id into the ON clause) — that preserves the LEFT JOIN semantics and ensures all collected changelogs are converted regardless of how the issue was collected. Option 1 (explicit INNER JOIN) would just make the current data loss intentional rather than fixing it.

Adding a log/warning for dropped items would also be valuable for debugging.

Is there a timeline for a fix?

[dosubot]

Glad the analysis matched what you're seeing. Your preference for option 2 (moving the board_id condition into the ON clause) makes sense — that would preserve all collected changelogs in the domain layer regardless of collection path, which is the correct behavior.

I don't have visibility into the maintainers' roadmap or release timeline, so I can't speak to when a fix might land. Since you've confirmed the root cause and the data impact is significant (4%–99% missing changelogs across projects), this issue is well-documented enough for someone to pick up. The fix itself is relatively contained — the core change in issue_changelog_convertor.go is moving _tool_jira_board_issues.board_id = ? from the WHERE clause into the LEFT JOIN's ON clause. But as noted, the same anti-pattern exists in 8+ other converter files, so an audit of those would be valuable as part of a broader fix.

If you or someone on your team is able to submit a PR, that would likely be the fastest path to a fix. The scope is well-defined and the change is low-risk for the changelog converter specifically.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}
   

[klesh]

Moving the board_id condition into the LEFT JOIN clause will cause all records in the connection to be converted into the domain layer whenever a task for a single board is executed, which can be quite slow and wasteful. It seems that you have multiple boards configured in DevLake that share many of the same issues. This overlap is not ideal, as issues between boards should ideally remain distinct.

[ciaramulligan]

Hi @klesh thanks — yes, we do have boards that share issues. This is common — teams create separate boards for different views (epics, bugs, backlog, kanban vs scrum) all pulling from the same project. We don't control how teams set up their boards, and asking them to restructure isn't practical.

Given that board overlap is a normal Jira pattern and not something we can change on our end, what would you suggest as a practical way forward to ensure collected changelogs aren't silently dropped during domain conversion while not impacting performance?

[klesh]

Hi @klesh thanks — yes, we do have boards that share issues. This is common — teams create separate boards for different views (epics, bugs, backlog, kanban vs scrum) all pulling from the same project. We don't control how teams set up their boards, and asking them to restructure isn't practical.

Given that board overlap is a normal Jira pattern and not something we can change on our end, what would you suggest as a practical way forward to ensure collected changelogs aren't silently dropped during domain conversion while not impacting performance?

I understand. In your case, I suggest that you do not use boards created by teams directly; instead, create new custom boards specifically for analysis.

[ciaramulligan]

Thanks @klesh.

Creating custom boards per team isn't practical for us — we're onboarding 100+ teams across multiple Jira projects, and we don't control how teams set up their boards. Maintaining a parallel set of "analysis boards" would add significant operational overhead and wouldn't scale.
The issue is that the converter drops collected data based on board association and we believe the converter should handle normal Jira board patterns — including overlapping boards — without data loss. The data is already collected and stored in the tool layer; it just needs to make it to the domain layer.