Skip to content

fix(ipc): increase scanner buffer size to 5MB to handle large outputs#8841

Open
rohitgulia wants to merge 2 commits intoapache:mainfrom
rohitgulia:fix/ipc-scanner-buffer-64kb-limit
Open

fix(ipc): increase scanner buffer size to 5MB to handle large outputs#8841
rohitgulia wants to merge 2 commits intoapache:mainfrom
rohitgulia:fix/ipc-scanner-buffer-64kb-limit

Conversation

@rohitgulia
Copy link
Copy Markdown

⚠️ Pre Checklist

Please complete ALL items in this checklist, and remove before submitting

  • I have read through the Contributing Documentation.
  • I have added relevant tests.
  • I have added relevant documentation.
  • I will add labels to the PR, such as pr-type/bug-fix, pr-type/feature-development, etc.

Summary

This PR increases the scanner buffer size from the default 64KB to 5MB in the IPC process utility (backend/core/utils/ipc.go). This fixes an issue where large outputs from subprocesses would be truncated when exceeding the 64KB buffer limit.

Changes:

  • Updated scanOutputPipe() function to set buffer size to 5MB
  • Updated scanErrorPipe() function to set buffer size to 5MB

This allows DevLake to properly handle large outputs from subprocess calls, particularly important for plugins that process large data payloads.

Does this close any open issues?

N/A (Please reference if applicable)

Screenshots

N/A

Other Information

  • Type: Bug Fix (pr-type/bug-fix)
  • Impact: Fixes subprocess output truncation for large outputs
  • Testing: Existing unit tests pass; buffer size increase is backward compatible with existing code

@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. pr-type/bug-fix This PR fixes a bug labels Apr 20, 2026
Copy link
Copy Markdown
Contributor

@klesh klesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe increasing the buffer size is a good idea. Would it be better to expose an Environment Variable to allow users to customize it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-type/bug-fix This PR fixes a bug size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants