Skip to content

fix: validate header names/values and prevent CRLF injection (#1817) - #2442

Closed
SaumyaT-21 wants to merge 948 commits into
utksh1:mainfrom
SaumyaT-21:fix/44-crawler-header-validation
Closed

fix: validate header names/values and prevent CRLF injection (#1817)#2442
SaumyaT-21 wants to merge 948 commits into
utksh1:mainfrom
SaumyaT-21:fix/44-crawler-header-validation

Conversation

@SaumyaT-21

Copy link
Copy Markdown
Collaborator

Description

Validates HTTP header names and values in crawler.py to ensure they conform to HTTP specs.
Specifically:

  • Enforces header name and value validation against expected HTTP token grammar.
  • Checks for and rejects \r (Carriage Return) and \n (Line Feed) characters to prevent header injection (CRLF) vulnerabilities.
  • Raises a ValueError with clear messaging when an invalid header format or illegal character is detected.

Related Issues

Closes #1817

Type of Change

  • [x] Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

How Has This Been Tested?

Tested manually via a local test block in crawler.py using Python in the project's virtual environment:

  1. Valid Headers Test: Verified compliant header keys and values execute cleanly without errors.
  2. CRLF Injection Tests: Verified that passing \r or \n characters in either the header name or the header value correctly raises a ValueError.

Checklist

  • [x] My code follows the code style of this project.
  • [x] I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • [x] My changes generate no new warnings.

aarushlohit and others added 30 commits June 30, 2026 12:32
- New reportTemplates.ts service with ReportTemplate type, three built-in
  templates (executive, technical, compliance), and render/preview/export.
- ReportTemplatePicker.tsx slide-over component with type filtering, inline
  preview, and .md export.
- Integrate Templates button into Reports.tsx report cards.
- 21 unit tests covering template lifecycle, edge cases, and output.
…odule (utksh1#1524)

The extract_target helper in executor.py is a pure function but lives in
a heavy import chain (FastAPI, cache, config). Per the maintainer's
approved extraction pattern (used for routes_json_helpers), this extracts
extract_target into a small import-safe executor_target_helpers module
and re-exports it from executor.py so existing call sites keep working.

Closes utksh1#1389.

Co-authored-by: tmdeveloper007 <[email protected]>
utksh1 and others added 19 commits July 20, 2026 14:23
The debug default was changed from True to False in the security fix.
Update the test to match the new secure default.
The saved_views_router now has require_api_key dependency.
Override it in tests to bypass authentication for unit testing.
Add shared time_utils helpers and use timezone-aware UTC with an
explicit offset for generated_at and discovered_at across reports,
findings API responses, and report generation.

Closes utksh1#1882
Default to_utc_iso to timespec=auto so finding intelligence tests can
compare against datetime.now(UTC). Update TLS verification mocks for
crawler client.stream() and stub crawl_target in API scanner tests.
…idable

_init_default_policies() built the entire network denylist from the
single Pydantic field settings.network_denylist. Pydantic replaces
(rather than merges) a list field's default when SECUSCAN_NETWORK_DENYLIST
is set via env var, so any operator adding even one custom denylist
entry silently dropped the built-in protection for cloud metadata
(169.254.169.254), loopback, RFC1918/CGNAT ranges, and IPv6
link-local/ULA space -- reopening SSRF to the metadata endpoint despite
the code comment claiming the denylist was 'always enforced'.

Fix: move those ranges into a new MANDATORY_DENYLIST module constant
that is not read from settings and is applied unconditionally in
_init_default_policies before any operator-configured entries. The
operator-facing network_denylist setting is now purely additive.

Also updates the existing default-denylist test and adds a regression
test reproducing the exact scenario from utksh1#1748.
…ne-standardize-9bb6

fix(backend): standardize timezone handling to UTC ISO-8601
…t-metadata-ssrf

Fix utksh1#1748: make cloud-metadata/private-range denylist non-overridable
Fix: add auth and owner isolation to saved views API (closes utksh1#1743)
Cover the scapy_recon plugin parser.py with targeted behavioural tests:

- Metadata contract: file existence, valid JSON, required fields, engine
  binary, target/type field declarations
- ARP output: host count, IP+MAC extraction, finding keys, category,
  severity, description content, metadata consistency, remediation
- ICMP output: host count, IP extraction, Unknown-MAC default
- Single-host edge case: IP+MAC in result and description
- Malformed/empty input: empty string, whitespace-only, no UP: lines,
  mixed noise lines, malformed UP: lines, missing MAC separator

No changes to backend source; test file only.
* fix: stop dashboard polling after health failure and add manual retry

* fix: skip pre-existing upstream auth tests that cannot pass with mocked auth

* fix: update postcss to resolve GHSA-r28c-9q8g-f849 high severity vulnerability

* fix: document localhost-only Docker binding, add opt-in network override
@SaumyaT-21

Copy link
Copy Markdown
Collaborator Author

Hey @utksh1 ! Added the fix for CRLF injection and HTTP token validation. All green on CI checks. Please go ahead and review the changes!

@utksh1 utksh1 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: _HEADER_NAME_re.match() is not anchored, so a value such as X-Test@invalid can pass by matching only the valid prefix. Use a full match/anchors and add tests for trailing invalid characters. Also remove the unrelated frontend/package-lock.json update and add coverage for invalid field values through the actual header-building path.

@utksh1 utksh1 added area:backend Backend API, database, or service work level:intermediate 35 pts difficulty label for moderate contributor PRs type:security Security work category bonus label labels Aug 4, 2026
@SaumyaT-21

Copy link
Copy Markdown
Collaborator Author

Hey @utksh1 ! All requested backend header validation logic and unit test coverage have been updated and are passing locally. As requested, I've left package-lock.json untouched to keep this PR focused purely on backend changes but frontend checks are currently failing. Please review the latest commits and let me know how to proceed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:backend Backend API, database, or service work level:intermediate 35 pts difficulty label for moderate contributor PRs type:security Security work category bonus label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[#44] crawler injects extra-header values unsanitized