Blog: add update to Security CI incident

apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md

@@ -1,13 +1,75 @@
 ---
-date: '2025-03-31T16:30:00.617Z'
+date: '2025-04-23T16:30:00.617Z'
 category: vulnerability
 title: Node.js Test CI Security Incident
 layout: blog-post
 author: Node.js Technical Steering Committee
 ---
 
-On March 21st, the Node.js project received a security report regarding our development infrastructure via [our bug bounty program](https://hackerone.com/nodejs). We immediately restricted access while implementing corrective actions.
+# _(Update 21-April-2025)_ Node.js Test CI Security Incident – Full Disclosure
+
+## Summary
+
+On March 21, 2025, we received a [security report via HackerOne](https://hackerone.com/reports/3050534) (link restricted at time of writing), detailing a successful compromise of several Node.js test CI hosts.
+
+According to the HackerOne report, the exploit proceeded as follows:
+
+1. Submit a valid pull request to nodejs/node.
+2. Wait for a maintainer to add the `request-ci` label (this label is added to every pull request with non-documentation changes).
+3. After approval, update the pull request using an outdated Git commit timestamp.
+4. When Jenkins pipelines trigger, they fetch and execute code from the forked pull request.
+5. Attain code execution on Node.js Jenkins agents.
+
+Upon review, we identified that the `request-ci` label step simplifies but is **not required** to carry out the attack. A similar attack could be used against the `commit-queue` label, thus potentially allowing an attacker to land an unauthorized code change.
+
+The core issue stems from a Time-of-Check-Time-of-Use (TOCTOU) vulnerability between initiating a CI build and the moment the Jenkins job checks out the code. Previously, CI jobs used Git references (`refs/pull/<pr_id>/head`), which attackers can alter after triggering the CI. Importantly, the collaborator initiating the CI build did nothing wrong—the pull request appeared safe when CI was triggered.
+
+![Example of workflow for starting Jenkins CI on a Github Pull Request][example_test_infra]
+
+![Example of attack in the Node.js test infra][example_attack_test_Infra]
+
+## Remediation
+
+In response to this security incident, the Node.js security team took measures to mitigate risks and secure the infrastructure.
+
+- Immediately upon confirmation of the vulnerability, access to initiate new Jenkins CI runs was restricted to prevent further compromise while the team validated the report.
+- All compromised hosts (24 machines) were swiftly identified, removed from Jenkins, and rebuilt to eliminate any potential residual risk left over from the initial ingress.
+- Security improvements were implemented in Jenkins jobs to validate commit SHAs, ensuring jobs only executed trusted and verified code.
+- `request-ci` and `commit-queue` labels now act relying on validated commit SHAs instead of comparing dates.
+- Comprehensive audits were carried out across 140 Jenkins jobs, prioritizing frequently used ones, to detect and remediate vulnerabilities.
+- Identified vulnerable GitHub workflows were temporarily disabled, promptly patched, and re-enabled with enhanced security measures.
+
+These targeted actions significantly strengthened the security posture of our CI infrastructure, preventing the recurrence of similar potential
+intrusions and ensuring safe operations moving forward.
 
+The change we implemented now requires every pull request to be approved before running the Jenkins CI - or for collaborators to specify the individual SHA.
+
+## Timeline
+
+1. **Friday, 21 March 2025**: Report received on Hackerone. Initial triage confirmed the report as a genuine issue. The ability to start new Jenkins CI runs was restricted to prevent any further machine compromises.
+2. **Monday, 24 March 2025:** All compromised machines (totalling 24\) were identified and removed from Jenkins (pending a complete rebuild). Initial attempts to evaluate all 140 jobs defined in our Jenkins instance for vulnerability. Work started on updating the most often used vulnerable jobs to take an expected commit SHA and only proceed if the SHA of the code checked out on the machine matches.
+3. **Tuesday, 25 March 2025:** Some affected hosts rebuilt. The updated jobs failed on macOS and were investigated and updated again.
+4. **Wednesday, 26 March 2025**: More jobs updated and affected hosts rebuilt. Some GitHub workflows also identified as being vulnerable to similar attacks and disabled.
+5. **Thursday, 27 March 2025**: Validation logic in the updated jobs tweaked again to allow daily testing on non-pull request branches. Decision taken to disable all remaining jobs that had not been evaluated for the vulnerability or identified as needing the fix applied. More machines rebuilt.
+6. **Friday, 28 March 2025:** GitHub workflows were patched and `commit-queue` was re-enabled.
+7. **Tuesday, 1 April 2025:** Ability to start jobs on Jenkins and via `request-ci` was reenabled. Some lesser used jobs were still disabled.
+7. **Wednesday, 2 April 2025**: More machines rebuilt.
+8. **Thursday, 3 April 2025**: Benchmarking and libuv CI jobs updated.
+
+## Security vs. Developer Experience
+
+Over 300 volunteers maintain the Node.js project. Our processes aim to streamline CI initiation and verification of contributions across approximately 100 Jenkins runners spanning multiple operating systems and CPU architectures.
+The existing CI system design anticipates potential compromises, recognizing the need to balance security with developer convenience.
+
+## Volunteer Organization
+
+As a volunteer-driven organization, we rely on people dedicating their time to work on unglamorous tasks, such as hardening CI, handling security reports, and assembling releases. Even good-faith research against our live systems could significantly disrupt our operations. As always, we welcome all sorts of contributions, including penetration testing. We ask researchers to give us a heads up on what they are attempting to do on live systems and to keep an auditable record of their actions through our HackerOne program or by contacting the Node.js Technical Steering Committee directly ([email protected]). More on that in our [SECURITY.md](https://github.com/nodejs/node/blob/main/SECURITY.md) file.
+
+---
+
+# Node.js Test CI Security Incident – Notice
+
+On March 21st, the Node.js project received a security report regarding our development infrastructure via [our bug bounty program](https://hackerone.com/nodejs). We immediately restricted access while implementing corrective actions.
 The reported issue did not impact the Node.js runtime and there is no risk to users of Node.js. No action by Node.js users is required.
 
 The development infrastructure is expected to be available to the community by April 15 or sooner.
@@ -19,3 +81,6 @@ A full report of this incident will be available forthcoming. We appreciate the
 The current Node.js security policy can be found at [https://nodejs.org/security/](/security/). Please follow the process outlined in <https://github.com/nodejs/node/security/policy> if you wish to report a vulnerability in Node.js.
 
 Subscribe to the low-volume announcement-only nodejs-sec mailing list at <https://groups.google.com/forum/#!forum/nodejs-sec> to stay up to date on security vulnerabilities and security-related releases of Node.js and the projects maintained in the nodejs GitHub organization.
+
+[example_test_infra]: /static/images/blog/vulnerability/example_test_infra.svg
+[example_attack_test_Infra]: /static/images/blog/vulnerability/example_attack_test_infra.svg