Skip to content

Commit 9ad38a2

Browse files
committed
fixup! Blog: add update to Security CI incident
1 parent 97a9c96 commit 9ad38a2

1 file changed

Lines changed: 17 additions & 17 deletions

File tree

apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
11
---
2-
date: '2025-04-17T16:30:00.617Z'
2+
date: '2025-04-21T16:30:00.617Z'
33
category: vulnerability
44
title: Node.js Test CI Security Incident
55
layout: blog-post
66
author: Node.js Technical Steering Committee
77
---
88

9-
# _(Update 16-April-2025)_ Node.js Test CI Security Incident – Full Disclosure
9+
# _(Update 21-April-2025)_ Node.js Test CI Security Incident – Full Disclosure
1010

11-
## **Summary**
11+
## Summary
1212

1313
On March 21, 2025, we received a [security report via HackerOne](https://hackerone.com/reports/3050534) (link restricted at time of writing), detailing a successful compromise of several Node.js test CI hosts.
1414

@@ -28,7 +28,7 @@ The core issue stems from a Time-of-Check-Time-of-Use (TOCTOU) vulnerability bet
2828

2929
![Example of attack in the Node.js test infra][example_attack_test_Infra]
3030

31-
## **Remediation**
31+
## Remediation
3232

3333
In response to this security incident, the Node.js security team took measures to mitigate risks and secure the infrastructure.
3434

@@ -42,25 +42,25 @@ In response to this security incident, the Node.js security team took measures t
4242
These targeted actions significantly strengthened the security posture of our CI infrastructure, preventing the recurrence of similar potential
4343
intrusions and ensuring safe operations moving forward.
4444

45-
## **Timeline**
45+
## Timeline
4646

47-
- **Friday, 21 March 2025**: Report received on Hackerone. Initial triage confirmed the report as a genuine issue. The ability to start new Jenkins CI runs was restricted to prevent any further machine compromises.
48-
- **Monday, 24 March 2025:** All compromised machines (totalling 24\) were identified and removed from Jenkins (pending a complete rebuild). Initial attempts to evaluate all 140 jobs defined in our Jenkins instance for vulnerability. Work started on updating the most often used vulnerable jobs to take an expected commit SHA and only proceed if the SHA of the code checked out on the machine matches.
49-
- **Tuesday, 25 March 2025:** Some affected hosts rebuilt. The updated jobs failed on macOS and were investigated and updated again.
50-
- **Wednesday, 26 March 2025**: More jobs updated and affected hosts rebuilt. Some GitHub workflows also identified as being vulnerable to similar attacks and disabled.
51-
- **Thursday, 27 March 2025**: Validation logic in the updated jobs tweaked again to allow daily testing on non-pull request branches. Decision taken to disable all remaining jobs that had not been evaluated for the vulnerability or identified as needing the fix applied. More machines rebuilt.
52-
- **Friday 28 March 2025:** Ability to start jobs on Jenkins was reenabled for Node.js collaborators. Some lesser used jobs are still disabled. GitHub workflows patched and re-enabled.
53-
- **Wednesday, 2 April 2025**: More machines rebuilt.
54-
- **Thursday, 3 April 2025**: Benchmarking and libuv CI jobs updated.
47+
1. **Friday, 21 March 2025**: Report received on Hackerone. Initial triage confirmed the report as a genuine issue. The ability to start new Jenkins CI runs was restricted to prevent any further machine compromises.
48+
2. **Monday, 24 March 2025:** All compromised machines (totalling 24\) were identified and removed from Jenkins (pending a complete rebuild). Initial attempts to evaluate all 140 jobs defined in our Jenkins instance for vulnerability. Work started on updating the most often used vulnerable jobs to take an expected commit SHA and only proceed if the SHA of the code checked out on the machine matches.
49+
3. **Tuesday, 25 March 2025:** Some affected hosts rebuilt. The updated jobs failed on macOS and were investigated and updated again.
50+
4. **Wednesday, 26 March 2025**: More jobs updated and affected hosts rebuilt. Some GitHub workflows also identified as being vulnerable to similar attacks and disabled.
51+
5. **Thursday, 27 March 2025**: Validation logic in the updated jobs tweaked again to allow daily testing on non-pull request branches. Decision taken to disable all remaining jobs that had not been evaluated for the vulnerability or identified as needing the fix applied. More machines rebuilt.
52+
6. **Friday, 28 March 2025:** Ability to start jobs on Jenkins was reenabled for Node.js collaborators. Some lesser used jobs are still disabled. GitHub workflows patched and re-enabled.
53+
7. **Wednesday, 2 April 2025**: More machines rebuilt.
54+
8. **Thursday, 3 April 2025**: Benchmarking and libuv CI jobs updated.
5555

56-
## **Security vs. Developer Experience**
56+
## Security vs. Developer Experience
5757

58-
Over 100 volunteers maintain the Node.js project. Our processes aim to streamline CI initiation and verification of contributions across approximately 100 Jenkins runners spanning multiple operating systems and CPU architectures.
58+
Over 300 volunteers maintain the Node.js project. Our processes aim to streamline CI initiation and verification of contributions across approximately 100 Jenkins runners spanning multiple operating systems and CPU architectures.
5959
The existing CI system design anticipates potential compromises, recognizing the need to balance security with developer convenience.
6060

61-
## **Volunteer Organization**
61+
## Volunteer Organization
6262

63-
As a volunteer-driven organization, such security incidents significantly disrupt our operational capabilities. We **strongly** **recommend** that security researchers **avoid** unauthorized attempts to breach our systems. Instead, please coordinate responsibly through our official HackerOne program.
63+
As a volunteer-driven organization, such security incidents significantly disrupt our operational capabilities. We **strongly recommend** that security researchers **avoid** unauthorized attempts to breach our systems. Instead, please coordinate responsibly through our official HackerOne program.
6464

6565
---
6666

0 commit comments

Comments
 (0)