You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md
+17-17Lines changed: 17 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,14 +1,14 @@
1
1
---
2
-
date: '2025-04-17T16:30:00.617Z'
2
+
date: '2025-04-21T16:30:00.617Z'
3
3
category: vulnerability
4
4
title: Node.js Test CI Security Incident
5
5
layout: blog-post
6
6
author: Node.js Technical Steering Committee
7
7
---
8
8
9
-
# _(Update 16-April-2025)_ Node.js Test CI Security Incident – Full Disclosure
9
+
# _(Update 21-April-2025)_ Node.js Test CI Security Incident – Full Disclosure
10
10
11
-
## **Summary**
11
+
## Summary
12
12
13
13
On March 21, 2025, we received a [security report via HackerOne](https://hackerone.com/reports/3050534) (link restricted at time of writing), detailing a successful compromise of several Node.js test CI hosts.
14
14
@@ -28,7 +28,7 @@ The core issue stems from a Time-of-Check-Time-of-Use (TOCTOU) vulnerability bet
28
28
29
29
![Example of attack in the Node.js test infra][example_attack_test_Infra]
30
30
31
-
## **Remediation**
31
+
## Remediation
32
32
33
33
In response to this security incident, the Node.js security team took measures to mitigate risks and secure the infrastructure.
34
34
@@ -42,25 +42,25 @@ In response to this security incident, the Node.js security team took measures t
42
42
These targeted actions significantly strengthened the security posture of our CI infrastructure, preventing the recurrence of similar potential
43
43
intrusions and ensuring safe operations moving forward.
44
44
45
-
## **Timeline**
45
+
## Timeline
46
46
47
-
-**Friday, 21 March 2025**: Report received on Hackerone. Initial triage confirmed the report as a genuine issue. The ability to start new Jenkins CI runs was restricted to prevent any further machine compromises.
48
-
-**Monday, 24 March 2025:** All compromised machines (totalling 24\) were identified and removed from Jenkins (pending a complete rebuild). Initial attempts to evaluate all 140 jobs defined in our Jenkins instance for vulnerability. Work started on updating the most often used vulnerable jobs to take an expected commit SHA and only proceed if the SHA of the code checked out on the machine matches.
49
-
-**Tuesday, 25 March 2025:** Some affected hosts rebuilt. The updated jobs failed on macOS and were investigated and updated again.
50
-
-**Wednesday, 26 March 2025**: More jobs updated and affected hosts rebuilt. Some GitHub workflows also identified as being vulnerable to similar attacks and disabled.
51
-
-**Thursday, 27 March 2025**: Validation logic in the updated jobs tweaked again to allow daily testing on non-pull request branches. Decision taken to disable all remaining jobs that had not been evaluated for the vulnerability or identified as needing the fix applied. More machines rebuilt.
52
-
-**Friday 28 March 2025:** Ability to start jobs on Jenkins was reenabled for Node.js collaborators. Some lesser used jobs are still disabled. GitHub workflows patched and re-enabled.
53
-
-**Wednesday, 2 April 2025**: More machines rebuilt.
54
-
-**Thursday, 3 April 2025**: Benchmarking and libuv CI jobs updated.
47
+
1.**Friday, 21 March 2025**: Report received on Hackerone. Initial triage confirmed the report as a genuine issue. The ability to start new Jenkins CI runs was restricted to prevent any further machine compromises.
48
+
2.**Monday, 24 March 2025:** All compromised machines (totalling 24\) were identified and removed from Jenkins (pending a complete rebuild). Initial attempts to evaluate all 140 jobs defined in our Jenkins instance for vulnerability. Work started on updating the most often used vulnerable jobs to take an expected commit SHA and only proceed if the SHA of the code checked out on the machine matches.
49
+
3.**Tuesday, 25 March 2025:** Some affected hosts rebuilt. The updated jobs failed on macOS and were investigated and updated again.
50
+
4.**Wednesday, 26 March 2025**: More jobs updated and affected hosts rebuilt. Some GitHub workflows also identified as being vulnerable to similar attacks and disabled.
51
+
5.**Thursday, 27 March 2025**: Validation logic in the updated jobs tweaked again to allow daily testing on non-pull request branches. Decision taken to disable all remaining jobs that had not been evaluated for the vulnerability or identified as needing the fix applied. More machines rebuilt.
52
+
6.**Friday, 28 March 2025:** Ability to start jobs on Jenkins was reenabled for Node.js collaborators. Some lesser used jobs are still disabled. GitHub workflows patched and re-enabled.
53
+
7.**Wednesday, 2 April 2025**: More machines rebuilt.
54
+
8.**Thursday, 3 April 2025**: Benchmarking and libuv CI jobs updated.
55
55
56
-
## **Security vs. Developer Experience**
56
+
## Security vs. Developer Experience
57
57
58
-
Over 100 volunteers maintain the Node.js project. Our processes aim to streamline CI initiation and verification of contributions across approximately 100 Jenkins runners spanning multiple operating systems and CPU architectures.
58
+
Over 300 volunteers maintain the Node.js project. Our processes aim to streamline CI initiation and verification of contributions across approximately 100 Jenkins runners spanning multiple operating systems and CPU architectures.
59
59
The existing CI system design anticipates potential compromises, recognizing the need to balance security with developer convenience.
60
60
61
-
## **Volunteer Organization**
61
+
## Volunteer Organization
62
62
63
-
As a volunteer-driven organization, such security incidents significantly disrupt our operational capabilities. We **strongly****recommend** that security researchers **avoid** unauthorized attempts to breach our systems. Instead, please coordinate responsibly through our official HackerOne program.
63
+
As a volunteer-driven organization, such security incidents significantly disrupt our operational capabilities. We **stronglyrecommend** that security researchers **avoid** unauthorized attempts to breach our systems. Instead, please coordinate responsibly through our official HackerOne program.
0 commit comments