[.NET main] ubuntu-latest build job killed without log output on every push since 2026-06-18

[tamirdresher]

Summary

The .NET main workflow has been failing on every push to main since 2026-06-18 (8+ consecutive failures). Always the same shape: build (windows-latest) passes, build (ubuntu-latest) fails in ~3–4 minutes with no Build step log file persisted — classic signature of the runner being killed before it can flush.

Evidence

Run	Date	Commit	windows	ubuntu	Build log captured?
28141532703	2026-06-25	d8c5674 (Squad #1394)	✅ 13m11s	❌ 3m41s	No — only Set-up/Checkout/Restore steps in the archive
28005035836	2026-06-23	typescript-apphost-backfill merge	✅	❌	No
28004682918	2026-06-23	SQLite audit suppress	✅	❌	No
28003556555	2026-06-23	Bitwarden SM #1329	✅	❌	No
27738932971	2026-06-18	XML comments #1422	✅	❌	No
27731598150	2026-06-18	Harden tests #1415	✅	❌	No
27729982356	2026-06-18	SeaweedFS #1349	✅	❌	No
27727583582	2026-06-18	SurrealDB v3 #1294	✅	✅	— last green run
27726664815	2026-06-17	dbx #1391	✅ 8m30s	❌ 4m7s	No — first failure

Likely cause

The breakpoint (#1391 dbx) lands right when the solution crossed some size threshold. Several new hosting integrations have landed since (dbx, SeaweedFS, Bitwarden, Squad), each adding 3–6 new csproj.

• GitHub-hosted ubuntu-latest runners have ~14 GB free disk; windows-latest has ~80 GB.
• A full dotnet build -c Release of the whole toolkit now produces a LOT of bin/ + obj/ output across 100+ projects.
• The "Build step has no log file at all" signature is classic runner-process killed without flushing — strongly suggests disk exhaustion (or OOM).
• The matching test job Hosting.Java.Tests-ubuntu-latest also timed out at exactly 1h on the first failure (run 27726664815) — additional symptom of the runner being unhealthy.

What dotnet-ci.yml (per-PR) does differently

PRs pass CI because dotnet-ci.yml only builds affected projects (using the path-based filter you set up), not the whole solution. dotnet-main.yml does a full dotnet restore + dotnet build on every push, which is what's blowing past the disk/memory budget.

Suggested fixes (any one should be enough)

1. Add disk cleanup step — easiest:
   - uses: jlumbroso/free-disk-space@main
   This reclaims 30+ GB on ubuntu-latest by removing Android SDK, GHC, .NET preinstalls we don't need, etc.

2. Shard the build — split dotnet build across N matrix shards by Directory.Solution.props slice.

3. Use --no-incremental + clean between project builds — slower but bounded memory.

4. Switch build (ubuntu-latest) to ubuntu-latest-large or ubuntu-22.04-large — paid runner with more disk.

Happy to send a PR for option 1 if you want — it's the lowest-risk and matches what other large .NET repos (including dotnet/aspire itself) do.

cc @aaronpowell — flagging because main is currently red and v13.5.0 can't be cut until this is fixed (release workflow runs run-tests → package → sign → publish-nuget).

[jmezach]

I am wondering if it is going to be sustainable to have a single huge pipeline that builds, tests and publishes all the integrations all at the same time. Wouldn't it make sense to have separate pipelines for each integration? I realise this will take some effort, but I feel like it is worth the effort.

[Odonno]

I am wondering if it is going to be sustainable to have a single huge pipeline that builds, tests and publishes all the integrations all at the same time. Wouldn't it make sense to have separate pipelines for each integration? I realise this will take some effort, but I feel like it is worth the effort.

Yep, probably just publish 1 artifact (nupack) per job, and then have a single job dependent of all the other jobs to publish the packages.

[aaronpowell]

There's a collection of unique problems that we have with the monorepo design that we have.

At the time of writing, we have 260 projects in the solution, so that's not even covering the TypeScript app hosts and the non-.NET sample applications. This is, simply put, a huge project - even though there really isn't that much code in there.

Because of this, it's entirely impractical to run the tests in a single agent, that why there's the matrix design for running tests (if anyone wants to know more on it check out the blog I wrote) - this radically reduces the resources required for doing test runs, which is what is most resource intensive.

We also have to do some tricks with Docker and caching - we hit rate limits with Docker a while back which we have worked around using ACR for our CI builds (#601) that caches Docker images for us so we don't hit the Docker registry directly.

There's minimal caching setup on things like NPM, NuGet, etc. that should improve the performance of setting up actions, that happens in https://github.com/CommunityToolkit/Aspire/blob/main/.github/actions/setup-runtimes-caching/action.yml and I'm pretty sure it's not very ruggedised so doesn't cache as aggressively as it should.

I made the decision when I was first setting up the CI that "push to main" (and release) would bump the version of every NuGet package, not just the one(s) that had changes in the diff since last time. This does mean that we push a lot of NuGet packages each time, often times with packages that have no changes since their last release other than dependencies. This reduces the overhead on the maintainers - don't have to be pickup on what is to be published, but at the code of a really aggressive publishing approach.

Recently we had a publishing failure because the NuGet API key was expired. I've changed the design to use the new Trusted Publishing on NuGet (you can read how I did that here), which means that we no longer use a long-lived API token, instead CI requests one on demand. This is a lot more secure and means that we don't have to worry about expiring tokens.

Lastly, I added some auto retry logic into the pipeline to handle failing tests. Because of the volume of runners we spin up to do the tests (it's like, 120+ now) I think we might hit some caps that cause tests to just randomly fail at an infra level, Docker not starting is a common one. The retry logic is pretty solid but if we're hitting cases where it should auto-rerun, let's tweak the rerun workflow to cover more cases.

Hopefully that gives some overall context of the problem we face with this project, and the things that have been added along the way to attempt to address the issues. The most common failures we see on the main branch now seem to be from a handful of integrations that "randomly fail" and if re-run, the tests pass. The problem is, they appear as tests themselves that are failing, not infrastructure around the tests, which makes it hard to identify the failures separately from legitimate test failures. Maybe we should have just a default "retry N times on failure" rather than attempting to do some dynamic detection of failure conditions?

I also know that most of these solutions have just come from myself doing things. Generally that's just been because I'll see a problem and be like "well, I'll throw Copilot at it" (or, back in the day, just code up a solution 🫨), but if people have ideas, raise them for discussion or prototype them out, and we can look at their viability. Things like larger runners (paid) runners are a harder thing to tackle because someone has to pay for them, but if it's the only possible solution, then it's something I'll have to look into.

[jmezach]

@aaronpowell Thanks for the extensive write-up and explanation of the reasoning behind most of these choices.

One thing that stands out to me is the decision that "push to main" (and release) bumps the version of every NuGet package. While I realise that makes things easier from the perspective of not having to figure out what has actually changed it does mean we push a lot of packages with no actual changes which could be confusing from the users perspective.

I wonder if we could take some inspiration from the Backstage Community Plugins repository (https://github.com/backstage/community-plugins/). They've created workspaces, which are folders with a bunch of files that are logically related. Each workspace has its own isolated release, so only those with updates are pushed.

I realise this is an entirely different ecosystem (C# and NuGet vs. TypeScript and npm), but I do think it works fairly well there. It avoids the problem of pushing too many packages, while also keeping CI relatively small.