fix(ci): fix the monorepo by nikitachapovskii-dev · Pull Request #299 · apify/actor-scraper

nikitachapovskii-dev · 2026-06-11T15:28:15Z

Reviving the generic scrapers after the split + pnpm migration

After thee migration to pnpm and SDK split, CI went red and the actor builds stopped matching the platform. This PR gets everything green again and, more importantly, makes the platform builds reproducible.

CI

The e2e workflow simply never installed the browsers, on a warm pnpm store the side-effects cache skips the puppeteer/playwright postinstall, so Chrome wasn't there and every browser suite died with Could not find Chrome. Added the install step the PR workflow already had. Both workflows now run on Node 24 with timeout-minutes, so if something hangs it fails fast instead of eating six hours in silence.

Deterministic builds from the monorepo root

We're a pnpm monorepo: one root pnpm-lock.yaml and a shared @apify/scraper-tools. For reproducible images the build has to install from that lockfile with --frozen-lockfile, and it has to see the local workspace package, not the published one. So every actor builds from the repo root via a Git source (dockerContextDir: ../../../..) and runs pnpm install --frozen-lockfile + pnpm --filter <actor>... deploy. apify push can't do this, it uploads only the actor folder, without the root lockfile or the local scraper-tools.

The release workflow drops push-actor-action and instead triggers each Actor's Git-source build through scripts/trigger-apify-build.mjs (checks the version is a GIT_REPO source with the right tag, fires the build, waits, fails on anything but success). This assumes each Actor on the platform has its source pointed at this repo (branch + :packages/actor-scraper/<actor>).

Browsers come from the base image, not from us

The browser binary lives in the Apify base image, and the JS driver has to match it. Pinning our own driver to chase that match is brittle, they version separately and drift, which is exactly what kept crashing. So the browser actors just use the base image's driver: we drop it from the deploy bundle and let it resolve from the base at runtime (the app sits in /home/myuser/app). Binary and driver now come from the same image and can't get out of sync, and the bases are pinned by digest.

camoufox is the one exception worth flagging: it's stuck on 24-1.59.1. The newer 24-1.60.0 (Camoufox 150) crashes the whole process on any page that throws a JS error, that's upstream, it reproduces on the untouched base image. 24-1.59.1 is the newest one that survives.

sitemap-extractor (Crawlee/Apify v4 beta)

Added to the release matrix. It rides the v4 beta while the other six stay on v3, so @apify/scraper-tools accepts ^3 || ^4, a tiny type shim keeps its v3 types from clashing in the v4 build, and a couple of pnpm overrides pin the beta's transitive @crawlee/*. All of that comes back out once we're fully on stable v4.

Cleanup

Manifests still pointed at apify-ts/apify-sdk-js, CONTRIBUTING said npm, a couple of READMEs were stale — fixed. Everything builds on Node 24.

Follow-ups (separate)

Unpin camoufox once a newer base survives page errors. And drop the v4 bridge bits (the ^3 peer half, the shim, the overrides) when everything's on stable v4.

P.S. sorry, the description is huge, I know 😬

nikitachapovskii-dev · 2026-06-15T14:43:45Z

https://github.com/apify/actor-scraper/actions/runs/27552963712/job/81444068729

ruocco-l

🚀 Thank you for figuring this out

…nistic pnpm

…t-pinned bases

… quote $GITHUB_ENV

nikitachapovskii-dev · 2026-06-25T07:38:01Z

build: https://github.com/apify/actor-scraper/actions/runs/28110720337

runs on each actor:
https://console.apify.com/view/runs/sFNJHaSjbnj7YAdgc
https://console.apify.com/view/runs/2IGk9tWak1fYu02hK
https://console.apify.com/view/runs/7Br3EqUELDv5uSgMY
https://console.apify.com/view/runs/k3eDWKIeMBAqhgpEz
https://console.apify.com/view/runs/WfCS3MsPi5a93Yj2C
https://console.apify.com/view/runs/jcuXqtqXk72fJmKEv
https://console.apify.com/view/runs/ECN1gZSsZvFhfTNRW

e2e: https://github.com/apify/actor-scraper/actions/runs/28112939851/job/83244863889

nikitachapovskii-dev · 2026-06-25T07:40:14Z

+RUN rm -rf \
+    /deploy/node_modules/playwright \
+    /deploy/node_modules/playwright-core \
+    /deploy/node_modules/camoufox-js \
+    /deploy/node_modules/better-sqlite3 \
+    /deploy/node_modules/.bin/playwright \
+    /deploy/node_modules/.bin/playwright-core \
+    /deploy/node_modules/.bin/camoufox-js


same pattern for playwright, puppeteer and web scrapers

We install the browser driver and then delete it and run the app from a sub-dir (COPY /deploy ./app + WORKDIR /home/myuser/app). Done like that because the browser binary lives in the base image and the JS driver must match it exactly. Rather than pin our own driver to match (fragile, they're versioned independently and drift, which caused the runtime crashes), we drop the driver from the bundle so the runtime resolves it from the base image's node_modules (resolution walks up from /home/myuser/app). Driver and binary then come from the same image and can't drift. We still install it in the build stage because tsc needs the types.

nikitachapovskii-dev · 2026-06-25T07:42:38Z

+# digest. The actor does NOT ship its own browser drivers — they're dropped from the deploy bundle
+# and resolved from the base at runtime, so driver and binary can never drift. Tag 24-1.59.1 is the
+# newest Camoufox base that survives page JS errors (24-1.60.0 crashes the process on them).
+FROM apify/actor-node-playwright-camoufox:24-1.59.1@sha256:eadc96fa9492284eb45ef70b6b91c841fae7f142d25a22ae2887a21bb78b3469 AS builder


Same for each browser. Base is pinned by sha256 digest, not just a tag, on purpose: tags are mutable, so a bare tag wouldn't give a reproducible browser binary. Since the whole runtime browser stack comes from the base (see the driver note), the digest is what actually makes the browser deterministic. Renovate bumps the digest.

Also, not the newest base. I've tested the latest camoufox base (24-1.60.0, Camoufox 150): it crashes the whole process on any page that emits a JS error (playwright's Firefox pageError handler). I also tried the latest Playwright (1.61.x), which won't even launch this Camoufox binary (Browser.setDefaultViewport protocol error). 24-1.59.1 is the newest camoufox base that survives page errors.

nikitachapovskii-dev · 2026-06-25T07:46:54Z

+        "@crawlee/browser-pool": "^3.8.2 || ^4.0.0-beta.0",
+        "@crawlee/core": "^3.8.2 || ^4.0.0-beta.0",
+        "@crawlee/types": "^3.8.2 || ^4.0.0-beta.0",
+        "@crawlee/utils": "^3.8.2 || ^4.0.0-beta.0",
+        "apify": "^3.1.8 || ^4.0.0-beta.0"


This package is shared by the crawlee-v3 generic actors and the v4-beta sitemap-extractor. We will make it 1 ver when we move all actors to crawlee v4

Also, after moving to v4 we will delete version-neutral slice: packages/actor-scraper/sitemap-scraper/src/scraper-tools-compat.d.ts

nikitachapovskii-dev · 2026-06-25T07:53:57Z

@@ -0,0 +1 @@
+inject-workspace-packages=false


it's an explicit guard: a normal pnpm install must symlink workspace packages (fast local dev); only the Docker builds opt in per-command with --config.inject-workspace-packages=true on pnpm deploy, so /deploy is self-contained.

Can you please expand on this? It seems this option is false by default in pnpm anyway https://pnpm.io/workspaces#injectworkspacepackages

(also, it seems this option should be camelCased and in pnpm-workspace.yaml instead)

nikitachapovskii-dev · 2026-06-25T07:55:48Z

+overrides:
+  "[email protected]>@crawlee/core": "4.0.0-beta.25"
+  "[email protected]>@crawlee/types": "4.0.0-beta.25"
+  "[email protected]>@crawlee/utils": "4.0.0-beta.25"


Remove once on stable crawlee v4

nikitachapovskii-dev · 2026-06-25T07:58:19Z

+    "dockerContextDir": "../../../..",
+    "dockerfile": "../Dockerfile",


same in every actor

points the Docker build context at the monorepo root so the build can use the root pnpm-lock.yaml for a frozen, deterministic install. apify push of a single folder can't do this (no root lockfile in the upload). that's why builds run from the Git source.

inspired by actor-monorepo-example

nikitachapovskii-dev · 2026-06-25T07:59:37Z

@@ -0,0 +1,80 @@
+import { ApifyClient } from 'apify-client';


It triggers a build of the Actor's configured Git source and waits for the result. For exact-commit reproducibility, point the source at a release tag.

…HA leftovers

barjin

Thank you @nikitachapovskii-dev !

The changes look mostly fine to me. 👍 I wrote down few ideas I had looking at the code. Please look into these at your own discretion and feel free to merge (no need to wait for me, @ruocco-l 's opinion has more weight here :)).

barjin · 2026-06-26T08:55:21Z

    build_and_test:
        name: Build & Test
        runs-on: ubuntu-22.04
+        timeout-minutes: 60


I believe this won't fix the timeouting e2e tests. Likely caused by microsoft/playwright#40724 (bumping Playwright or reverting Node version might help)

it's not the fix 😄, it's just for the future (if something happens then fail is better than getting stuck) The actual fix is browser-install step

barjin · 2026-06-26T08:57:34Z

+                // camoufox-js launch options intentionally diverge from Playwright's, and the pinned
+                // playwright-core type identity differs from @crawlee/playwright's — cast through unknown.


Huh, this sounds curious - can you please investigate this and make an issue in crawlee or camoufox-js?

I don't think we changed anything recently, at least not intentionally.

barjin · 2026-06-26T08:59:00Z

 .vscode
 yarn.lock
 .yarn
+# npm locks are generated locally in actor dirs (the monorepo uses pnpm-lock.yaml)


But this addition is just a comment, right? What is it commenting on?

Aha, that's a leftover from when I tried to make it via apify push. Thanks for noticing

barjin · 2026-06-26T09:11:20Z

@@ -0,0 +1 @@
+inject-workspace-packages=false


Can you please expand on this? It seems this option is false by default in pnpm anyway https://pnpm.io/workspaces#injectworkspacepackages

(also, it seems this option should be camelCased and in pnpm-workspace.yaml instead)

barjin · 2026-06-26T09:29:53Z

 const { SESSION_MAX_USAGE_COUNTS, DEFAULT_VIEWPORT, DEVTOOLS_TIMEOUT_SECS, META_KEY } = scraperToolsConstants;
 const SCHEMA = JSON.parse(await readFile(new URL('../../INPUT_SCHEMA.json', import.meta.url), 'utf8'));

+type CrawleePuppeteerPage = PuppeteerCrawlingContext['page'];


Why is this change necessary? PuppeteerCrawlingContext['page'] should be the exactly the Page type from puppeteer. A mismatch here might point to some issues with package version resolution / dual installs etc.

You are right! I was lost fighting with browsers and didn't even check if it was a dual install here :/

barjin · 2026-06-26T09:37:01Z

+ENV APIFY_DISABLE_OUTDATED_WARNING=1

-CMD npm run start:prod --silent
+CMD ["node", "dist/main.js"]


nit: we should either call (p)npm run start:prod here or drop the start:prod script from the package.json files. Keeping the script but not using it will eventually lead to a frustrating debugging experience 😄

…ing them

fix(ci): unblock e2e and PR workflows

e692213

nikitachapovskii-dev marked this pull request as draft June 11, 2026 15:28

feat(ci): add sitemap-extractor to the release workflow

250178a

nikitachapovskii-dev changed the title ~~fix(ci): unblock e2e and PR workflows~~ fix(ci): fix the repo Jun 15, 2026

nikitachapovskii-dev added 2 commits June 15, 2026 15:37

chore: post-split fixups (crawling depth, repo metadata, docs)

9accd17

fix(docker): ignore local npm lock so isolated builds match the platform

c3f6fce

nikitachapovskii-dev self-assigned this Jun 16, 2026

nikitachapovskii-dev added 2 commits June 16, 2026 11:13

chore(scraper-tools): cover crawling depth in enqueueRequest

719da43

fix(ci): keep stable as the default release channel

bbb7464

nikitachapovskii-dev marked this pull request as ready for review June 16, 2026 12:04

nikitachapovskii-dev requested a review from ruocco-l June 16, 2026 12:05

revert: undo enqueueRequest depth fix

968ca46

nikitachapovskii-dev changed the title ~~fix(ci): fix the repo~~ fix(ci): fix the monorepo Jun 16, 2026

ruocco-l approved these changes Jun 16, 2026

View reviewed changes

fix(ci): unpin node, update puppeteer & playwright to latest, determi…

027ea86

…nistic pnpm

nikitachapovskii-dev marked this pull request as draft June 23, 2026 12:43

fix(actors): deterministic monorepo builds + browser stack from diges…

c1046bd

…t-pinned bases

nikitachapovskii-dev force-pushed the fix/revive-actor-scraper branch from 57d00a2 to c1046bd Compare June 24, 2026 16:02

ci: clean up release workflow, drop non-schema top-level description,…

21d66fc

… quote $GITHUB_ENV

nikitachapovskii-dev commented Jun 25, 2026

View reviewed changes

nikitachapovskii-dev marked this pull request as ready for review June 25, 2026 12:38

nikitachapovskii-dev requested review from barjin and ruocco-l June 25, 2026 12:38

chore(ci): rename release build script to trigger-apify-build, drop S…

cc0d075

…HA leftovers

barjin reviewed Jun 26, 2026

View reviewed changes

nikitachapovskii-dev added 3 commits June 26, 2026 12:58

fix(deps): dedupe puppeteer to a single 25.2.0, revert type workarounds

8e88775

refactor(docker): pin browser drivers to base image instead of stripp…

9d914a0

…ing them

chore: drop the start:prod script from the package.json

5a802cb

ruocco-l approved these changes Jun 29, 2026

View reviewed changes

nikitachapovskii-dev merged commit 21b714a into master Jun 29, 2026
12 checks passed

nikitachapovskii-dev deleted the fix/revive-actor-scraper branch June 29, 2026 11:55

		"dockerContextDir": "../../../..",
		"dockerfile": "../Dockerfile",

		// camoufox-js launch options intentionally diverge from Playwright's, and the pinned
		// playwright-core type identity differs from @crawlee/playwright's — cast through unknown.

		@@ -0,0 +1 @@
		inject-workspace-packages=false

		@@ -0,0 +1 @@
		inject-workspace-packages=false

Uh oh!

Conversation

nikitachapovskii-dev commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviving the generic scrapers after the split + pnpm migration

CI

Deterministic builds from the monorepo root

Browsers come from the base image, not from us

sitemap-extractor (Crawlee/Apify v4 beta)

Cleanup

Follow-ups (separate)

Uh oh!

nikitachapovskii-dev commented Jun 15, 2026

Uh oh!

ruocco-l left a comment

Choose a reason for hiding this comment

Uh oh!

nikitachapovskii-dev commented Jun 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

barjin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nikitachapovskii-dev Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nikitachapovskii-dev commented Jun 11, 2026 •

edited

Loading

nikitachapovskii-dev Jun 26, 2026 •

edited

Loading