Skip to content

chore: migrate to pnpm and enable minimum release age #3568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
B4nan wants to merge 23 commits into from
Closed

chore: migrate to pnpm and enable minimum release age #3568

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
221fd91
chore: migrate to pnpm and enable minimum release age (WIP)
B4nan Apr 14, 2026
afdb318
chore: fix crawlee pnpm build — minimatch overrides + non-incremental…
B4nan Apr 15, 2026
236665c
chore: retrigger CI
B4nan Apr 15, 2026
fd39f22
ci: approve build scripts blocked by pnpm 10
B4nan Apr 15, 2026
9a8da4d
ci: remove --loglevel error to surface install failures
B4nan Apr 15, 2026
5663e82
ci: don't fail install on stale patch-package errors
B4nan Apr 15, 2026
9107a74
ci: remove website/docs from workspace (docusaurus needs Node 20+)
B4nan Apr 15, 2026
1af634d
ci: use shared apify/workflows/pnpm-install action
B4nan Apr 15, 2026
d50bbe3
ci: drop public-hoist-pattern to reduce eslint plugin aliasing
B4nan Apr 15, 2026
a4d6199
ci: pin @apify/eslint-config to 2.0.0 + clean inquirer import
B4nan Apr 15, 2026
74d8679
chore: whitelist internal apify-client/apify/crawlee/got-scraping fro…
B4nan Apr 15, 2026
9060b35
ci: disable import/export and related rules in crawlee monorepo
B4nan Apr 15, 2026
501b138
ci: drop Node 18 from test matrix (vitest/rolldown require Node 20+)
B4nan Apr 15, 2026
f636faa
ci: re-add docs and website to workspace, clean nested READMEs
B4nan Apr 15, 2026
928a132
ci: explicitly install puppeteer Chrome
B4nan Apr 15, 2026
ebc3768
test: drop unused callback arg from dom_crawler afterAll
B4nan Apr 15, 2026
0313236
chore: move pnpm config from .npmrc to pnpm-workspace.yaml
B4nan Apr 15, 2026
2468161
chore: enforce pnpm via devEngines
B4nan Apr 15, 2026
dcc3307
chore: normalize .gitignore — add yarn.lock and .yarn, drop .pnpm-store
B4nan Apr 16, 2026
f038ab3
fix: use workspace:^ for peerDeps, pin peerDeps in copy script
B4nan Apr 16, 2026
7522e61
chore: sync renovate allowlist with pnpm-workspace exclusions
B4nan Apr 16, 2026
ec96e5c
chore: replace devEngines with only-allow preinstall
B4nan Apr 16, 2026
ba448e5
ci: add volta pnpm entry + rebase before release push
B4nan Apr 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 3 additions & 12 deletions .github/workflows/docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,27 +19,18 @@ jobs:
steps:
- uses: actions/checkout@v6

- name: Use Node.js 20
- name: Use Node.js 24
uses: actions/setup-node@v6
with:
node-version: 24
package-manager-cache: false

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate
- uses: apify/workflows/pnpm-install@main

- name: Build & deploy docs
run: |
# install project deps
yarn
# go to website dir
cd website
# install website deps
yarn
# build the docs
yarn build
pnpm build
env:
APIFY_SIGNING_TOKEN: ${{ secrets.APIFY_SIGNING_TOKEN }}
SEGMENT_TOKEN: ${{ secrets.SEGMENT_TOKEN }}
Expand Down
21 changes: 5 additions & 16 deletions .github/workflows/publish-to-npm.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,16 +49,6 @@ jobs:
node-version: 24
package-manager-cache: false

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate

- name: Activate cache for Node.js 24
uses: actions/setup-node@v6
with:
cache: 'yarn'

- name: Turbo cache
id: turbo-cache
uses: actions/cache@v5
Expand All @@ -68,16 +58,15 @@ jobs:
restore-keys: |
turbo-${{ github.job }}-${{ github.ref_name }}-

- name: Install dependencies
run: yarn
- uses: apify/workflows/pnpm-install@main

- name: Build packages
run: yarn ci:build
run: pnpm ci:build

- name: Bump canary versions
if: inputs.dist-tag == 'next'
run: |
yarn turbo copy --force -- --canary --preid=beta
pnpm turbo copy --force -- --canary --preid=beta

- name: Commit changes
if: inputs.dist-tag == 'next'
Expand All @@ -96,7 +85,7 @@ jobs:
timeout_minutes: 10
max_attempts: 5
retry_wait_seconds: 30
command: git checkout . && yarn publish:prod --yes
command: git checkout . && pnpm publish:prod -- --yes
env:
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}

Expand All @@ -107,6 +96,6 @@ jobs:
timeout_minutes: 10
max_attempts: 5
retry_wait_seconds: 30
command: git checkout . && yarn publish:next --yes
command: git checkout . && pnpm publish:next -- --yes
env:
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
73 changes: 18 additions & 55 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
name: Release @latest

env:
YARN_IGNORE_NODE: 1
RETRY_TESTS: 1

on:
Expand Down Expand Up @@ -51,16 +50,6 @@ jobs:
node-version: ${{ matrix.node-version }}
package-manager-cache: false

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate

- name: Activate cache for Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v6
with:
cache: 'yarn'

- name: Turbo cache
id: turbo-cache
uses: actions/cache@v5
Expand All @@ -70,16 +59,16 @@ jobs:
restore-keys: |
turbo-${{ github.job }}-${{ github.ref_name }}-

- name: Install Dependencies
run: |
yarn
yarn playwright install --with-deps
- uses: apify/workflows/pnpm-install@main

- name: Install Playwright browsers
run: pnpm exec playwright install --with-deps

- name: Build
run: yarn ci:build
run: pnpm ci:build

- name: Tests
run: yarn test
run: pnpm test

release:
name: "Bump Crawlee: ${{ inputs.version }} version (${{ inputs.custom_version || 'n/a' }} custom version)"
Expand All @@ -94,22 +83,12 @@ jobs:
token: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
fetch-depth: 0

- name: Use Node.js 20
- name: Use Node.js 24
uses: actions/setup-node@v6
with:
node-version: 24
package-manager-cache: false

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate

- name: Activate cache for Node.js 20
uses: actions/setup-node@v6
with:
cache: 'yarn'

- name: Turbo cache
id: turbo-cache
uses: actions/cache@v5
Expand All @@ -119,15 +98,14 @@ jobs:
restore-keys: |
turbo-${{ github.job }}-${{ github.ref_name }}-

- name: Install Dependencies
run: yarn
- uses: apify/workflows/pnpm-install@main

- name: Build
run: yarn ci:build
run: pnpm ci:build

- name: Bump version to custom version
if: ${{ github.event.inputs.version == 'custom' && github.event.inputs.custom_version != '' }}
run: yarn lerna version ${{ github.event.inputs.custom_version }} --force-publish --yes
run: pnpm exec lerna version ${{ github.event.inputs.custom_version }} --force-publish --yes
env:
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
GIT_AUTHOR_NAME: Apify Release Bot
Expand All @@ -137,7 +115,7 @@ jobs:

- name: Bump version to ${{ github.event.inputs.version }} version
if: ${{ github.event.inputs.version != 'custom' }}
run: yarn lerna version ${{ github.event.inputs.version }} --force-publish --yes
run: pnpm exec lerna version ${{ github.event.inputs.version }} --force-publish --yes
env:
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
GIT_AUTHOR_NAME: Apify Release Bot
Expand All @@ -147,8 +125,8 @@ jobs:

- name: Pin versions in internal dependencies and update lockfile
run: |
yarn release:pin-versions
yarn install --no-immutable
pnpm release:pin-versions
pnpm install --no-frozen-lockfile

- name: Commit changes
id: commit
Expand All @@ -157,6 +135,7 @@ jobs:
author_name: Apify Release Bot
author_email: [email protected]
message: 'chore(release): update internal dependencies [skip ci]'
pull: '--rebase --autostash'

- name: Publish packages
uses: apify/workflows/execute-workflow@main
Expand Down Expand Up @@ -188,7 +167,7 @@ jobs:
token: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
fetch-depth: 0

- name: Use Node.js 20
- name: Use Node.js 24
uses: actions/setup-node@v6
with:
node-version: 24
Expand All @@ -197,23 +176,7 @@ jobs:
- name: Install jq
run: sudo apt-get install jq

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate

- name: Activate cache for Node.js 20
uses: actions/setup-node@v6
with:
cache: 'yarn'

- name: Install dependencies
run: |
# install project deps
yarn
# install website deps
cd website
yarn
- uses: apify/workflows/pnpm-install@main

- name: Snapshot the current version
run: |
Expand All @@ -224,8 +187,8 @@ jobs:
exit 1
fi
MAJOR_MINOR=$(echo $VERSION | cut -d. -f1,2)
yarn docusaurus docs:version $MAJOR_MINOR
yarn docusaurus api:version $MAJOR_MINOR
pnpm docusaurus docs:version $MAJOR_MINOR
pnpm docusaurus api:version $MAJOR_MINOR

- name: Commit and push the version snapshot
run: |
Expand Down
Loading
Loading