Is there selenium / browser / javascript execution support?

[thomasmoors]

In the docs I could not find anything currently, but as many websites use frontend frameworks nowadays it might be handy to have this option also.

[ksassnowski]

This is already possible: https://roach-php.dev/docs/downloader-middleware#executing-javascript Granted, it might not be the most discoverable place in the docs 😅

[henryonsoftware]

Do we have any examples for this Javascript feature? I'm stuck when trying to fetch data from a website using Ajax to render their posts.

[ksassnowski]

Stuck how? You should only have to register the downloader middleware for your spider (assuming you have installed all the prerequisites as explained in the docs). Here’s how the middleware itself is tested: https://github.com/roach-php/core/blob/main/tests/Downloader/Middleware/ExecuteJavascriptMiddlewareTest.php

Which site are you trying to crawl, specifically? Maybe that site is doing something weird.

[henryonsoftware]

Sorry, I just found the error that I don't install puppeteer success.
After I try with this:

curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
sudo apt-get install -y nodejs gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget libgbm-dev libxshmfence-dev
sudo npm install --global --unsafe-perm puppeteer
sudo chmod -R o+rx /usr/lib/node_modules/puppeteer/.local-chromium

I can fetch now. Thanks