Google/Bing/DuckDuckGo/Qwant were tested as a way to find prices on the
3 remaining requested merchants and all four block scraping immediately
(visible captcha or JS-only anti-bot challenge on the very first
automated request) - not usable as an intermediary. Instead, a headless
browser was used to scrape these 3 merchants directly, since their
blocking (JS-only search, empty body, 403) turned out to be bypassable
with real browser rendering rather than needing to get past a captcha:
- Materiel.net: search is entirely client-side (Stimulus/Turbo), a plain
GET on /recherche/?query=... always returns the empty "no results"
template. Fixed by actually filling the search input and clicking
submit in a real page.
- Cdiscount: returns a fully empty response body to plain HTTP fetches
(no error, just nothing) but renders fine in a real browser.
- PCComponentes: blocks plain HTTP requests with a 403, but a headless
browser with a normal user agent goes through.
Implementation:
- New playwright-core dependency, pointed at a system-installed
Chromium binary via CHROMIUM_EXECUTABLE_PATH (set to /usr/bin/chromium
here) instead of downloading Playwright's own ~300MB browser
- New headless_browser_service.ts: lazily launches a single shared
Chromium instance reused across lookups (avoids paying browser launch
cost per request), with a withPage() helper that always closes its
browser context afterwards. A new start/browser.ts preload registers
an app.terminating() hook so the browser process is closed on shutdown
instead of leaking a child process.
- fetchMaterielNetPrice/fetchCdiscountPrice/fetchPcComponentesPrice
follow the same model-token matching strategy as the existing
cheerio-based merchants (LDLC/Grosbill/TopAchat): search on a short
brand+series query, then pick the result whose label/attribute
contains the extracted model number, to avoid misreporting an
unrelated product's price
- PCComponentes exposes clean data-product-price/data-product-category
attributes directly on result links, so full pre-built "Sobremesa" PCs
that happen to mention the searched CPU are excluded rather than
reported as its price
- Fixed two bugs found while testing with a Ryzen 5 9600X (a case with
multiple search results, unlike the Ryzen 3 3200G single-redirect case
used earlier):
- Materiel.net: a search results list page also has .o-product__price
elements (one per card), so checking for their mere presence isn't
enough to detect the "redirected to a single product" case; now
disambiguated by URL pattern (/produit/ vs /recherche/)
- New parseFlattenedFragmentedPriceText(): Playwright's innerText()
flattens "<whole>€<sup>cents</sup>" prices to plain strings like
"284€95" or "2 049€95", which the existing parser misread as
28495/204995. Cheerio-based scrapers (LDLC/Grosbill) already handled
this correctly via DOM structure (parseFragmentedPrice); this adds
the equivalent for already-flattened text.
- findCheapestOffers() now queries all 7 merchants in parallel
- Extended the MerchantOffer type/favicon domain map on the frontend for
the 3 new merchants
All 30 existing tests still pass unchanged (they don't exercise live
network scraping, only the surrounding auth/validation/caching logic).
Idealo and PCPartPicker were evaluated and ruled out:
- idealo.fr search pages return 503 (Cloudflare)
- the internal Idealo GraphQL endpoint used by a public scraper
(app.idealo.de/app-backend/api) is protected by Akamai Bot Manager and
silently returns empty data even with mobile app headers
- PCPartPicker's product list/search relies on a session-bound CSRF token
refreshed per page load, against a scraping-hostile ToS
pc-builder.io category pages are server-rendered, include name/price/image
for ~100 products per category, and have no bot protection, so the feature
is built on top of that source only.
- New scraped_parts cache table (type_id, name, price, image_url, source,
source_url, source_ref unique per source) populated by a batch job
rather than scraped live per search, since only ~100 first results per
category are available without reverse-engineering pc-builder.io's
DataTables/CSRF pagination flow
- part_scraper_service.ts: maps our ComponentType names to pc-builder.io
category slugs (SSD -> storage, since pc-builder.io has no separate HDD
category) and parses each category page with cheerio, skipping rows
without a resolved price (generic 'search on Amazon' links)
- node ace scrape:parts: refreshes the cache for every mapped category
(upsert on source+source_ref), runnable manually or later via cron
- GET /scraped-parts (authenticated): search the cache by name (LOWER()
LIKE, avoids a MySQL utf8_bin/utf8mb4 collation mismatch from Lucid's
whereLike) and optional typeId filter
- NewPartSearch.vue: debounced search widget wired into
components/create.vue; clicking 'Import' pre-fills the existing
component creation form (name, price, type, purchase date = today)
instead of creating the Component directly, so the user still reviews
and saves it like any manually entered part
- 3 functional tests covering search by name, filter by typeId, and
unauthenticated access being redirected to login
- Initial scrape run: 989 parts across 10 categories
- Add @japa/api-client devDependency (was missing despite being required
for the 'client' fixture used in functional tests)
- Wire up sessionApiClient/authApiClient/shieldApiClient plugins in
tests/bootstrap.ts so tests can authenticate (loginAs) and pass CSRF
validation (withCsrfToken)
- Wrap each functional test in a global DB transaction (withGlobalTransaction)
for isolation and rollback
- Add .env.test with SESSION_DRIVER=memory: the cookie session driver used
in dev/prod is incompatible with Japa's session test client, which needs
a store it can read/write by session id
- tests/functional/computers.spec.ts: covers owner can update; a user
cannot update/delete another user's computer (404); a user cannot
attach another user's component via componentsId (404); two components
of the same unique type cannot be attached (422); deleting a computer
detaches its components instead of destroying them
- tests/functional/components.spec.ts: covers owner can update; a user
cannot update/delete another user's component (404); an unauthenticated
request cannot delete a component (redirects to /auth/login)
All 10 tests pass, exercising the security fixes from commit 3cc189c.
Kevin L'Hereec