Kevin L'Hereec
a7788e593e
Google/Bing/DuckDuckGo/Qwant were tested as a way to find prices on the
3 remaining requested merchants and all four block scraping immediately
(visible captcha or JS-only anti-bot challenge on the very first
automated request) - not usable as an intermediary. Instead, a headless
browser was used to scrape these 3 merchants directly, since their
blocking (JS-only search, empty body, 403) turned out to be bypassable
with real browser rendering rather than needing to get past a captcha:
- Materiel.net: search is entirely client-side (Stimulus/Turbo), a plain
GET on /recherche/?query=... always returns the empty "no results"
template. Fixed by actually filling the search input and clicking
submit in a real page.
- Cdiscount: returns a fully empty response body to plain HTTP fetches
(no error, just nothing) but renders fine in a real browser.
- PCComponentes: blocks plain HTTP requests with a 403, but a headless
browser with a normal user agent goes through.
Implementation:
- New playwright-core dependency, pointed at a system-installed
Chromium binary via CHROMIUM_EXECUTABLE_PATH (set to /usr/bin/chromium
here) instead of downloading Playwright's own ~300MB browser
- New headless_browser_service.ts: lazily launches a single shared
Chromium instance reused across lookups (avoids paying browser launch
cost per request), with a withPage() helper that always closes its
browser context afterwards. A new start/browser.ts preload registers
an app.terminating() hook so the browser process is closed on shutdown
instead of leaking a child process.
- fetchMaterielNetPrice/fetchCdiscountPrice/fetchPcComponentesPrice
follow the same model-token matching strategy as the existing
cheerio-based merchants (LDLC/Grosbill/TopAchat): search on a short
brand+series query, then pick the result whose label/attribute
contains the extracted model number, to avoid misreporting an
unrelated product's price
- PCComponentes exposes clean data-product-price/data-product-category
attributes directly on result links, so full pre-built "Sobremesa" PCs
that happen to mention the searched CPU are excluded rather than
reported as its price
- Fixed two bugs found while testing with a Ryzen 5 9600X (a case with
multiple search results, unlike the Ryzen 3 3200G single-redirect case
used earlier):
- Materiel.net: a search results list page also has .o-product__price
elements (one per card), so checking for their mere presence isn't
enough to detect the "redirected to a single product" case; now
disambiguated by URL pattern (/produit/ vs /recherche/)
- New parseFlattenedFragmentedPriceText(): Playwright's innerText()
flattens "<whole>€<sup>cents</sup>" prices to plain strings like
"284€95" or "2 049€95", which the existing parser misread as
28495/204995. Cheerio-based scrapers (LDLC/Grosbill) already handled
this correctly via DOM structure (parseFragmentedPrice); this adds
the equivalent for already-flattened text.
- findCheapestOffers() now queries all 7 merchants in parallel
- Extended the MerchantOffer type/favicon domain map on the frontend for
the 3 new merchants
All 30 existing tests still pass unchanged (they don't exercise live
network scraping, only the surrounding auth/validation/caching logic).
70 lines
2.2 KiB
TypeScript
70 lines
2.2 KiB
TypeScript
import { chromium, type Browser } from 'playwright-core'
|
|
import env from '#start/env'
|
|
import logger from '@adonisjs/core/services/logger'
|
|
|
|
const USER_AGENT =
|
|
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
|
|
|
|
let browserPromise: Promise<Browser> | null = null
|
|
|
|
/**
|
|
* Lazily launches a single shared headless Chromium instance, reused
|
|
* across requests instead of paying the ~1-2s browser startup cost on
|
|
* every merchant lookup. Points at a system-installed Chromium/Chrome
|
|
* binary via CHROMIUM_EXECUTABLE_PATH (see .env.example) rather than
|
|
* bundling Playwright's own ~300MB download.
|
|
*/
|
|
function getBrowser(): Promise<Browser> {
|
|
if (!browserPromise) {
|
|
browserPromise = chromium
|
|
.launch({
|
|
executablePath: env.get('CHROMIUM_EXECUTABLE_PATH') || undefined,
|
|
headless: true,
|
|
args: ['--no-sandbox', '--disable-setuid-sandbox'],
|
|
})
|
|
.catch((error) => {
|
|
browserPromise = null
|
|
throw error
|
|
})
|
|
}
|
|
return browserPromise
|
|
}
|
|
|
|
/**
|
|
* Runs `fn` with a fresh browser page/context, always closing the context
|
|
* afterwards (success or failure) so pages don't accumulate across many
|
|
* lookups. The underlying browser process itself stays warm between calls.
|
|
* Returns null (rather than throwing) if the browser fails to launch or the
|
|
* callback throws, since merchant lookups are best-effort.
|
|
*/
|
|
async function withPage<T>(
|
|
fn: (page: import('playwright-core').Page) => Promise<T>
|
|
): Promise<T | null> {
|
|
try {
|
|
const browser = await getBrowser()
|
|
const context = await browser.newContext({ userAgent: USER_AGENT, locale: 'fr-FR' })
|
|
try {
|
|
const page = await context.newPage()
|
|
return await fn(page)
|
|
} finally {
|
|
await context.close()
|
|
}
|
|
} catch (error) {
|
|
logger.warn({ err: error }, 'Headless browser lookup failed')
|
|
return null
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Closes the shared browser instance, if running. Intended for graceful
|
|
* shutdown hooks / tests, not needed during normal request handling.
|
|
*/
|
|
async function closeBrowser(): Promise<void> {
|
|
if (!browserPromise) return
|
|
const browser = await browserPromise.catch(() => null)
|
|
browserPromise = null
|
|
await browser?.close()
|
|
}
|
|
|
|
export { withPage, closeBrowser }
|