feat: add Materiel.net, Cdiscount and PCComponentes via headless browser
Build & Deploy / build (push) Failing after 4s
Build & Deploy / docker (push) Skipped

Google/Bing/DuckDuckGo/Qwant were tested as a way to find prices on the
3 remaining requested merchants and all four block scraping immediately
(visible captcha or JS-only anti-bot challenge on the very first
automated request) - not usable as an intermediary. Instead, a headless
browser was used to scrape these 3 merchants directly, since their
blocking (JS-only search, empty body, 403) turned out to be bypassable
with real browser rendering rather than needing to get past a captcha:

- Materiel.net: search is entirely client-side (Stimulus/Turbo), a plain
  GET on /recherche/?query=... always returns the empty "no results"
  template. Fixed by actually filling the search input and clicking
  submit in a real page.
- Cdiscount: returns a fully empty response body to plain HTTP fetches
  (no error, just nothing) but renders fine in a real browser.
- PCComponentes: blocks plain HTTP requests with a 403, but a headless
  browser with a normal user agent goes through.

Implementation:
- New playwright-core dependency, pointed at a system-installed
  Chromium binary via CHROMIUM_EXECUTABLE_PATH (set to /usr/bin/chromium
  here) instead of downloading Playwright's own ~300MB browser
- New headless_browser_service.ts: lazily launches a single shared
  Chromium instance reused across lookups (avoids paying browser launch
  cost per request), with a withPage() helper that always closes its
  browser context afterwards. A new start/browser.ts preload registers
  an app.terminating() hook so the browser process is closed on shutdown
  instead of leaking a child process.
- fetchMaterielNetPrice/fetchCdiscountPrice/fetchPcComponentesPrice
  follow the same model-token matching strategy as the existing
  cheerio-based merchants (LDLC/Grosbill/TopAchat): search on a short
  brand+series query, then pick the result whose label/attribute
  contains the extracted model number, to avoid misreporting an
  unrelated product's price
- PCComponentes exposes clean data-product-price/data-product-category
  attributes directly on result links, so full pre-built "Sobremesa" PCs
  that happen to mention the searched CPU are excluded rather than
  reported as its price
- Fixed two bugs found while testing with a Ryzen 5 9600X (a case with
  multiple search results, unlike the Ryzen 3 3200G single-redirect case
  used earlier):
  - Materiel.net: a search results list page also has .o-product__price
    elements (one per card), so checking for their mere presence isn't
    enough to detect the "redirected to a single product" case; now
    disambiguated by URL pattern (/produit/ vs /recherche/)
  - New parseFlattenedFragmentedPriceText(): Playwright's innerText()
    flattens "<whole>€<sup>cents</sup>" prices to plain strings like
    "284€95" or "2 049€95", which the existing parser misread as
    28495/204995. Cheerio-based scrapers (LDLC/Grosbill) already handled
    this correctly via DOM structure (parseFragmentedPrice); this adds
    the equivalent for already-flattened text.
- findCheapestOffers() now queries all 7 merchants in parallel
- Extended the MerchantOffer type/favicon domain map on the frontend for
  the 3 new merchants

All 30 existing tests still pass unchanged (they don't exercise live
network scraping, only the surrounding auth/validation/caching logic).
This commit is contained in:
Kevin L'Hereec committed 2026-07-23 15:25:12 +02:00
1 parent 62ec796e05
commit a7788e593e
10 files changed
+319 -12

No files matched your search

+8 -1
View File
@@ -62,7 +62,14 @@ export type ScrapedPart = {
}
export type MerchantOffer = {
merchant: 'Amazon' | 'LDLC' | 'Grosbill' | 'TopAchat'
merchant:
| 'Amazon'
| 'LDLC'
| 'Grosbill'
| 'TopAchat'
| 'Materiel.net'
| 'Cdiscount'
| 'PCComponentes'
price: number
url: string
}
+7 -4
View File
@@ -78,10 +78,13 @@ function cheapestOffer(part: ScrapedPart): MerchantOffer | null {
function merchantDomain(merchant: MerchantOffer['merchant']): string {
const domains: Record<MerchantOffer['merchant'], string> = {
Amazon: 'amazon.fr',
LDLC: 'ldlc.com',
Grosbill: 'grosbill.com',
TopAchat: 'topachat.com',
'Amazon': 'amazon.fr',
'LDLC': 'ldlc.com',
'Grosbill': 'grosbill.com',
'TopAchat': 'topachat.com',
'Materiel.net': 'materiel.net',
'Cdiscount': 'cdiscount.com',
'PCComponentes': 'pccomponentes.com',
}
return domains[merchant]
}