Files
Kevin L'Hereec a7788e593e
Build & Deploy / build (push) Failing after 4s
Build & Deploy / docker (push) Skipped
feat: add Materiel.net, Cdiscount and PCComponentes via headless browser
Google/Bing/DuckDuckGo/Qwant were tested as a way to find prices on the
3 remaining requested merchants and all four block scraping immediately
(visible captcha or JS-only anti-bot challenge on the very first
automated request) - not usable as an intermediary. Instead, a headless
browser was used to scrape these 3 merchants directly, since their
blocking (JS-only search, empty body, 403) turned out to be bypassable
with real browser rendering rather than needing to get past a captcha:

- Materiel.net: search is entirely client-side (Stimulus/Turbo), a plain
  GET on /recherche/?query=... always returns the empty "no results"
  template. Fixed by actually filling the search input and clicking
  submit in a real page.
- Cdiscount: returns a fully empty response body to plain HTTP fetches
  (no error, just nothing) but renders fine in a real browser.
- PCComponentes: blocks plain HTTP requests with a 403, but a headless
  browser with a normal user agent goes through.

Implementation:
- New playwright-core dependency, pointed at a system-installed
  Chromium binary via CHROMIUM_EXECUTABLE_PATH (set to /usr/bin/chromium
  here) instead of downloading Playwright's own ~300MB browser
- New headless_browser_service.ts: lazily launches a single shared
  Chromium instance reused across lookups (avoids paying browser launch
  cost per request), with a withPage() helper that always closes its
  browser context afterwards. A new start/browser.ts preload registers
  an app.terminating() hook so the browser process is closed on shutdown
  instead of leaking a child process.
- fetchMaterielNetPrice/fetchCdiscountPrice/fetchPcComponentesPrice
  follow the same model-token matching strategy as the existing
  cheerio-based merchants (LDLC/Grosbill/TopAchat): search on a short
  brand+series query, then pick the result whose label/attribute
  contains the extracted model number, to avoid misreporting an
  unrelated product's price
- PCComponentes exposes clean data-product-price/data-product-category
  attributes directly on result links, so full pre-built "Sobremesa" PCs
  that happen to mention the searched CPU are excluded rather than
  reported as its price
- Fixed two bugs found while testing with a Ryzen 5 9600X (a case with
  multiple search results, unlike the Ryzen 3 3200G single-redirect case
  used earlier):
  - Materiel.net: a search results list page also has .o-product__price
    elements (one per card), so checking for their mere presence isn't
    enough to detect the "redirected to a single product" case; now
    disambiguated by URL pattern (/produit/ vs /recherche/)
  - New parseFlattenedFragmentedPriceText(): Playwright's innerText()
    flattens "<whole>€<sup>cents</sup>" prices to plain strings like
    "284€95" or "2 049€95", which the existing parser misread as
    28495/204995. Cheerio-based scrapers (LDLC/Grosbill) already handled
    this correctly via DOM structure (parseFragmentedPrice); this adds
    the equivalent for already-flattened text.
- findCheapestOffers() now queries all 7 merchants in parallel
- Extended the MerchantOffer type/favicon domain map on the frontend for
  the 3 new merchants

All 30 existing tests still pass unchanged (they don't exercise live
network scraping, only the surrounding auth/validation/caching logic).
2026-07-23 15:25:12 +02:00

140 lines
4.0 KiB
TypeScript

import { indexPages } from '@adonisjs/inertia'
import { indexEntities } from '@adonisjs/core'
import { defineConfig } from '@adonisjs/core/app'
import { generateRegistry } from '@tuyau/core/hooks'
export default defineConfig({
/*
|--------------------------------------------------------------------------
| Experimental flags
|--------------------------------------------------------------------------
|
| The following features will be enabled by default in the next major release
| of AdonisJS. You can opt into them today to avoid any breaking changes
| during upgrade.
|
*/
experimental: {},
/*
|--------------------------------------------------------------------------
| Commands
|--------------------------------------------------------------------------
|
| List of ace commands to register from packages. The application commands
| will be scanned automatically from the "./commands" directory.
|
*/
commands: [
() => import('@adonisjs/core/commands'),
() => import('@adonisjs/lucid/commands'),
() => import('@adonisjs/session/commands'),
() => import('@adonisjs/inertia/commands'),
],
/*
|--------------------------------------------------------------------------
| Service providers
|--------------------------------------------------------------------------
|
| List of service providers to import and register when booting the
| application
|
*/
providers: [
() => import('@adonisjs/core/providers/app_provider'),
() => import('@adonisjs/core/providers/hash_provider'),
{
file: () => import('@adonisjs/core/providers/repl_provider'),
environment: ['repl', 'test'],
},
() => import('@adonisjs/core/providers/vinejs_provider'),
() => import('@adonisjs/core/providers/edge_provider'),
() => import('@adonisjs/session/session_provider'),
() => import('@adonisjs/vite/vite_provider'),
() => import('@adonisjs/shield/shield_provider'),
() => import('@adonisjs/static/static_provider'),
() => import('@adonisjs/lucid/database_provider'),
() => import('@adonisjs/cors/cors_provider'),
() => import('@adonisjs/inertia/inertia_provider'),
() => import('@adonisjs/auth/auth_provider'),
() => import('#providers/api_provider'),
],
/*
|--------------------------------------------------------------------------
| Preloads
|--------------------------------------------------------------------------
|
| List of modules to import before starting the application.
|
*/
preloads: [
() => import('#start/routes'),
() => import('#start/kernel'),
() => import('#start/validator'),
() => import('#start/browser'),
],
/*
|--------------------------------------------------------------------------
| Tests
|--------------------------------------------------------------------------
|
| List of test suites to organize tests by their type. Feel free to remove
| and add additional suites.
|
*/
tests: {
suites: [
{
files: ['tests/unit/**/*.spec.{ts,js}'],
name: 'unit',
timeout: 2000,
},
{
files: ['tests/functional/**/*.spec.{ts,js}'],
name: 'functional',
timeout: 30000,
},
{
files: ['tests/browser/**/*.spec.{ts,js}'],
name: 'browser',
timeout: 300000,
},
],
forceExit: false,
},
/*
|--------------------------------------------------------------------------
| Metafiles
|--------------------------------------------------------------------------
|
| A collection of files you want to copy to the build folder when creating
| the production build.
|
*/
metaFiles: [
{
pattern: 'resources/views/**/*.edge',
reloadServer: false,
},
{
pattern: 'public/**',
reloadServer: false,
},
],
hooks: {
init: [
indexEntities({
transformers: { enabled: true, withSharedProps: true },
}),
indexPages({ framework: 'vue3' }),
generateRegistry(),
],
buildStarting: [() => import('@adonisjs/vite/build_hook')],
},
})