Google/Bing/DuckDuckGo/Qwant were tested as a way to find prices on the
3 remaining requested merchants and all four block scraping immediately
(visible captcha or JS-only anti-bot challenge on the very first
automated request) - not usable as an intermediary. Instead, a headless
browser was used to scrape these 3 merchants directly, since their
blocking (JS-only search, empty body, 403) turned out to be bypassable
with real browser rendering rather than needing to get past a captcha:
- Materiel.net: search is entirely client-side (Stimulus/Turbo), a plain
GET on /recherche/?query=... always returns the empty "no results"
template. Fixed by actually filling the search input and clicking
submit in a real page.
- Cdiscount: returns a fully empty response body to plain HTTP fetches
(no error, just nothing) but renders fine in a real browser.
- PCComponentes: blocks plain HTTP requests with a 403, but a headless
browser with a normal user agent goes through.
Implementation:
- New playwright-core dependency, pointed at a system-installed
Chromium binary via CHROMIUM_EXECUTABLE_PATH (set to /usr/bin/chromium
here) instead of downloading Playwright's own ~300MB browser
- New headless_browser_service.ts: lazily launches a single shared
Chromium instance reused across lookups (avoids paying browser launch
cost per request), with a withPage() helper that always closes its
browser context afterwards. A new start/browser.ts preload registers
an app.terminating() hook so the browser process is closed on shutdown
instead of leaking a child process.
- fetchMaterielNetPrice/fetchCdiscountPrice/fetchPcComponentesPrice
follow the same model-token matching strategy as the existing
cheerio-based merchants (LDLC/Grosbill/TopAchat): search on a short
brand+series query, then pick the result whose label/attribute
contains the extracted model number, to avoid misreporting an
unrelated product's price
- PCComponentes exposes clean data-product-price/data-product-category
attributes directly on result links, so full pre-built "Sobremesa" PCs
that happen to mention the searched CPU are excluded rather than
reported as its price
- Fixed two bugs found while testing with a Ryzen 5 9600X (a case with
multiple search results, unlike the Ryzen 3 3200G single-redirect case
used earlier):
- Materiel.net: a search results list page also has .o-product__price
elements (one per card), so checking for their mere presence isn't
enough to detect the "redirected to a single product" case; now
disambiguated by URL pattern (/produit/ vs /recherche/)
- New parseFlattenedFragmentedPriceText(): Playwright's innerText()
flattens "<whole>€<sup>cents</sup>" prices to plain strings like
"284€95" or "2 049€95", which the existing parser misread as
28495/204995. Cheerio-based scrapers (LDLC/Grosbill) already handled
this correctly via DOM structure (parseFragmentedPrice); this adds
the equivalent for already-flattened text.
- findCheapestOffers() now queries all 7 merchants in parallel
- Extended the MerchantOffer type/favicon domain map on the frontend for
the 3 new merchants
All 30 existing tests still pass unchanged (they don't exercise live
network scraping, only the surrounding auth/validation/caching logic).
140 lines
4.0 KiB
TypeScript
140 lines
4.0 KiB
TypeScript
import { indexPages } from '@adonisjs/inertia'
|
|
import { indexEntities } from '@adonisjs/core'
|
|
import { defineConfig } from '@adonisjs/core/app'
|
|
import { generateRegistry } from '@tuyau/core/hooks'
|
|
|
|
export default defineConfig({
|
|
/*
|
|
|--------------------------------------------------------------------------
|
|
| Experimental flags
|
|
|--------------------------------------------------------------------------
|
|
|
|
|
| The following features will be enabled by default in the next major release
|
|
| of AdonisJS. You can opt into them today to avoid any breaking changes
|
|
| during upgrade.
|
|
|
|
|
*/
|
|
experimental: {},
|
|
|
|
/*
|
|
|--------------------------------------------------------------------------
|
|
| Commands
|
|
|--------------------------------------------------------------------------
|
|
|
|
|
| List of ace commands to register from packages. The application commands
|
|
| will be scanned automatically from the "./commands" directory.
|
|
|
|
|
*/
|
|
commands: [
|
|
() => import('@adonisjs/core/commands'),
|
|
() => import('@adonisjs/lucid/commands'),
|
|
() => import('@adonisjs/session/commands'),
|
|
() => import('@adonisjs/inertia/commands'),
|
|
],
|
|
|
|
/*
|
|
|--------------------------------------------------------------------------
|
|
| Service providers
|
|
|--------------------------------------------------------------------------
|
|
|
|
|
| List of service providers to import and register when booting the
|
|
| application
|
|
|
|
|
*/
|
|
providers: [
|
|
() => import('@adonisjs/core/providers/app_provider'),
|
|
() => import('@adonisjs/core/providers/hash_provider'),
|
|
{
|
|
file: () => import('@adonisjs/core/providers/repl_provider'),
|
|
environment: ['repl', 'test'],
|
|
},
|
|
() => import('@adonisjs/core/providers/vinejs_provider'),
|
|
() => import('@adonisjs/core/providers/edge_provider'),
|
|
() => import('@adonisjs/session/session_provider'),
|
|
() => import('@adonisjs/vite/vite_provider'),
|
|
() => import('@adonisjs/shield/shield_provider'),
|
|
() => import('@adonisjs/static/static_provider'),
|
|
() => import('@adonisjs/lucid/database_provider'),
|
|
() => import('@adonisjs/cors/cors_provider'),
|
|
() => import('@adonisjs/inertia/inertia_provider'),
|
|
() => import('@adonisjs/auth/auth_provider'),
|
|
() => import('#providers/api_provider'),
|
|
],
|
|
|
|
/*
|
|
|--------------------------------------------------------------------------
|
|
| Preloads
|
|
|--------------------------------------------------------------------------
|
|
|
|
|
| List of modules to import before starting the application.
|
|
|
|
|
*/
|
|
preloads: [
|
|
() => import('#start/routes'),
|
|
() => import('#start/kernel'),
|
|
() => import('#start/validator'),
|
|
() => import('#start/browser'),
|
|
],
|
|
|
|
/*
|
|
|--------------------------------------------------------------------------
|
|
| Tests
|
|
|--------------------------------------------------------------------------
|
|
|
|
|
| List of test suites to organize tests by their type. Feel free to remove
|
|
| and add additional suites.
|
|
|
|
|
*/
|
|
tests: {
|
|
suites: [
|
|
{
|
|
files: ['tests/unit/**/*.spec.{ts,js}'],
|
|
name: 'unit',
|
|
timeout: 2000,
|
|
},
|
|
{
|
|
files: ['tests/functional/**/*.spec.{ts,js}'],
|
|
name: 'functional',
|
|
timeout: 30000,
|
|
},
|
|
{
|
|
files: ['tests/browser/**/*.spec.{ts,js}'],
|
|
name: 'browser',
|
|
timeout: 300000,
|
|
},
|
|
],
|
|
forceExit: false,
|
|
},
|
|
|
|
/*
|
|
|--------------------------------------------------------------------------
|
|
| Metafiles
|
|
|--------------------------------------------------------------------------
|
|
|
|
|
| A collection of files you want to copy to the build folder when creating
|
|
| the production build.
|
|
|
|
|
*/
|
|
metaFiles: [
|
|
{
|
|
pattern: 'resources/views/**/*.edge',
|
|
reloadServer: false,
|
|
},
|
|
{
|
|
pattern: 'public/**',
|
|
reloadServer: false,
|
|
},
|
|
],
|
|
|
|
hooks: {
|
|
init: [
|
|
indexEntities({
|
|
transformers: { enabled: true, withSharedProps: true },
|
|
}),
|
|
indexPages({ framework: 'vue3' }),
|
|
generateRegistry(),
|
|
],
|
|
buildStarting: [() => import('@adonisjs/vite/build_hook')],
|
|
},
|
|
})
|