a7788e593e3907748ea399d9afd65ab85fb14533
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Kevin L'Hereec
|
a7788e593e |
feat: add Materiel.net, Cdiscount and PCComponentes via headless browser
Google/Bing/DuckDuckGo/Qwant were tested as a way to find prices on the
3 remaining requested merchants and all four block scraping immediately
(visible captcha or JS-only anti-bot challenge on the very first
automated request) - not usable as an intermediary. Instead, a headless
browser was used to scrape these 3 merchants directly, since their
blocking (JS-only search, empty body, 403) turned out to be bypassable
with real browser rendering rather than needing to get past a captcha:
- Materiel.net: search is entirely client-side (Stimulus/Turbo), a plain
GET on /recherche/?query=... always returns the empty "no results"
template. Fixed by actually filling the search input and clicking
submit in a real page.
- Cdiscount: returns a fully empty response body to plain HTTP fetches
(no error, just nothing) but renders fine in a real browser.
- PCComponentes: blocks plain HTTP requests with a 403, but a headless
browser with a normal user agent goes through.
Implementation:
- New playwright-core dependency, pointed at a system-installed
Chromium binary via CHROMIUM_EXECUTABLE_PATH (set to /usr/bin/chromium
here) instead of downloading Playwright's own ~300MB browser
- New headless_browser_service.ts: lazily launches a single shared
Chromium instance reused across lookups (avoids paying browser launch
cost per request), with a withPage() helper that always closes its
browser context afterwards. A new start/browser.ts preload registers
an app.terminating() hook so the browser process is closed on shutdown
instead of leaking a child process.
- fetchMaterielNetPrice/fetchCdiscountPrice/fetchPcComponentesPrice
follow the same model-token matching strategy as the existing
cheerio-based merchants (LDLC/Grosbill/TopAchat): search on a short
brand+series query, then pick the result whose label/attribute
contains the extracted model number, to avoid misreporting an
unrelated product's price
- PCComponentes exposes clean data-product-price/data-product-category
attributes directly on result links, so full pre-built "Sobremesa" PCs
that happen to mention the searched CPU are excluded rather than
reported as its price
- Fixed two bugs found while testing with a Ryzen 5 9600X (a case with
multiple search results, unlike the Ryzen 3 3200G single-redirect case
used earlier):
- Materiel.net: a search results list page also has .o-product__price
elements (one per card), so checking for their mere presence isn't
enough to detect the "redirected to a single product" case; now
disambiguated by URL pattern (/produit/ vs /recherche/)
- New parseFlattenedFragmentedPriceText(): Playwright's innerText()
flattens "<whole>€<sup>cents</sup>" prices to plain strings like
"284€95" or "2 049€95", which the existing parser misread as
28495/204995. Cheerio-based scrapers (LDLC/Grosbill) already handled
this correctly via DOM structure (parseFragmentedPrice); this adds
the equivalent for already-flattened text.
- findCheapestOffers() now queries all 7 merchants in parallel
- Extended the MerchantOffer type/favicon domain map on the frontend for
the 3 new merchants
All 30 existing tests still pass unchanged (they don't exercise live
network scraping, only the surrounding auth/validation/caching logic).
|
||
|
Kevin L'Hereec
|
62ec796e05 |
fix: repair broken CSP and compare live merchant prices for new parts
CSP fix:
- The previous shield.ts config used '@viteDevUrl'/'@viteHmrUrl' CSP
keywords that don't exist in this version of @adonisjs/shield (only
'@viteUrl' is registered by @adonisjs/vite, and it resolves to an empty
string here since Vite assets are proxied through AdonisJS on the same
origin). Both placeholders were sent to the browser literally, breaking
Google Fonts, the Vite HMR websocket, and pc-builder.io/favicon images.
- Replaced with concrete directives: 'self' covers same-origin Vite
assets, ws://localhost:{VITE_HMR_PORT|24678} is added only in dev for
the HMR socket, and explicit allowlists for Google Fonts
(fonts.googleapis.com/fonts.gstatic.com), pc-builder.io product images,
and Google's favicon service (www.google.com) used by the merchant
comparison UI
Real purchase links + live price comparison:
- part_scraper_service.ts: ScrapedPart.sourceUrl now stores the actual
Amazon.fr purchase link scraped from pc-builder.io's category pages
(https://www.amazon.fr/dp/{ASIN}?tag=... or a search fallback), instead
of the pc-builder.io product page, which isn't a place anyone can buy
anything from
- New merchant_price_service.ts: on-demand live price lookup across
Amazon, LDLC, Grosbill and TopAchat. Of 6 requested merchants,
Materiel.net (search requires JS/Turbo, no results via plain GET) and
Cdiscount/PCComponentes (empty body / 403) were dropped as not scrapable
without a headless browser or paid API
- Matching strategy: pc-builder.io's full product names (e.g. "AMD Ryzen 3
3200G 3.6 GHz Quad-Core Processor") return zero results on every
merchant's search engine, so we search on a short brand+series query
and then pick the specific result whose label contains the extracted
model token (e.g. "3200G"). No token match means no offer returned,
rather than risk quoting the price of a different, unrelated product
- Fixed price parsing for LDLC/Grosbill, which render prices as nested
price/sup elements for whole euros vs cents (and LDLC additionally
wraps that in a duplicate outer .price div) - naive text concatenation
was misreading "64,95" as "6495"
- New GET /scraped-parts/compare endpoint (authenticated), called on
demand per search result rather than for every row, since it makes
several outbound requests and can take a few seconds
- NewPartSearch.vue: "Compare prices" button per result shows the
cheapest merchant with its favicon and links to the actual offer;
importing a part now prefers the live cheapest price/link over
pc-builder.io's potentially stale catalog price (confirmed stale in
practice: pc-builder.io listed a Ryzen 3 3200G at 128 euros while
Amazon sells it for 66.89 euros)
- 3 new functional tests for the compare endpoint (auth required, 404 on
unknown part, 422 on invalid id)
|
||
|
Kevin L'Hereec
|
5854ff8ed1 |
feat: search and import new parts scraped from pc-builder.io
Idealo and PCPartPicker were evaluated and ruled out: - idealo.fr search pages return 503 (Cloudflare) - the internal Idealo GraphQL endpoint used by a public scraper (app.idealo.de/app-backend/api) is protected by Akamai Bot Manager and silently returns empty data even with mobile app headers - PCPartPicker's product list/search relies on a session-bound CSRF token refreshed per page load, against a scraping-hostile ToS pc-builder.io category pages are server-rendered, include name/price/image for ~100 products per category, and have no bot protection, so the feature is built on top of that source only. - New scraped_parts cache table (type_id, name, price, image_url, source, source_url, source_ref unique per source) populated by a batch job rather than scraped live per search, since only ~100 first results per category are available without reverse-engineering pc-builder.io's DataTables/CSRF pagination flow - part_scraper_service.ts: maps our ComponentType names to pc-builder.io category slugs (SSD -> storage, since pc-builder.io has no separate HDD category) and parses each category page with cheerio, skipping rows without a resolved price (generic 'search on Amazon' links) - node ace scrape:parts: refreshes the cache for every mapped category (upsert on source+source_ref), runnable manually or later via cron - GET /scraped-parts (authenticated): search the cache by name (LOWER() LIKE, avoids a MySQL utf8_bin/utf8mb4 collation mismatch from Lucid's whereLike) and optional typeId filter - NewPartSearch.vue: debounced search widget wired into components/create.vue; clicking 'Import' pre-fills the existing component creation form (name, price, type, purchase date = today) instead of creating the Component directly, so the user still reviews and saves it like any manually entered part - 3 functional tests covering search by name, filter by typeId, and unauthenticated access being redirected to login - Initial scrape run: 989 parts across 10 categories |
||
|
Kevin L'Hereec
|
da71d33450 |
feat: add public read-only sharing for computers
Adds the ability for an owner to share a computer build via a public, unauthenticated link, inspired by a feature request from the original pcbuilder repo (no such feature actually existed there — implemented from scratch here). - New columns on computers: share_token (unique, nullable), share_enabled, share_show_sell_price, share_show_cost - computer_share_service.ts: generates a non-guessable share token (crypto.randomBytes, base64url), enables/disables sharing, and supports rotating the token to invalidate previously distributed links - ComputerShareController (authenticated, owner-scoped like the rest of the app): PUT /computers/:id/share to update settings, POST /computers/:id/share/rotate to rotate the token - PublicComputerController + GET /share/:token (outside any auth/guest group, reachable anonymously): resolves only computers with share_enabled = true, 404 otherwise - ComputerPublicTransformer: dedicated read-only projection that never leaks userId, and only includes sellPrice/cost when the owner opted in via share_show_sell_price / share_show_cost — everything else (title, scores, state, components, pictures) is always visible on a shared link - ComputerShareModal.vue: owner-facing UI (toggle sharing, choose which prices to expose, copy the public link, revoke/rotate it) wired into computers/show.vue via a new 'Share' button - share/show.vue: public page rendered with EmptyLayout (no authenticated header/sidebar) - Boolean columns use a cast since MySQL returns tinyint(1), not native booleans, through Lucid - 6 new functional tests covering: enabling sharing, price visibility toggles, disabling sharing revokes the link, cross-user protection on the share settings endpoint (404), token rotation invalidating the old link, and anonymous access to a shared computer |
||
|
Kevin
|
f3cbbe901a | fix: cookies HTTP, logout route prefix, header logout method | ||
|
Kevin
|
648d0af538 | Initial commit: PC Builder AdonisJS project |