RTX 5060 investigation: confirmed via pc-builder.io's own server-side
search (search_query=RTX 5060 against the full 6222-product GPU catalog,
0 results) that it simply isn't in pc-builder.io's catalog yet - not a
scraping bug, the source data doesn't have it. However, while checking
whether the other merchants had it in stock, found a real matching bug:
labelMatchesModelToken bug: extractModelToken("...RTX 5060 Ti...") only
returned "5060", identical to extractModelToken("...RTX 5060...") without
Ti. Every merchant lookup for either card matched on plain "5060" and
could silently return the *other* card's price (confirmed live: LDLC and
Grosbill returned the exact same product/price for both "RTX 5060" and
"RTX 5060 Ti" queries before this fix).
Fixed by:
- extractModelToken now appends a trailing suffix token (Ti, Super, XT,
XTX) when it immediately follows the model number, so "5060 Ti" and
"5060" are extracted as distinct tokens
- New labelMatchesModelToken() replaces the naive
label.includes(modelToken) check everywhere: a bare token ("5060") no
longer matches a label that actually has a suffix right after it
("5060 Ti"), while a suffixed token ("5060 Ti") only matches labels
carrying that exact suffix
- 9 new unit tests covering extraction and matching, including the
regression case (RTX 5060 must not match "RTX 5060 Ti" labels)
Verified fix live: LDLC/Grosbill now return correctly distinct prices for
"RTX 5060" (369.95/364.99) vs "RTX 5060 Ti" (489.95/329.99).
New manual_part_seeder_service.ts: since pc-builder.io's catalog lags
behind real product releases, this seeds a small manually-tracked list of
recent parts it doesn't carry yet (currently RTX 5060 8GB and 5060 Ti
16GB), resolving their price live from the already-supported merchants
rather than hardcoding a number that would go stale. `scrape:parts` now
runs this after the regular catalog scrape. Skips an entry rather than
inserting a guessed price if no merchant has it in stock.
Also fixes a real bug hit while testing the seeder: scraped_parts and
components.purchase_url were varchar(255), which truncates/rejects
Grosbill's long tracking-redirect URLs (500+ characters with encoded
analytics params) - both widened to TEXT via a new migration.
Bug found: "Gigabyte GAMING OC GeForce RTX 5070 Ti 16 GB Video Card" kept
showing a stale price of 431.17 even after pc-builder.io itself no longer
had a fixed offer for it (its "Add to Build" HTML only had a generic
Amazon search link, no .price-box element at all, confirmed by fetching
the live page directly).
Root cause: scrapeComponentType's "never lose a better price" rule (added
to make ScrapedPartsController#compare's persisted merchant prices
durable across catalog refreshes) was too broad. It treated ANY existing
price as worth protecting whenever the incoming catalog price was null or
higher - including a price that had only ever come from a *previous*
pc-builder.io scrape itself (priceSource: null), with no independent
verification. Since pc-builder.io's own listings change constantly (a
GPU can lose its priced Amazon link if the referenced ASIN goes out of
stock), that logic effectively froze old pc-builder.io prices forever
even after the site removed them.
Fix: extracted the decision into shouldKeepExistingPrice(), which now
only protects a price when priceSource is a real merchant name (set by
updatePriceIfCheaper after a live comparison confirmed it on the
merchant's own page) - never when priceSource is null (an unverified,
previous pc-builder.io catalog price). A plain re-scrape can always
correct/clear an unverified price, matching whatever pc-builder.io
currently shows.
4 new unit tests for shouldKeepExistingPrice covering: unverified
pc-builder.io price gets overwritten (the regression case), verified
merchant price is protected from worse/missing catalog data, verified
merchant price is still replaced by a genuinely cheaper catalog price,
and no-existing-part returns false.
Also re-ran `node ace scrape:parts` to apply the fix to already-cached
data with stale unverified prices (affected ~1500 rows across the
catalog before the fix, mostly likely unaffected in practice but no
longer silently trusted going forward).
30 results was too low once pc-builder.io's full catalog is scraped
(e.g. 47 RTX 5070/5070 Ti variants alone), silently truncating results
a user would otherwise see. Also added a result count hint in
NewPartSearch.vue since the list is scrollable (max-h-96) rather than
paginated, so it wasn't obvious more results existed below the fold.
The RTX 5070 case that prompted this: only 100 GPUs out of pc-builder.io's
actual 6222 were being scraped (its category pages only render the first
page server-side), so cheaper RTX 5070 variants existed on the site but
were invisible to our search. Investigating further also surfaced that
even with full pagination, most recent GPU listings (RTX 5070 included)
have no fixed Amazon price on pc-builder.io at all - just a generic
"search on Amazon" link - so pagination alone doesn't fully solve price
coverage either.
Pagination (part_scraper_service.ts):
- Reverse-engineered pc-builder.io's /products/data endpoint, which powers
the client-side DataTables widget behind every category page. Requires
a session cookie + CSRF token scraped from a fresh category page load,
and an exact field shape (columns[].data must be the literal strings
"Name"/"Add to Build", not numeric indices - a naive DataTables-shaped
request 500s otherwise).
- CATEGORY_CONFIG now tracks each category's real pc-builder.io
category_id and a per-category maxProducts depth, chosen by actual
catalog size and price-sensitivity: GPU near-full coverage (6000),
smaller/less price-volatile categories capped lower (300-800) to keep a
full `scrape:parts` run (all categories) in a reasonable time.
- ensureFreshCategory (triggered inline by a user's search when the cache
is stale) uses a much shallower QUICK_REFRESH_MAX_PRODUCTS=200 cap
instead of the category's full depth, since it runs inside the
request/response cycle of a search - full-depth coverage is only done
by the standalone `scrape:parts` command.
Always keep the best known price:
- scraped_parts.price is now nullable (migration): parts with no fixed
pc-builder.io offer are kept and searchable instead of being silently
skipped, so at least the model is discoverable even before a price is
resolved.
- New price_source / price_checked_at columns track where the currently
stored price came from ("pc-builder.io" or a merchant name from
merchant_price_service.ts).
- ScrapedPartsController#compare now persists the cheapest offer found
(via the new updatePriceIfCheaper()) when it beats the currently stored
price - so comparing prices benefits every future search/import for
that part, not just the user who triggered it. Never regresses the
price: a comparison finding a higher price than what's stored is
ignored.
- scrapeComponentType's catalog refresh applies the same "keep whichever
price is lower" rule per row, so a routine pc-builder.io re-scrape can
never overwrite a better price a comparison had already found and
persisted with a worse (or missing) one.
- Search results (GET /scraped-parts) are now ordered priced-cheapest-first,
then parts with no known price last (`ORDER BY price IS NULL, price ASC`),
so unpriced parts stay visible without pushing priced results down.
- NewPartSearch.vue displays the catalog price inline (or "No price yet"),
and the existing "Compare prices" flow now works for those unpriced
parts the same way it does for priced ones.
7 new/updated functional tests: nullable price is searchable and sorted
last, and 3 tests for updatePriceIfCheaper (cheaper offer replaces price,
pricier offer is ignored, first-ever price is accepted for a previously
unpriced part).
Google/Bing/DuckDuckGo/Qwant were tested as a way to find prices on the
3 remaining requested merchants and all four block scraping immediately
(visible captcha or JS-only anti-bot challenge on the very first
automated request) - not usable as an intermediary. Instead, a headless
browser was used to scrape these 3 merchants directly, since their
blocking (JS-only search, empty body, 403) turned out to be bypassable
with real browser rendering rather than needing to get past a captcha:
- Materiel.net: search is entirely client-side (Stimulus/Turbo), a plain
GET on /recherche/?query=... always returns the empty "no results"
template. Fixed by actually filling the search input and clicking
submit in a real page.
- Cdiscount: returns a fully empty response body to plain HTTP fetches
(no error, just nothing) but renders fine in a real browser.
- PCComponentes: blocks plain HTTP requests with a 403, but a headless
browser with a normal user agent goes through.
Implementation:
- New playwright-core dependency, pointed at a system-installed
Chromium binary via CHROMIUM_EXECUTABLE_PATH (set to /usr/bin/chromium
here) instead of downloading Playwright's own ~300MB browser
- New headless_browser_service.ts: lazily launches a single shared
Chromium instance reused across lookups (avoids paying browser launch
cost per request), with a withPage() helper that always closes its
browser context afterwards. A new start/browser.ts preload registers
an app.terminating() hook so the browser process is closed on shutdown
instead of leaking a child process.
- fetchMaterielNetPrice/fetchCdiscountPrice/fetchPcComponentesPrice
follow the same model-token matching strategy as the existing
cheerio-based merchants (LDLC/Grosbill/TopAchat): search on a short
brand+series query, then pick the result whose label/attribute
contains the extracted model number, to avoid misreporting an
unrelated product's price
- PCComponentes exposes clean data-product-price/data-product-category
attributes directly on result links, so full pre-built "Sobremesa" PCs
that happen to mention the searched CPU are excluded rather than
reported as its price
- Fixed two bugs found while testing with a Ryzen 5 9600X (a case with
multiple search results, unlike the Ryzen 3 3200G single-redirect case
used earlier):
- Materiel.net: a search results list page also has .o-product__price
elements (one per card), so checking for their mere presence isn't
enough to detect the "redirected to a single product" case; now
disambiguated by URL pattern (/produit/ vs /recherche/)
- New parseFlattenedFragmentedPriceText(): Playwright's innerText()
flattens "<whole>€<sup>cents</sup>" prices to plain strings like
"284€95" or "2 049€95", which the existing parser misread as
28495/204995. Cheerio-based scrapers (LDLC/Grosbill) already handled
this correctly via DOM structure (parseFragmentedPrice); this adds
the equivalent for already-flattened text.
- findCheapestOffers() now queries all 7 merchants in parallel
- Extended the MerchantOffer type/favicon domain map on the frontend for
the 3 new merchants
All 30 existing tests still pass unchanged (they don't exercise live
network scraping, only the surrounding auth/validation/caching logic).
CSP fix:
- The previous shield.ts config used '@viteDevUrl'/'@viteHmrUrl' CSP
keywords that don't exist in this version of @adonisjs/shield (only
'@viteUrl' is registered by @adonisjs/vite, and it resolves to an empty
string here since Vite assets are proxied through AdonisJS on the same
origin). Both placeholders were sent to the browser literally, breaking
Google Fonts, the Vite HMR websocket, and pc-builder.io/favicon images.
- Replaced with concrete directives: 'self' covers same-origin Vite
assets, ws://localhost:{VITE_HMR_PORT|24678} is added only in dev for
the HMR socket, and explicit allowlists for Google Fonts
(fonts.googleapis.com/fonts.gstatic.com), pc-builder.io product images,
and Google's favicon service (www.google.com) used by the merchant
comparison UI
Real purchase links + live price comparison:
- part_scraper_service.ts: ScrapedPart.sourceUrl now stores the actual
Amazon.fr purchase link scraped from pc-builder.io's category pages
(https://www.amazon.fr/dp/{ASIN}?tag=... or a search fallback), instead
of the pc-builder.io product page, which isn't a place anyone can buy
anything from
- New merchant_price_service.ts: on-demand live price lookup across
Amazon, LDLC, Grosbill and TopAchat. Of 6 requested merchants,
Materiel.net (search requires JS/Turbo, no results via plain GET) and
Cdiscount/PCComponentes (empty body / 403) were dropped as not scrapable
without a headless browser or paid API
- Matching strategy: pc-builder.io's full product names (e.g. "AMD Ryzen 3
3200G 3.6 GHz Quad-Core Processor") return zero results on every
merchant's search engine, so we search on a short brand+series query
and then pick the specific result whose label contains the extracted
model token (e.g. "3200G"). No token match means no offer returned,
rather than risk quoting the price of a different, unrelated product
- Fixed price parsing for LDLC/Grosbill, which render prices as nested
price/sup elements for whole euros vs cents (and LDLC additionally
wraps that in a duplicate outer .price div) - naive text concatenation
was misreading "64,95" as "6495"
- New GET /scraped-parts/compare endpoint (authenticated), called on
demand per search result rather than for every row, since it makes
several outbound requests and can take a few seconds
- NewPartSearch.vue: "Compare prices" button per result shows the
cheapest merchant with its favicon and links to the actual offer;
importing a part now prefers the live cheapest price/link over
pc-builder.io's potentially stale catalog price (confirmed stale in
practice: pc-builder.io listed a Ryzen 3 3200G at 128 euros while
Amazon sells it for 66.89 euros)
- 3 new functional tests for the compare endpoint (auth required, 404 on
unknown part, 422 on invalid id)
- New nullable purchase_url column on components
- Store/update validators accept an optional purchaseUrl (must be a valid
absolute URL with a protocol when provided)
- components/create.vue: NewPartSearch import now also fills purchaseUrl
from the scraped part's source URL (pc-builder.io product page); the
field remains manually editable for second-hand parts entered by hand
- components/edit.vue: purchaseUrl is editable after creation
- components/show.vue: displays the purchase site as a clickable link
when present
- 4 new functional tests: create with/without a purchase URL, invalid URL
rejected (asserts the Inertia inputErrorsBag flash, not a raw 422 -
content negotiation only returns JSON for XHR/API-style requests), and
updating an existing component's purchase URL
- part_scraper_service.ts: new CACHE_TTL_HOURS (24), isCategoryStale()
checks the most recent scrapedAt for a given ComponentType's mapped
category, ensureFreshCategory() re-scrapes it synchronously if missing
or stale (errors are logged, not thrown, so search still falls back to
whatever is already cached)
- ScrapedPartsController#index: when a typeId filter is provided, the
corresponding category cache is refreshed eagerly if stale before
querying results. Free-text searches without a typeId skip this to
avoid re-scraping every mapped category on every keystroke -
'node ace scrape:parts' still refreshes everything at once on demand
- 4 new tests covering staleness detection (missing cache, <24h, >24h)
and that unmapped types never trigger a scrape/network call
Idealo and PCPartPicker were evaluated and ruled out:
- idealo.fr search pages return 503 (Cloudflare)
- the internal Idealo GraphQL endpoint used by a public scraper
(app.idealo.de/app-backend/api) is protected by Akamai Bot Manager and
silently returns empty data even with mobile app headers
- PCPartPicker's product list/search relies on a session-bound CSRF token
refreshed per page load, against a scraping-hostile ToS
pc-builder.io category pages are server-rendered, include name/price/image
for ~100 products per category, and have no bot protection, so the feature
is built on top of that source only.
- New scraped_parts cache table (type_id, name, price, image_url, source,
source_url, source_ref unique per source) populated by a batch job
rather than scraped live per search, since only ~100 first results per
category are available without reverse-engineering pc-builder.io's
DataTables/CSRF pagination flow
- part_scraper_service.ts: maps our ComponentType names to pc-builder.io
category slugs (SSD -> storage, since pc-builder.io has no separate HDD
category) and parses each category page with cheerio, skipping rows
without a resolved price (generic 'search on Amazon' links)
- node ace scrape:parts: refreshes the cache for every mapped category
(upsert on source+source_ref), runnable manually or later via cron
- GET /scraped-parts (authenticated): search the cache by name (LOWER()
LIKE, avoids a MySQL utf8_bin/utf8mb4 collation mismatch from Lucid's
whereLike) and optional typeId filter
- NewPartSearch.vue: debounced search widget wired into
components/create.vue; clicking 'Import' pre-fills the existing
component creation form (name, price, type, purchase date = today)
instead of creating the Component directly, so the user still reviews
and saves it like any manually entered part
- 3 functional tests covering search by name, filter by typeId, and
unauthenticated access being redirected to login
- Initial scrape run: 989 parts across 10 categories
Adds the ability for an owner to share a computer build via a public,
unauthenticated link, inspired by a feature request from the original
pcbuilder repo (no such feature actually existed there — implemented
from scratch here).
- New columns on computers: share_token (unique, nullable), share_enabled,
share_show_sell_price, share_show_cost
- computer_share_service.ts: generates a non-guessable share token
(crypto.randomBytes, base64url), enables/disables sharing, and supports
rotating the token to invalidate previously distributed links
- ComputerShareController (authenticated, owner-scoped like the rest of
the app): PUT /computers/:id/share to update settings, POST
/computers/:id/share/rotate to rotate the token
- PublicComputerController + GET /share/:token (outside any auth/guest
group, reachable anonymously): resolves only computers with
share_enabled = true, 404 otherwise
- ComputerPublicTransformer: dedicated read-only projection that never
leaks userId, and only includes sellPrice/cost when the owner opted in
via share_show_sell_price / share_show_cost — everything else (title,
scores, state, components, pictures) is always visible on a shared link
- ComputerShareModal.vue: owner-facing UI (toggle sharing, choose which
prices to expose, copy the public link, revoke/rotate it) wired into
computers/show.vue via a new 'Share' button
- share/show.vue: public page rendered with EmptyLayout (no authenticated
header/sidebar)
- Boolean columns use a cast since MySQL returns tinyint(1),
not native booleans, through Lucid
- 6 new functional tests covering: enabling sharing, price visibility
toggles, disabling sharing revokes the link, cross-user protection on
the share settings endpoint (404), token rotation invalidating the old
link, and anonymous access to a shared computer
Security config (3.1):
- config/session.ts: cookie.secure now follows app.inProduction instead of
being hardcoded to false
- config/shield.ts: enable CSP with baseline directives compatible with
Vite dev server and Inertia (self + @viteDevUrl/@viteHmrUrl)
Frontend typing (3.2):
- Replace 'any' props with concrete types (Computer, Component, ComponentType,
ComputerState) across computers/*.vue and components/*.vue, using the
existing inertia/pages/types/UITypes.ts definitions
- Fix UITypes.ts: export ComponentType/ComputerState, switch from interface
to type (interfaces are never assignable to Inertia's ComponentProps index
signature), drop the stale componentTypeId/componentStateId fields that
didn't match the actual Lucid schema (real field is typeId, no component
state exists)
- Add an asPageProp() boundary cast in the two controllers: Lucid's
serialize() returns a loosely typed ModelObject that isn't structurally
assignable to the concrete page prop shapes (and the inertia/** tree is
excluded from the backend tsconfig, so real types can't be imported
directly across that boundary)
UTF-8 mojibake fixes (3.3):
- Repair broken accented characters (├®/├á/├┤ -> é/à/ô, Ôé¼ -> €) left over
from a bad encoding during the GWT->Inertia migration, across
ComponentEditor.vue, ComputerComponentsTable.vue, SimpleGallery.vue,
TableComponent.vue, isAdminMode.ts and the custom directives
Misc cleanup (3.5):
- computer.ts: replace console.warn with the app logger (includes
computerId) when .cost is accessed without preloading components
- Remove dead registerValidator (Auth/register.ts), unused since signupValidator
is the one actually wired to NewAccountController
- computers/create.vue: switch to useForm() + @submit.prevent so the form
submits on Enter and shows a processing state, matching components/create.vue
- components/edit.vue: drop the 'origin' field, which was never provided by
the backend (edit() never shared it) and therefore always undefined
- Extract FilterDropdown.vue and reuse it in ComponentsViewer/ComputersViewer,
removing ~80 lines of duplicated dropdown markup/logic and fixing duplicate
HTML ids (id="used" appeared twice in ComputersViewer)
- Move MIGRATION_*.md artifacts into docs/migration-history/
- Scope update/destroy queries by userId (Computers & Components), matching
the pattern already used in index/show/edit
- Add missing auth.authenticate() calls on destroy actions
- handleComponents now requires userId and scopes component lookups to the
current user, preventing cross-user component takeover
- handleComponents rejects attaching more than one component of the same
'unique' type to a computer (new InvalidComponentSelectionException, 422)
- handlePictures scopes picture lookups by computerId
- destroy(Computer) now detaches its components (computerId = null) instead
of losing them to cascade delete, and removes the computer's uploads dir
- Switch store/update actions to request.validateUsing() for consistency
- Fix relative redirects to absolute paths (/computers, /computers/:id)
- Untrack .env.production and add it to .gitignore (secrets must not be
committed); local file kept on disk
Kevin L'Hereec