a1f6ff419a297f4ba905e453d2dbe7fb859a5887
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Kevin L'Hereec
|
6053290501 |
feat: paginate pc-builder.io scraping and always keep the best known price
The RTX 5070 case that prompted this: only 100 GPUs out of pc-builder.io's
actual 6222 were being scraped (its category pages only render the first
page server-side), so cheaper RTX 5070 variants existed on the site but
were invisible to our search. Investigating further also surfaced that
even with full pagination, most recent GPU listings (RTX 5070 included)
have no fixed Amazon price on pc-builder.io at all - just a generic
"search on Amazon" link - so pagination alone doesn't fully solve price
coverage either.
Pagination (part_scraper_service.ts):
- Reverse-engineered pc-builder.io's /products/data endpoint, which powers
the client-side DataTables widget behind every category page. Requires
a session cookie + CSRF token scraped from a fresh category page load,
and an exact field shape (columns[].data must be the literal strings
"Name"/"Add to Build", not numeric indices - a naive DataTables-shaped
request 500s otherwise).
- CATEGORY_CONFIG now tracks each category's real pc-builder.io
category_id and a per-category maxProducts depth, chosen by actual
catalog size and price-sensitivity: GPU near-full coverage (6000),
smaller/less price-volatile categories capped lower (300-800) to keep a
full `scrape:parts` run (all categories) in a reasonable time.
- ensureFreshCategory (triggered inline by a user's search when the cache
is stale) uses a much shallower QUICK_REFRESH_MAX_PRODUCTS=200 cap
instead of the category's full depth, since it runs inside the
request/response cycle of a search - full-depth coverage is only done
by the standalone `scrape:parts` command.
Always keep the best known price:
- scraped_parts.price is now nullable (migration): parts with no fixed
pc-builder.io offer are kept and searchable instead of being silently
skipped, so at least the model is discoverable even before a price is
resolved.
- New price_source / price_checked_at columns track where the currently
stored price came from ("pc-builder.io" or a merchant name from
merchant_price_service.ts).
- ScrapedPartsController#compare now persists the cheapest offer found
(via the new updatePriceIfCheaper()) when it beats the currently stored
price - so comparing prices benefits every future search/import for
that part, not just the user who triggered it. Never regresses the
price: a comparison finding a higher price than what's stored is
ignored.
- scrapeComponentType's catalog refresh applies the same "keep whichever
price is lower" rule per row, so a routine pc-builder.io re-scrape can
never overwrite a better price a comparison had already found and
persisted with a worse (or missing) one.
- Search results (GET /scraped-parts) are now ordered priced-cheapest-first,
then parts with no known price last (`ORDER BY price IS NULL, price ASC`),
so unpriced parts stay visible without pushing priced results down.
- NewPartSearch.vue displays the catalog price inline (or "No price yet"),
and the existing "Compare prices" flow now works for those unpriced
parts the same way it does for priced ones.
7 new/updated functional tests: nullable price is searchable and sorted
last, and 3 tests for updatePriceIfCheaper (cheaper offer replaces price,
pricier offer is ignored, first-ever price is accepted for a previously
unpriced part).
|
||
|
Kevin L'Hereec
|
62ec796e05 |
fix: repair broken CSP and compare live merchant prices for new parts
CSP fix:
- The previous shield.ts config used '@viteDevUrl'/'@viteHmrUrl' CSP
keywords that don't exist in this version of @adonisjs/shield (only
'@viteUrl' is registered by @adonisjs/vite, and it resolves to an empty
string here since Vite assets are proxied through AdonisJS on the same
origin). Both placeholders were sent to the browser literally, breaking
Google Fonts, the Vite HMR websocket, and pc-builder.io/favicon images.
- Replaced with concrete directives: 'self' covers same-origin Vite
assets, ws://localhost:{VITE_HMR_PORT|24678} is added only in dev for
the HMR socket, and explicit allowlists for Google Fonts
(fonts.googleapis.com/fonts.gstatic.com), pc-builder.io product images,
and Google's favicon service (www.google.com) used by the merchant
comparison UI
Real purchase links + live price comparison:
- part_scraper_service.ts: ScrapedPart.sourceUrl now stores the actual
Amazon.fr purchase link scraped from pc-builder.io's category pages
(https://www.amazon.fr/dp/{ASIN}?tag=... or a search fallback), instead
of the pc-builder.io product page, which isn't a place anyone can buy
anything from
- New merchant_price_service.ts: on-demand live price lookup across
Amazon, LDLC, Grosbill and TopAchat. Of 6 requested merchants,
Materiel.net (search requires JS/Turbo, no results via plain GET) and
Cdiscount/PCComponentes (empty body / 403) were dropped as not scrapable
without a headless browser or paid API
- Matching strategy: pc-builder.io's full product names (e.g. "AMD Ryzen 3
3200G 3.6 GHz Quad-Core Processor") return zero results on every
merchant's search engine, so we search on a short brand+series query
and then pick the specific result whose label contains the extracted
model token (e.g. "3200G"). No token match means no offer returned,
rather than risk quoting the price of a different, unrelated product
- Fixed price parsing for LDLC/Grosbill, which render prices as nested
price/sup elements for whole euros vs cents (and LDLC additionally
wraps that in a duplicate outer .price div) - naive text concatenation
was misreading "64,95" as "6495"
- New GET /scraped-parts/compare endpoint (authenticated), called on
demand per search result rather than for every row, since it makes
several outbound requests and can take a few seconds
- NewPartSearch.vue: "Compare prices" button per result shows the
cheapest merchant with its favicon and links to the actual offer;
importing a part now prefers the live cheapest price/link over
pc-builder.io's potentially stale catalog price (confirmed stale in
practice: pc-builder.io listed a Ryzen 3 3200G at 128 euros while
Amazon sells it for 66.89 euros)
- 3 new functional tests for the compare endpoint (auth required, 404 on
unknown part, 422 on invalid id)
|
||
|
Kevin L'Hereec
|
4174d92711 |
feat: track purchase site URL for new components
- New nullable purchase_url column on components - Store/update validators accept an optional purchaseUrl (must be a valid absolute URL with a protocol when provided) - components/create.vue: NewPartSearch import now also fills purchaseUrl from the scraped part's source URL (pc-builder.io product page); the field remains manually editable for second-hand parts entered by hand - components/edit.vue: purchaseUrl is editable after creation - components/show.vue: displays the purchase site as a clickable link when present - 4 new functional tests: create with/without a purchase URL, invalid URL rejected (asserts the Inertia inputErrorsBag flash, not a raw 422 - content negotiation only returns JSON for XHR/API-style requests), and updating an existing component's purchase URL |
||
|
Kevin L'Hereec
|
497ae797ca |
feat: add 24h cache freshness with auto-refresh on search
- part_scraper_service.ts: new CACHE_TTL_HOURS (24), isCategoryStale() checks the most recent scrapedAt for a given ComponentType's mapped category, ensureFreshCategory() re-scrapes it synchronously if missing or stale (errors are logged, not thrown, so search still falls back to whatever is already cached) - ScrapedPartsController#index: when a typeId filter is provided, the corresponding category cache is refreshed eagerly if stale before querying results. Free-text searches without a typeId skip this to avoid re-scraping every mapped category on every keystroke - 'node ace scrape:parts' still refreshes everything at once on demand - 4 new tests covering staleness detection (missing cache, <24h, >24h) and that unmapped types never trigger a scrape/network call |
||
|
Kevin L'Hereec
|
5854ff8ed1 |
feat: search and import new parts scraped from pc-builder.io
Idealo and PCPartPicker were evaluated and ruled out: - idealo.fr search pages return 503 (Cloudflare) - the internal Idealo GraphQL endpoint used by a public scraper (app.idealo.de/app-backend/api) is protected by Akamai Bot Manager and silently returns empty data even with mobile app headers - PCPartPicker's product list/search relies on a session-bound CSRF token refreshed per page load, against a scraping-hostile ToS pc-builder.io category pages are server-rendered, include name/price/image for ~100 products per category, and have no bot protection, so the feature is built on top of that source only. - New scraped_parts cache table (type_id, name, price, image_url, source, source_url, source_ref unique per source) populated by a batch job rather than scraped live per search, since only ~100 first results per category are available without reverse-engineering pc-builder.io's DataTables/CSRF pagination flow - part_scraper_service.ts: maps our ComponentType names to pc-builder.io category slugs (SSD -> storage, since pc-builder.io has no separate HDD category) and parses each category page with cheerio, skipping rows without a resolved price (generic 'search on Amazon' links) - node ace scrape:parts: refreshes the cache for every mapped category (upsert on source+source_ref), runnable manually or later via cron - GET /scraped-parts (authenticated): search the cache by name (LOWER() LIKE, avoids a MySQL utf8_bin/utf8mb4 collation mismatch from Lucid's whereLike) and optional typeId filter - NewPartSearch.vue: debounced search widget wired into components/create.vue; clicking 'Import' pre-fills the existing component creation form (name, price, type, purchase date = today) instead of creating the Component directly, so the user still reviews and saves it like any manually entered part - 3 functional tests covering search by name, filter by typeId, and unauthenticated access being redirected to login - Initial scrape run: 989 parts across 10 categories |
||
|
Kevin L'Hereec
|
da71d33450 |
feat: add public read-only sharing for computers
Adds the ability for an owner to share a computer build via a public, unauthenticated link, inspired by a feature request from the original pcbuilder repo (no such feature actually existed there — implemented from scratch here). - New columns on computers: share_token (unique, nullable), share_enabled, share_show_sell_price, share_show_cost - computer_share_service.ts: generates a non-guessable share token (crypto.randomBytes, base64url), enables/disables sharing, and supports rotating the token to invalidate previously distributed links - ComputerShareController (authenticated, owner-scoped like the rest of the app): PUT /computers/:id/share to update settings, POST /computers/:id/share/rotate to rotate the token - PublicComputerController + GET /share/:token (outside any auth/guest group, reachable anonymously): resolves only computers with share_enabled = true, 404 otherwise - ComputerPublicTransformer: dedicated read-only projection that never leaks userId, and only includes sellPrice/cost when the owner opted in via share_show_sell_price / share_show_cost — everything else (title, scores, state, components, pictures) is always visible on a shared link - ComputerShareModal.vue: owner-facing UI (toggle sharing, choose which prices to expose, copy the public link, revoke/rotate it) wired into computers/show.vue via a new 'Share' button - share/show.vue: public page rendered with EmptyLayout (no authenticated header/sidebar) - Boolean columns use a cast since MySQL returns tinyint(1), not native booleans, through Lucid - 6 new functional tests covering: enabling sharing, price visibility toggles, disabling sharing revokes the link, cross-user protection on the share settings endpoint (404), token rotation invalidating the old link, and anonymous access to a shared computer |
||
|
Kevin L'Hereec
|
562cea39de |
test: add functional regression tests for IDOR fixes (Phase 4)
- Add @japa/api-client devDependency (was missing despite being required
for the 'client' fixture used in functional tests)
- Wire up sessionApiClient/authApiClient/shieldApiClient plugins in
tests/bootstrap.ts so tests can authenticate (loginAs) and pass CSRF
validation (withCsrfToken)
- Wrap each functional test in a global DB transaction (withGlobalTransaction)
for isolation and rollback
- Add .env.test with SESSION_DRIVER=memory: the cookie session driver used
in dev/prod is incompatible with Japa's session test client, which needs
a store it can read/write by session id
- tests/functional/computers.spec.ts: covers owner can update; a user
cannot update/delete another user's computer (404); a user cannot
attach another user's component via componentsId (404); two components
of the same unique type cannot be attached (422); deleting a computer
detaches its components instead of destroying them
- tests/functional/components.spec.ts: covers owner can update; a user
cannot update/delete another user's component (404); an unauthenticated
request cannot delete a component (redirects to /auth/login)
All 10 tests pass, exercising the security fixes from commit
|