Kevin L'Hereec
54c5e37766
RTX 5060 investigation: confirmed via pc-builder.io's own server-side
search (search_query=RTX 5060 against the full 6222-product GPU catalog,
0 results) that it simply isn't in pc-builder.io's catalog yet - not a
scraping bug, the source data doesn't have it. However, while checking
whether the other merchants had it in stock, found a real matching bug:
labelMatchesModelToken bug: extractModelToken("...RTX 5060 Ti...") only
returned "5060", identical to extractModelToken("...RTX 5060...") without
Ti. Every merchant lookup for either card matched on plain "5060" and
could silently return the *other* card's price (confirmed live: LDLC and
Grosbill returned the exact same product/price for both "RTX 5060" and
"RTX 5060 Ti" queries before this fix).
Fixed by:
- extractModelToken now appends a trailing suffix token (Ti, Super, XT,
XTX) when it immediately follows the model number, so "5060 Ti" and
"5060" are extracted as distinct tokens
- New labelMatchesModelToken() replaces the naive
label.includes(modelToken) check everywhere: a bare token ("5060") no
longer matches a label that actually has a suffix right after it
("5060 Ti"), while a suffixed token ("5060 Ti") only matches labels
carrying that exact suffix
- 9 new unit tests covering extraction and matching, including the
regression case (RTX 5060 must not match "RTX 5060 Ti" labels)
Verified fix live: LDLC/Grosbill now return correctly distinct prices for
"RTX 5060" (369.95/364.99) vs "RTX 5060 Ti" (489.95/329.99).
New manual_part_seeder_service.ts: since pc-builder.io's catalog lags
behind real product releases, this seeds a small manually-tracked list of
recent parts it doesn't carry yet (currently RTX 5060 8GB and 5060 Ti
16GB), resolving their price live from the already-supported merchants
rather than hardcoding a number that would go stale. `scrape:parts` now
runs this after the regular catalog scrape. Skips an entry rather than
inserting a guessed price if no merchant has it in stock.
Also fixes a real bug hit while testing the seeder: scraped_parts and
components.purchase_url were varchar(255), which truncates/rejects
Grosbill's long tracking-redirect URLs (500+ characters with encoded
analytics params) - both widened to TEXT via a new migration.
83 lines
2.8 KiB
TypeScript
83 lines
2.8 KiB
TypeScript
import { DateTime } from 'luxon'
|
|
import logger from '@adonisjs/core/services/logger'
|
|
import ComponentType from '#models/component_type'
|
|
import ScrapedPart from '#models/scraped_part'
|
|
import { findCheapestOffers } from '#services/merchant_price_service'
|
|
|
|
const SOURCE = 'manual'
|
|
|
|
/**
|
|
* Very recent parts pc-builder.io doesn't carry yet (its catalog appears
|
|
* to lag several months behind actual product releases — confirmed via
|
|
* its own server-side search returning 0 results for "RTX 5060"). These
|
|
* are seeded manually so they're at least searchable/importable, with
|
|
* their price resolved live from the merchants merchant_price_service.ts
|
|
* already supports (LDLC/Grosbill both carry the RTX 5060 series, for
|
|
* example) rather than a hardcoded number that would go stale immediately.
|
|
*
|
|
* sourceRef must be stable and unique per entry (used as the upsert key,
|
|
* like pc-builder.io's own product ID), and is namespaced under
|
|
* "manual-" to guarantee it can never collide with a numeric
|
|
* pc-builder.io data-id.
|
|
*/
|
|
const MANUAL_PARTS: { typeName: string; name: string; sourceRef: string }[] = [
|
|
{
|
|
typeName: 'Graphic Card',
|
|
name: 'NVIDIA GeForce RTX 5060 8 GB Video Card',
|
|
sourceRef: 'manual-rtx-5060-8gb',
|
|
},
|
|
{
|
|
typeName: 'Graphic Card',
|
|
name: 'NVIDIA GeForce RTX 5060 Ti 16 GB Video Card',
|
|
sourceRef: 'manual-rtx-5060-ti-16gb',
|
|
},
|
|
]
|
|
|
|
/**
|
|
* Upserts every entry in MANUAL_PARTS, resolving each one's price live
|
|
* from the supported merchants (Amazon/LDLC/Grosbill/...). An entry with
|
|
* no ComponentType match in the current database, or no merchant offer
|
|
* found at all, is skipped rather than inserted with a null/guessed
|
|
* price — this only fills in real, verified prices.
|
|
*/
|
|
async function seedManualParts(): Promise<{ inserted: number; skipped: number }> {
|
|
let inserted = 0
|
|
let skipped = 0
|
|
|
|
for (const manualPart of MANUAL_PARTS) {
|
|
const componentType = await ComponentType.findBy('name', manualPart.typeName)
|
|
if (!componentType) {
|
|
logger.warn({ part: manualPart.name }, 'No matching ComponentType, skipping manual part')
|
|
skipped++
|
|
continue
|
|
}
|
|
|
|
const offers = await findCheapestOffers(manualPart.name, null)
|
|
if (offers.length === 0) {
|
|
logger.warn({ part: manualPart.name }, 'No merchant offer found, skipping manual part')
|
|
skipped++
|
|
continue
|
|
}
|
|
|
|
const cheapest = offers[0]
|
|
await ScrapedPart.updateOrCreate(
|
|
{ source: SOURCE, sourceRef: manualPart.sourceRef },
|
|
{
|
|
typeId: componentType.id,
|
|
name: manualPart.name,
|
|
imageUrl: null,
|
|
price: cheapest.price,
|
|
sourceUrl: cheapest.url,
|
|
priceSource: cheapest.merchant,
|
|
priceCheckedAt: DateTime.now(),
|
|
scrapedAt: DateTime.now(),
|
|
}
|
|
)
|
|
inserted++
|
|
}
|
|
|
|
return { inserted, skipped }
|
|
}
|
|
|
|
export { seedManualParts, MANUAL_PARTS }
|