feat: paginate pc-builder.io scraping and always keep the best known price
The RTX 5070 case that prompted this: only 100 GPUs out of pc-builder.io's
actual 6222 were being scraped (its category pages only render the first
page server-side), so cheaper RTX 5070 variants existed on the site but
were invisible to our search. Investigating further also surfaced that
even with full pagination, most recent GPU listings (RTX 5070 included)
have no fixed Amazon price on pc-builder.io at all - just a generic
"search on Amazon" link - so pagination alone doesn't fully solve price
coverage either.
Pagination (part_scraper_service.ts):
- Reverse-engineered pc-builder.io's /products/data endpoint, which powers
the client-side DataTables widget behind every category page. Requires
a session cookie + CSRF token scraped from a fresh category page load,
and an exact field shape (columns[].data must be the literal strings
"Name"/"Add to Build", not numeric indices - a naive DataTables-shaped
request 500s otherwise).
- CATEGORY_CONFIG now tracks each category's real pc-builder.io
category_id and a per-category maxProducts depth, chosen by actual
catalog size and price-sensitivity: GPU near-full coverage (6000),
smaller/less price-volatile categories capped lower (300-800) to keep a
full `scrape:parts` run (all categories) in a reasonable time.
- ensureFreshCategory (triggered inline by a user's search when the cache
is stale) uses a much shallower QUICK_REFRESH_MAX_PRODUCTS=200 cap
instead of the category's full depth, since it runs inside the
request/response cycle of a search - full-depth coverage is only done
by the standalone `scrape:parts` command.
Always keep the best known price:
- scraped_parts.price is now nullable (migration): parts with no fixed
pc-builder.io offer are kept and searchable instead of being silently
skipped, so at least the model is discoverable even before a price is
resolved.
- New price_source / price_checked_at columns track where the currently
stored price came from ("pc-builder.io" or a merchant name from
merchant_price_service.ts).
- ScrapedPartsController#compare now persists the cheapest offer found
(via the new updatePriceIfCheaper()) when it beats the currently stored
price - so comparing prices benefits every future search/import for
that part, not just the user who triggered it. Never regresses the
price: a comparison finding a higher price than what's stored is
ignored.
- scrapeComponentType's catalog refresh applies the same "keep whichever
price is lower" rule per row, so a routine pc-builder.io re-scrape can
never overwrite a better price a comparison had already found and
persisted with a worse (or missing) one.
- Search results (GET /scraped-parts) are now ordered priced-cheapest-first,
then parts with no known price last (`ORDER BY price IS NULL, price ASC`),
so unpriced parts stay visible without pushing priced results down.
- NewPartSearch.vue displays the catalog price inline (or "No price yet"),
and the existing "Compare prices" flow now works for those unpriced
parts the same way it does for priced ones.
7 new/updated functional tests: nullable price is searchable and sorted
last, and 3 tests for updatePriceIfCheaper (cheaper offer replaces price,
pricier offer is ignored, first-ever price is accepted for a previously
unpriced part).
This commit is contained in:
1 parent
9702d24a01
commit
6053290501
8 files changed
+418
-83
No files matched your search
@@ -6,6 +6,7 @@ import ScrapedPart from '#models/scraped_part'
|
||||
import {
|
||||
isCategoryStale,
|
||||
ensureFreshCategory,
|
||||
updatePriceIfCheaper,
|
||||
CACHE_TTL_HOURS,
|
||||
} from '#services/part_scraper_service'
|
||||
|
||||
@@ -21,11 +22,16 @@ async function createType(name: string, unique = true) {
|
||||
return ComponentType.firstOrCreate({ name }, { name, unique })
|
||||
}
|
||||
|
||||
async function createScrapedPart(typeId: number, name: string, sourceRef: string) {
|
||||
async function createScrapedPart(
|
||||
typeId: number,
|
||||
name: string,
|
||||
sourceRef: string,
|
||||
price: number | null = 199.99
|
||||
) {
|
||||
return ScrapedPart.create({
|
||||
typeId,
|
||||
name,
|
||||
price: 199.99,
|
||||
price,
|
||||
imageUrl: 'https://fr.pc-builder.io/images/cpu/example.jpg',
|
||||
source: 'pc-builder.io',
|
||||
sourceUrl: `https://fr.pc-builder.io/cpu/${sourceRef}`,
|
||||
@@ -72,6 +78,24 @@ test.group('Scraped parts search', () => {
|
||||
|
||||
response.assertRedirectsTo('/auth/login')
|
||||
})
|
||||
|
||||
test('parts without a known price are kept and sorted last', async ({ client, assert }) => {
|
||||
const user = await createUser('scraped-search-c@example.com')
|
||||
const type = await createType('Graphic Card-scraped-3')
|
||||
await createScrapedPart(type.id, 'ZZZ-Test-RTX Cheap', 'rtx-cheap-p1', 300)
|
||||
await createScrapedPart(type.id, 'ZZZ-Test-RTX Expensive', 'rtx-expensive-p2', 900)
|
||||
await createScrapedPart(type.id, 'ZZZ-Test-RTX NoPrice', 'rtx-noprice-p3', null)
|
||||
|
||||
const response = await client.get('/scraped-parts?q=ZZZ-Test-RTX').loginAs(user)
|
||||
|
||||
response.assertStatus(200)
|
||||
const body = response.body()
|
||||
assert.lengthOf(body, 3)
|
||||
assert.equal(body[0].name, 'ZZZ-Test-RTX Cheap')
|
||||
assert.equal(body[1].name, 'ZZZ-Test-RTX Expensive')
|
||||
assert.equal(body[2].name, 'ZZZ-Test-RTX NoPrice')
|
||||
assert.isNull(body[2].price)
|
||||
})
|
||||
})
|
||||
|
||||
test.group('Scraped parts 24h cache freshness', () => {
|
||||
@@ -150,3 +174,44 @@ test.group('Scraped parts merchant price comparison', () => {
|
||||
response.assertStatus(422)
|
||||
})
|
||||
})
|
||||
|
||||
test.group('Best-known-price persistence', () => {
|
||||
test('a cheaper offer replaces the current price', async ({ assert }) => {
|
||||
const type = await createType('Graphic Card-price-1')
|
||||
const part = await createScrapedPart(type.id, 'ZZZ-Test-Price-A', 'price-a-p1', 500)
|
||||
|
||||
await updatePriceIfCheaper(part, { merchant: 'LDLC', price: 350, url: 'https://ldlc.com/x' })
|
||||
await part.refresh()
|
||||
|
||||
assert.equal(part.price, 350)
|
||||
assert.equal(part.priceSource, 'LDLC')
|
||||
assert.equal(part.sourceUrl, 'https://ldlc.com/x')
|
||||
assert.isNotNull(part.priceCheckedAt)
|
||||
})
|
||||
|
||||
test('a more expensive offer does not replace the current price', async ({ assert }) => {
|
||||
const type = await createType('Graphic Card-price-2')
|
||||
const part = await createScrapedPart(type.id, 'ZZZ-Test-Price-B', 'price-b-p1', 300)
|
||||
|
||||
await updatePriceIfCheaper(part, { merchant: 'LDLC', price: 450, url: 'https://ldlc.com/y' })
|
||||
await part.refresh()
|
||||
|
||||
assert.equal(part.price, 300)
|
||||
assert.isNull(part.priceSource)
|
||||
})
|
||||
|
||||
test('an offer is accepted when no price was known yet', async ({ assert }) => {
|
||||
const type = await createType('Graphic Card-price-3')
|
||||
const part = await createScrapedPart(type.id, 'ZZZ-Test-Price-C', 'price-c-p1', null)
|
||||
|
||||
await updatePriceIfCheaper(part, {
|
||||
merchant: 'Grosbill',
|
||||
price: 599.99,
|
||||
url: 'https://grosbill.com/z',
|
||||
})
|
||||
await part.refresh()
|
||||
|
||||
assert.equal(part.price, 599.99)
|
||||
assert.equal(part.priceSource, 'Grosbill')
|
||||
})
|
||||
})
|
||||
Reference in new issue
Block a user