feat: paginate pc-builder.io scraping and always keep the best known price
Build & Deploy / build (push) Failing after 5s
Build & Deploy / docker (push) Skipped

The RTX 5070 case that prompted this: only 100 GPUs out of pc-builder.io's
actual 6222 were being scraped (its category pages only render the first
page server-side), so cheaper RTX 5070 variants existed on the site but
were invisible to our search. Investigating further also surfaced that
even with full pagination, most recent GPU listings (RTX 5070 included)
have no fixed Amazon price on pc-builder.io at all - just a generic
"search on Amazon" link - so pagination alone doesn't fully solve price
coverage either.

Pagination (part_scraper_service.ts):
- Reverse-engineered pc-builder.io's /products/data endpoint, which powers
  the client-side DataTables widget behind every category page. Requires
  a session cookie + CSRF token scraped from a fresh category page load,
  and an exact field shape (columns[].data must be the literal strings
  "Name"/"Add to Build", not numeric indices - a naive DataTables-shaped
  request 500s otherwise).
- CATEGORY_CONFIG now tracks each category's real pc-builder.io
  category_id and a per-category maxProducts depth, chosen by actual
  catalog size and price-sensitivity: GPU near-full coverage (6000),
  smaller/less price-volatile categories capped lower (300-800) to keep a
  full `scrape:parts` run (all categories) in a reasonable time.
- ensureFreshCategory (triggered inline by a user's search when the cache
  is stale) uses a much shallower QUICK_REFRESH_MAX_PRODUCTS=200 cap
  instead of the category's full depth, since it runs inside the
  request/response cycle of a search - full-depth coverage is only done
  by the standalone `scrape:parts` command.

Always keep the best known price:
- scraped_parts.price is now nullable (migration): parts with no fixed
  pc-builder.io offer are kept and searchable instead of being silently
  skipped, so at least the model is discoverable even before a price is
  resolved.
- New price_source / price_checked_at columns track where the currently
  stored price came from ("pc-builder.io" or a merchant name from
  merchant_price_service.ts).
- ScrapedPartsController#compare now persists the cheapest offer found
  (via the new updatePriceIfCheaper()) when it beats the currently stored
  price - so comparing prices benefits every future search/import for
  that part, not just the user who triggered it. Never regresses the
  price: a comparison finding a higher price than what's stored is
  ignored.
- scrapeComponentType's catalog refresh applies the same "keep whichever
  price is lower" rule per row, so a routine pc-builder.io re-scrape can
  never overwrite a better price a comparison had already found and
  persisted with a worse (or missing) one.
- Search results (GET /scraped-parts) are now ordered priced-cheapest-first,
  then parts with no known price last (`ORDER BY price IS NULL, price ASC`),
  so unpriced parts stay visible without pushing priced results down.
- NewPartSearch.vue displays the catalog price inline (or "No price yet"),
  and the existing "Compare prices" flow now works for those unpriced
  parts the same way it does for priced ones.

7 new/updated functional tests: nullable price is searchable and sorted
last, and 3 tests for updatePriceIfCheaper (cheaper offer replaces price,
pricier offer is ignored, first-ever price is accepted for a previously
unpriced part).
This commit is contained in:
Kevin L'Hereec committed 2026-07-23 16:12:26 +02:00
1 parent 9702d24a01
commit 6053290501
8 files changed
+418 -83

No files matched your search

+67 -2
View File
@@ -6,6 +6,7 @@ import ScrapedPart from '#models/scraped_part'
import {
isCategoryStale,
ensureFreshCategory,
updatePriceIfCheaper,
CACHE_TTL_HOURS,
} from '#services/part_scraper_service'
@@ -21,11 +22,16 @@ async function createType(name: string, unique = true) {
return ComponentType.firstOrCreate({ name }, { name, unique })
}
async function createScrapedPart(typeId: number, name: string, sourceRef: string) {
async function createScrapedPart(
typeId: number,
name: string,
sourceRef: string,
price: number | null = 199.99
) {
return ScrapedPart.create({
typeId,
name,
price: 199.99,
price,
imageUrl: 'https://fr.pc-builder.io/images/cpu/example.jpg',
source: 'pc-builder.io',
sourceUrl: `https://fr.pc-builder.io/cpu/${sourceRef}`,
@@ -72,6 +78,24 @@ test.group('Scraped parts search', () => {
response.assertRedirectsTo('/auth/login')
})
test('parts without a known price are kept and sorted last', async ({ client, assert }) => {
const user = await createUser('scraped-search-c@example.com')
const type = await createType('Graphic Card-scraped-3')
await createScrapedPart(type.id, 'ZZZ-Test-RTX Cheap', 'rtx-cheap-p1', 300)
await createScrapedPart(type.id, 'ZZZ-Test-RTX Expensive', 'rtx-expensive-p2', 900)
await createScrapedPart(type.id, 'ZZZ-Test-RTX NoPrice', 'rtx-noprice-p3', null)
const response = await client.get('/scraped-parts?q=ZZZ-Test-RTX').loginAs(user)
response.assertStatus(200)
const body = response.body()
assert.lengthOf(body, 3)
assert.equal(body[0].name, 'ZZZ-Test-RTX Cheap')
assert.equal(body[1].name, 'ZZZ-Test-RTX Expensive')
assert.equal(body[2].name, 'ZZZ-Test-RTX NoPrice')
assert.isNull(body[2].price)
})
})
test.group('Scraped parts 24h cache freshness', () => {
@@ -150,3 +174,44 @@ test.group('Scraped parts merchant price comparison', () => {
response.assertStatus(422)
})
})
test.group('Best-known-price persistence', () => {
test('a cheaper offer replaces the current price', async ({ assert }) => {
const type = await createType('Graphic Card-price-1')
const part = await createScrapedPart(type.id, 'ZZZ-Test-Price-A', 'price-a-p1', 500)
await updatePriceIfCheaper(part, { merchant: 'LDLC', price: 350, url: 'https://ldlc.com/x' })
await part.refresh()
assert.equal(part.price, 350)
assert.equal(part.priceSource, 'LDLC')
assert.equal(part.sourceUrl, 'https://ldlc.com/x')
assert.isNotNull(part.priceCheckedAt)
})
test('a more expensive offer does not replace the current price', async ({ assert }) => {
const type = await createType('Graphic Card-price-2')
const part = await createScrapedPart(type.id, 'ZZZ-Test-Price-B', 'price-b-p1', 300)
await updatePriceIfCheaper(part, { merchant: 'LDLC', price: 450, url: 'https://ldlc.com/y' })
await part.refresh()
assert.equal(part.price, 300)
assert.isNull(part.priceSource)
})
test('an offer is accepted when no price was known yet', async ({ assert }) => {
const type = await createType('Graphic Card-price-3')
const part = await createScrapedPart(type.id, 'ZZZ-Test-Price-C', 'price-c-p1', null)
await updatePriceIfCheaper(part, {
merchant: 'Grosbill',
price: 599.99,
url: 'https://grosbill.com/z',
})
await part.refresh()
assert.equal(part.price, 599.99)
assert.equal(part.priceSource, 'Grosbill')
})
})