Files
Kevin L'Hereec 6053290501
Build & Deploy / build (push) Failing after 5s
Build & Deploy / docker (push) Skipped
feat: paginate pc-builder.io scraping and always keep the best known price
The RTX 5070 case that prompted this: only 100 GPUs out of pc-builder.io's
actual 6222 were being scraped (its category pages only render the first
page server-side), so cheaper RTX 5070 variants existed on the site but
were invisible to our search. Investigating further also surfaced that
even with full pagination, most recent GPU listings (RTX 5070 included)
have no fixed Amazon price on pc-builder.io at all - just a generic
"search on Amazon" link - so pagination alone doesn't fully solve price
coverage either.

Pagination (part_scraper_service.ts):
- Reverse-engineered pc-builder.io's /products/data endpoint, which powers
  the client-side DataTables widget behind every category page. Requires
  a session cookie + CSRF token scraped from a fresh category page load,
  and an exact field shape (columns[].data must be the literal strings
  "Name"/"Add to Build", not numeric indices - a naive DataTables-shaped
  request 500s otherwise).
- CATEGORY_CONFIG now tracks each category's real pc-builder.io
  category_id and a per-category maxProducts depth, chosen by actual
  catalog size and price-sensitivity: GPU near-full coverage (6000),
  smaller/less price-volatile categories capped lower (300-800) to keep a
  full `scrape:parts` run (all categories) in a reasonable time.
- ensureFreshCategory (triggered inline by a user's search when the cache
  is stale) uses a much shallower QUICK_REFRESH_MAX_PRODUCTS=200 cap
  instead of the category's full depth, since it runs inside the
  request/response cycle of a search - full-depth coverage is only done
  by the standalone `scrape:parts` command.

Always keep the best known price:
- scraped_parts.price is now nullable (migration): parts with no fixed
  pc-builder.io offer are kept and searchable instead of being silently
  skipped, so at least the model is discoverable even before a price is
  resolved.
- New price_source / price_checked_at columns track where the currently
  stored price came from ("pc-builder.io" or a merchant name from
  merchant_price_service.ts).
- ScrapedPartsController#compare now persists the cheapest offer found
  (via the new updatePriceIfCheaper()) when it beats the currently stored
  price - so comparing prices benefits every future search/import for
  that part, not just the user who triggered it. Never regresses the
  price: a comparison finding a higher price than what's stored is
  ignored.
- scrapeComponentType's catalog refresh applies the same "keep whichever
  price is lower" rule per row, so a routine pc-builder.io re-scrape can
  never overwrite a better price a comparison had already found and
  persisted with a worse (or missing) one.
- Search results (GET /scraped-parts) are now ordered priced-cheapest-first,
  then parts with no known price last (`ORDER BY price IS NULL, price ASC`),
  so unpriced parts stay visible without pushing priced results down.
- NewPartSearch.vue displays the catalog price inline (or "No price yet"),
  and the existing "Compare prices" flow now works for those unpriced
  parts the same way it does for priced ones.

7 new/updated functional tests: nullable price is searchable and sorted
last, and 3 tests for updatePriceIfCheaper (cheaper offer replaces price,
pricier offer is ignored, first-ever price is accepted for a previously
unpriced part).
2026-07-23 16:12:26 +02:00

127 lines
4.3 KiB
Vue

<script setup lang="ts">
import { reactive } from 'vue'
import { Link } from '@adonisjs/inertia/vue'
import { useForm } from '@inertiajs/vue3'
import CustomSelect from '~/pages/widgets/CustomSelect.vue'
import NewPartSearch from '~/pages/widgets/NewPartSearch.vue'
import type { ComponentType, MerchantOffer, ScrapedPart } from '~/pages/types/UITypes'
defineProps<{ types: ComponentType[]; errors?: Record<string, string> }>()
const form = reactive({
name: '',
price: 0,
typeId: null as number | null,
purchaseDate: new Date().toISOString().split('T')[0],
purchaseUrl: '',
quantity: 1,
})
function importPart(part: ScrapedPart, cheapestOffer: MerchantOffer | null) {
form.name = part.name
// Prefer the live price/link found by comparing merchants (Amazon, LDLC,
// Grosbill, TopAchat, ...) when available, since pc-builder.io's catalog
// price can be stale (or simply unknown — see ScrapedPart.price doc);
// fall back to the catalog price/link, or leave the price field as-is
// for the user to fill in manually if neither is known.
form.price = cheapestOffer?.price ?? part.price ?? form.price
form.typeId = part.type.id
form.purchaseDate = new Date().toISOString().split('T')[0]
form.purchaseUrl = cheapestOffer?.url ?? part.sourceUrl
}
function submitForm() {
const inertiaForm = useForm(form)
inertiaForm.post('/components')
}
</script>
<template>
<div class="max-w-2xl mx-auto p-8 bg-white shadow-lg rounded-lg">
<h1 class="text-2xl font-semibold mb-8 text-gray-800">Create New Component</h1>
<NewPartSearch class="mb-8" @import="importPart" />
<form @submit.prevent="submitForm" class="space-y-6">
<div>
<label class="block text-gray-700 text-sm font-bold mb-2" for="name">Name</label>
<input
name="name"
v-model="form.name"
class="w-full mt-2 border-gray-200 focus:border-indigo-600 focus:ring focus:ring-indigo-500/40 px-3 py-2 border rounded text-gray-700"
type="text"
/>
</div>
<div>
<label class="block text-gray-700 text-sm font-bold mb-2" for="price">Price</label>
<input
name="price"
v-model.number="form.price"
class="w-full mt-2 border-gray-200 focus:border-indigo-600 focus:ring focus:ring-indigo-500/40 px-3 py-2 border rounded text-gray-700"
type="number"
step="0.01"
min="0"
/>
</div>
<div>
<label class="block text-gray-700 text-sm font-bold mb-2" for="type">Type</label>
<CustomSelect v-model="form.typeId" :options="types" placeholder="Choose a type" />
</div>
<div>
<label class="block text-gray-700 text-sm font-bold mb-2" for="purchaseDate"
>Purchase Date</label
>
<input
name="purchaseDate"
v-model="form.purchaseDate"
class="w-full mt-2 border-gray-200 focus:border-indigo-600 focus:ring focus:ring-indigo-500/40 border rounded text-gray-700 px-3 py-2"
type="date"
/>
</div>
<div>
<label class="block text-gray-700 text-sm font-bold mb-2" for="purchaseUrl"
>Purchase Site (optional, filled automatically when importing a new part)</label
>
<input
name="purchaseUrl"
v-model="form.purchaseUrl"
class="w-full mt-2 border-gray-200 focus:border-indigo-600 focus:ring focus:ring-indigo-500/40 px-3 py-2 border rounded text-gray-700"
type="url"
placeholder="https://..."
/>
</div>
<div>
<label class="block text-gray-700 text-sm font-bold mb-2" for="quantity">Quantity</label>
<input
name="quantity"
v-model.number="form.quantity"
class="w-full mt-2 border-gray-200 focus:border-indigo-600 focus:ring focus:ring-indigo-500/40 px-3 py-2 border rounded text-gray-700"
type="number"
min="1"
/>
</div>
<div class="flex gap-4">
<button
type="submit"
class="px-4 py-2 bg-indigo-600 text-white rounded hover:bg-indigo-700 transition"
>
Create
</button>
<Link
href="/components"
class="px-4 py-2 bg-gray-300 text-gray-800 rounded hover:bg-gray-400 transition"
>
Cancel
</Link>
</div>
</form>
</div>
</template>
<style scoped></style>