- part_scraper_service.ts: new CACHE_TTL_HOURS (24), isCategoryStale()
checks the most recent scrapedAt for a given ComponentType's mapped
category, ensureFreshCategory() re-scrapes it synchronously if missing
or stale (errors are logged, not thrown, so search still falls back to
whatever is already cached)
- ScrapedPartsController#index: when a typeId filter is provided, the
corresponding category cache is refreshed eagerly if stale before
querying results. Free-text searches without a typeId skip this to
avoid re-scraping every mapped category on every keystroke -
'node ace scrape:parts' still refreshes everything at once on demand
- 4 new tests covering staleness detection (missing cache, <24h, >24h)
and that unmapped types never trigger a scrape/network call
Idealo and PCPartPicker were evaluated and ruled out:
- idealo.fr search pages return 503 (Cloudflare)
- the internal Idealo GraphQL endpoint used by a public scraper
(app.idealo.de/app-backend/api) is protected by Akamai Bot Manager and
silently returns empty data even with mobile app headers
- PCPartPicker's product list/search relies on a session-bound CSRF token
refreshed per page load, against a scraping-hostile ToS
pc-builder.io category pages are server-rendered, include name/price/image
for ~100 products per category, and have no bot protection, so the feature
is built on top of that source only.
- New scraped_parts cache table (type_id, name, price, image_url, source,
source_url, source_ref unique per source) populated by a batch job
rather than scraped live per search, since only ~100 first results per
category are available without reverse-engineering pc-builder.io's
DataTables/CSRF pagination flow
- part_scraper_service.ts: maps our ComponentType names to pc-builder.io
category slugs (SSD -> storage, since pc-builder.io has no separate HDD
category) and parses each category page with cheerio, skipping rows
without a resolved price (generic 'search on Amazon' links)
- node ace scrape:parts: refreshes the cache for every mapped category
(upsert on source+source_ref), runnable manually or later via cron
- GET /scraped-parts (authenticated): search the cache by name (LOWER()
LIKE, avoids a MySQL utf8_bin/utf8mb4 collation mismatch from Lucid's
whereLike) and optional typeId filter
- NewPartSearch.vue: debounced search widget wired into
components/create.vue; clicking 'Import' pre-fills the existing
component creation form (name, price, type, purchase date = today)
instead of creating the Component directly, so the user still reviews
and saves it like any manually entered part
- 3 functional tests covering search by name, filter by typeId, and
unauthenticated access being redirected to login
- Initial scrape run: 989 parts across 10 categories
Kevin L'Hereec