Kevin L'Hereec
5854ff8ed1
Idealo and PCPartPicker were evaluated and ruled out: - idealo.fr search pages return 503 (Cloudflare) - the internal Idealo GraphQL endpoint used by a public scraper (app.idealo.de/app-backend/api) is protected by Akamai Bot Manager and silently returns empty data even with mobile app headers - PCPartPicker's product list/search relies on a session-bound CSRF token refreshed per page load, against a scraping-hostile ToS pc-builder.io category pages are server-rendered, include name/price/image for ~100 products per category, and have no bot protection, so the feature is built on top of that source only. - New scraped_parts cache table (type_id, name, price, image_url, source, source_url, source_ref unique per source) populated by a batch job rather than scraped live per search, since only ~100 first results per category are available without reverse-engineering pc-builder.io's DataTables/CSRF pagination flow - part_scraper_service.ts: maps our ComponentType names to pc-builder.io category slugs (SSD -> storage, since pc-builder.io has no separate HDD category) and parses each category page with cheerio, skipping rows without a resolved price (generic 'search on Amazon' links) - node ace scrape:parts: refreshes the cache for every mapped category (upsert on source+source_ref), runnable manually or later via cron - GET /scraped-parts (authenticated): search the cache by name (LOWER() LIKE, avoids a MySQL utf8_bin/utf8mb4 collation mismatch from Lucid's whereLike) and optional typeId filter - NewPartSearch.vue: debounced search widget wired into components/create.vue; clicking 'Import' pre-fills the existing component creation form (name, price, type, purchase date = today) instead of creating the Component directly, so the user still reviews and saves it like any manually entered part - 3 functional tests covering search by name, filter by typeId, and unauthenticated access being redirected to login - Initial scrape run: 989 parts across 10 categories
35 lines
1.0 KiB
TypeScript
35 lines
1.0 KiB
TypeScript
import type { HttpContext } from '@adonisjs/core/http'
|
|
import vine from '@vinejs/vine'
|
|
import ScrapedPart from '#models/scraped_part'
|
|
|
|
const searchValidator = vine.compile(
|
|
vine.object({
|
|
q: vine.string().trim().minLength(1).optional(),
|
|
typeId: vine.number().min(1).optional(),
|
|
})
|
|
)
|
|
|
|
export default class ScrapedPartsController {
|
|
/**
|
|
* Search the new-parts cache (scraped from pc-builder.io) by name and/or
|
|
* component type. Read-only, authenticated (the search UI lives inside
|
|
* the authenticated "create component" flow).
|
|
*/
|
|
async index({ request, auth, response }: HttpContext) {
|
|
await auth.authenticate()
|
|
const { q, typeId } = await request.validateUsing(searchValidator)
|
|
|
|
const query = ScrapedPart.query().preload('type').orderBy('name')
|
|
|
|
if (q) {
|
|
query.whereRaw('LOWER(name) LIKE ?', [`%${q.toLowerCase()}%`])
|
|
}
|
|
if (typeId) {
|
|
query.where('typeId', typeId)
|
|
}
|
|
|
|
const parts = await query.limit(30)
|
|
return response.json(parts.map((part) => part.serialize()))
|
|
}
|
|
}
|