Commit Graph

12 Commits

Author SHA1 Message Date
7b9a17f395 fix(connector): reduce ThreadPoolExecutor max_workers 2026-04-16 16:37:27 +01:00
ec64551881 fix(connectors): update User-Agent header for BoardsAPI 2026-04-08 19:34:30 +01:00
6efa75dfe6 chore(connectors): reduce aggressive parallel connections to boards.ie 2026-04-04 12:33:06 +01:00
376773a0cc style: run python linter & prettifier on backend code 2026-03-25 19:34:43 +00:00
2a00795cc2 chore(connectors): implement category_exists for Boards API 2026-03-14 17:11:49 +00:00
8a423b2a29 feat(connectors): implement category validation in scraping process 2026-03-14 16:59:43 +00:00
d96f459104 fix(connectors): update URL references to use base_url in BoardsAPI 2026-03-13 21:59:17 +00:00
c12f1b4371 chore(connectors): add category and search validation fields 2026-03-12 09:56:34 +00:00
01d6bd0164 fix(connectors): category / search fields breaking
Ideally category and search are fully optional, however some sites break if one or the other is not provided.

Unfortuntely `boards.ie` has a different page type for searches and I'm not bothered to implement a scraper from scratch.

In addition, removed comment limit options.
2026-03-11 21:16:26 +00:00
5ccb2e73cd fix(connectors): incorrect registry location
Registry paths were using the incorrect connector path locations.
2026-03-10 18:18:42 +00:00
cc799f7368 feat(connectors): add base connector and registry for detection
Idea is to have a "plugin-type" system, where new connectors can extend the `BaseConnector` class and implement the fetch posts method.

These are automatically detected by the registry, and automatically used in new Flask endpoints that give a list of possible sources.

Allows for an open-ended system where new data scrapers / API consumers can be added dynamically.
2026-03-09 21:29:03 +00:00
ca444e9cb0 refactor: move connectors to backend dir
They will now be more used in the backend.
2026-03-09 20:53:13 +00:00