7b9a17f395
fix(connector): reduce ThreadPoolExecutor max_workers
2026-04-16 16:37:27 +01:00
703a7c435c
fix(youtube_api): video search capped at 50
2026-04-14 17:54:43 +01:00
02ba727d05
chore(connector): add buffer to ratelimit reset
2026-04-14 17:41:09 +01:00
e35e51d295
fix(reddit_api): handle rate limit wait time conversion error
2026-04-14 17:35:21 +01:00
ec64551881
fix(connectors): update User-Agent header for BoardsAPI
2026-04-08 19:34:30 +01:00
6efa75dfe6
chore(connectors): reduce aggressive parallel connections to boards.ie
2026-04-04 12:33:06 +01:00
de61e7653f
perf(connector): add reddit API authentication to speed up fetching
...
This aligns better with ethics and massively increases rate limits.
2026-04-04 12:26:54 +01:00
98aa04256b
fix(reddit_api): fix reddit ratelimit check
2026-04-04 10:20:48 +01:00
376773a0cc
style: run python linter & prettifier on backend code
2026-03-25 19:34:43 +00:00
5b0441c34b
fix(connector): unnecessary comment limits
...
In addition, I made some methods private to better align with the BaseConnector parent class.
2026-03-14 21:53:13 +00:00
2a00795cc2
chore(connectors): implement category_exists for Boards API
2026-03-14 17:11:49 +00:00
8a423b2a29
feat(connectors): implement category validation in scraping process
2026-03-14 16:59:43 +00:00
d96f459104
fix(connectors): update URL references to use base_url in BoardsAPI
2026-03-13 21:59:17 +00:00
c12f1b4371
chore(connectors): add category and search validation fields
2026-03-12 09:56:34 +00:00
01d6bd0164
fix(connectors): category / search fields breaking
...
Ideally category and search are fully optional, however some sites break if one or the other is not provided.
Unfortuntely `boards.ie` has a different page type for searches and I'm not bothered to implement a scraper from scratch.
In addition, removed comment limit options.
2026-03-11 21:16:26 +00:00
eff416c34e
fix(connectors): hardcoded source name in Youtube connector
2026-03-10 23:36:09 +00:00
6ec47256d0
feat(api): add database scraping endpoints
2026-03-10 19:04:33 +00:00
17bd4702b2
fix(connectors): connector detectors returning name of ID alongside connector obj
2026-03-10 18:36:40 +00:00
5ccb2e73cd
fix(connectors): incorrect registry location
...
Registry paths were using the incorrect connector path locations.
2026-03-10 18:18:42 +00:00
2a8d7c7972
refactor(connectors): Youtube & Reddit connectors implement BaseConnector
2026-03-10 18:11:33 +00:00
e7a8c17be4
chore(connectors): add base connector inheritance
2026-03-10 18:08:01 +00:00
cc799f7368
feat(connectors): add base connector and registry for detection
...
Idea is to have a "plugin-type" system, where new connectors can extend the `BaseConnector` class and implement the fetch posts method.
These are automatically detected by the registry, and automatically used in new Flask endpoints that give a list of possible sources.
Allows for an open-ended system where new data scrapers / API consumers can be added dynamically.
2026-03-09 21:29:03 +00:00
ca444e9cb0
refactor: move connectors to backend dir
...
They will now be more used in the backend.
2026-03-09 20:53:13 +00:00
