dylan/crosspost
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Automatic Scraping of dataset options #9

Merged
dylan merged 36 commits from feat/automatic-scraping-datasets into main 2026-03-14 21:58:49 +00:00
Conversation 0 Commits 36 Files Changed 20 +885 -203
dylan commented 2026-03-09 20:53:41 +00:00
Owner
Copy Link

They will now be more used in the backend.

They will now be more used in the backend.
dylan added 1 commit 2026-03-09 20:53:41 +00:00
refactor: move connectors to backend dir ca444e9cb0 
They will now be more used in the backend.
dylan added 1 commit 2026-03-09 20:55:15 +00:00
refactor(api): rename /upload endpoint 262a70dbf3 
Ensures consistency with the other dataset-based endpoints and follows the REST-API rules more cleanly.
dylan added 1 commit 2026-03-09 21:29:11 +00:00
feat(connectors): add base connector and registry for detection cc799f7368 
Idea is to have a "plugin-type" system, where new connectors can extend the `BaseConnector` class and implement the fetch posts method.

These are automatically detected by the registry, and automatically used in new Flask endpoints that give a list of possible sources.

Allows for an open-ended system where new data scrapers / API consumers can be added dynamically.
dylan added 1 commit 2026-03-10 18:08:03 +00:00
chore(connectors): add base connector inheritance e7a8c17be4
dylan added 1 commit 2026-03-10 18:11:37 +00:00
refactor(connectors): Youtube & Reddit connectors implement BaseConnector 2a8d7c7972
dylan added 1 commit 2026-03-10 18:18:46 +00:00
fix(connectors): incorrect registry location 5ccb2e73cd 
Registry paths were using the incorrect connector path locations.
dylan added 3 commits 2026-03-10 18:41:09 +00:00
chore: add util to always split evenly 0866dda8b3
feat(topics): add generalised topic list 53cb5c2ea5 
This is easier and quicker compared to deriving a topics list based on the dataset that has been scraped.

While using LLMs to create a personalised topic list based on the query, category or dataset itself would yield better results for most, it is beyond the scope of this project.
fix(connectors): connector detectors returning name of ID alongside connector obj 17bd4702b2
dylan added 4 commits 2026-03-10 19:17:45 +00:00
chore(utils): add env getter that fails if env not found 2572664e26
feat(api): add database scraping endpoints 6ec47256d0
chore(db): update db schema to include "fetching" status 15704a0782
fix(api): flask delegates dataset fetch to celery a65c4a461c
dylan added 1 commit 2026-03-10 19:23:51 +00:00
fix(frontend): option to delete dataset not shown after fail a3dbe04a57
dylan added 2 commits 2026-03-10 22:17:04 +00:00
build: change celery to debug mode 7ccc934f71
fix(celery): process dataset directly in fetch task dc330b87b9 
Calling the original `process_dataset` function led to issues with JSON serialisation.
dylan added 2 commits 2026-03-10 22:48:08 +00:00
fix: data leak when opening topics file 8fe84a30f6
fix(auth): missing email and username business rules d520e2af98
dylan added 1 commit 2026-03-10 23:15:36 +00:00
feat(api): support per-source search, category and limit configuration 2ab74d922a
dylan added 1 commit 2026-03-10 23:28:33 +00:00
fix(api): incorrect dataset status update message 524c9c50a0
dylan added 2 commits 2026-03-11 19:41:37 +00:00
fix(connectors): hardcoded source name in Youtube connector eff416c34e
feat(frontend): add page for scraping endpoint b2ae1a9f70
dylan added 1 commit 2026-03-11 19:44:40 +00:00
chore: remove unused dataset creation script 0658713f42
dylan added 1 commit 2026-03-11 19:47:45 +00:00
chore(utils): remove split_limit function 12cbc24074
dylan added 1 commit 2026-03-11 21:16:28 +00:00
fix(connectors): category / search fields breaking 01d6bd0164 
Ideally category and search are fully optional, however some sites break if one or the other is not provided.

Unfortuntely `boards.ie` has a different page type for searches and I'm not bothered to implement a scraper from scratch.

In addition, removed comment limit options.
dylan added 4 commits 2026-03-13 21:59:19 +00:00
chore(connectors): add category and search validation fields c12f1b4371
fix(connectors): add stronger validation to scrape endpoint 6684780d23 
Strong validation needed, otherwise data goes to Celery and crashes silently. In addition it checks if that specific source supports search or category.
fix(frontend): detects which sources support category or search 162a4de64e
fix(connectors): update URL references to use base_url in BoardsAPI d96f459104
dylan added 4 commits 2026-03-14 17:14:07 +00:00
feat(connectors): implement category validation in scraping process 8a423b2a29
fix(frontend): misaligned loading page for datasets c990f29645
chore(connectors): implement category_exists for Boards API 2a00795cc2
fix(api): incorrect validation on search 062937ec3c
dylan added 1 commit 2026-03-14 17:35:10 +00:00
fix(api): enforce integer limit and cap at 1000 in scrape_data function d2b919cd66
dylan added 1 commit 2026-03-14 21:53:16 +00:00
fix(connector): unnecessary comment limits 5b0441c34b 
In addition, I made some methods private to better align with the BaseConnector parent class.
dylan added 1 commit 2026-03-14 21:58:03 +00:00
fix(api): remove error exceptions in API responses 12f5953146 
Mainly a security thing, we don't want actual code errors being given in the API response, as someone could find out how the inner workings of the code behaves.
dylan changed title from WIP: Automatic Scraping of dataset options to Automatic Scraping of dataset options 2026-03-14 21:58:45 +00:00
dylan merged commit 94befb61c5 into main 2026-03-14 21:58:49 +00:00
dylan deleted branch feat/automatic-scraping-datasets 2026-03-14 21:58:49 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dylan
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dylan/crosspost#9
Delete Branch "feat/automatic-scraping-datasets"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?