6ec47256d0
feat(api): add database scraping endpoints
2026-03-10 19:04:33 +00:00
2572664e26
chore(utils): add env getter that fails if env not found
2026-03-10 18:50:53 +00:00
17bd4702b2
fix(connectors): connector detectors returning name of ID alongside connector obj
2026-03-10 18:36:40 +00:00
53cb5c2ea5
feat(topics): add generalised topic list
...
This is easier and quicker compared to deriving a topics list based on the dataset that has been scraped.
While using LLMs to create a personalised topic list based on the query, category or dataset itself would yield better results for most, it is beyond the scope of this project.
2026-03-10 18:36:08 +00:00
0866dda8b3
chore: add util to always split evenly
2026-03-10 18:25:05 +00:00
5ccb2e73cd
fix(connectors): incorrect registry location
...
Registry paths were using the incorrect connector path locations.
2026-03-10 18:18:42 +00:00
2a8d7c7972
refactor(connectors): Youtube & Reddit connectors implement BaseConnector
2026-03-10 18:11:33 +00:00
e7a8c17be4
chore(connectors): add base connector inheritance
2026-03-10 18:08:01 +00:00
cc799f7368
feat(connectors): add base connector and registry for detection
...
Idea is to have a "plugin-type" system, where new connectors can extend the `BaseConnector` class and implement the fetch posts method.
These are automatically detected by the registry, and automatically used in new Flask endpoints that give a list of possible sources.
Allows for an open-ended system where new data scrapers / API consumers can be added dynamically.
2026-03-09 21:29:03 +00:00
262a70dbf3
refactor(api): rename /upload endpoint
...
Ensures consistency with the other dataset-based endpoints and follows the REST-API rules more cleanly.
2026-03-09 20:55:12 +00:00
ca444e9cb0
refactor: move connectors to backend dir
...
They will now be more used in the backend.
2026-03-09 20:53:13 +00:00
738af5415b
Merge pull request 'Editable and removable datasets' ( #8 ) from feat/editable-datasets into main
...
Reviewed-on: #8
2026-03-05 16:55:48 +00:00
2b14a8a417
feat(frontend): add deletion modal confirmation box
2026-03-05 12:29:53 +00:00
a154b25415
fix(db): missing rollback on execute_batch method
...
Arguably more important on a batch function to have rollback.
2026-03-05 10:09:14 +00:00
eb273efe61
Merge remote-tracking branch 'origin/main' into feat/editable-datasets
2026-03-04 22:34:55 +00:00
a9001c79e1
build: add frontend to main docker compose
...
Forgot to add this earlier
2026-03-04 22:34:32 +00:00
eec8f2417e
feat(frontend): add ability to delete datasets
2026-03-04 22:32:19 +00:00
f5835b5a97
feat(frontend): add frontend option to change name
2026-03-04 22:17:31 +00:00
64e3f9eea8
feat: implement PATCH dataset route
...
At the moment only allows for the updating of the name. Which seems to be the only editable part of dataset metadata.
2026-03-04 21:38:06 +00:00
4f01bf0419
fix(db): incorrect SQL condition when deleting dataset content
2026-03-04 21:35:10 +00:00
6948891677
Merge remote-tracking branch 'origin/main' into feat/editable-datasets
2026-03-04 21:30:13 +00:00
f1f33e2fe4
feat: implement delete dataset route
2026-03-04 21:29:01 +00:00
e20d0689e8
fix(celery): adjust try-catch logic to improve error handling
...
Capturing the instantiation of the database and dataset manager objects inside the try-catch will cause errors if something else fails.
If an exception occurs and the dataset_manager is not initialised, the code inside the catch block will fail.
2026-03-04 21:18:59 +00:00
fcdac6f3bb
Merge pull request 'Fix the frontend API calls and implement logins on frontend' ( #7 ) from feat/update-frontend-api-calls into main
...
Reviewed-on: #7
2026-03-04 20:20:50 +00:00
5fc1f1532f
feat(user stats): updated styling and stats in user page
...
Interaction graph was taking up too much space and was the only thing on the screen. Further statistics were added however these may be removed in favour of more informative statistics
2026-03-04 20:20:34 +00:00
24277e0104
fix(frontend): move loading card higher up
...
Looks weird lower down on the screen
2026-03-04 20:09:55 +00:00
4e99b77492
fix(db): missing post ID in db schema
...
Caused surprisingly little errors. It only broke the interaction graph.
2026-03-04 20:05:20 +00:00
b6815c490a
feat: add loading page for when dataset is loading
...
Originally there was a simple "Loading" text, however this looked bad and might lead a user to think that the page had frozen.
There is now a more comprehensive loading animation which users might be happy to sit on for a few minutes.
2026-03-04 18:39:20 +00:00
29c90ddfff
feat: update name on topbar
...
Crosspost Analysis Engine sounds far cooler than "Ethnograph View"
2026-03-04 18:37:48 +00:00
3fe08b9c67
fix(backend): buggy reply_time_by_emotion metric
...
This metric was never stastically significant and held no real value. It also so happened to hold accidental NaN values in the dataframe which broke the frontend.
Happy to remove.
2026-03-04 18:37:11 +00:00
f9bc9cf9c9
fix: remove Datasets tab when not logged in
2026-03-03 20:32:33 +00:00
249528bb5c
feat(frontend): remove "Upload" and "Last Stats" page
...
These are redundant and clunky, everything can be accessed from the Dataset tab
2026-03-03 20:30:42 +00:00
bd0e1a9050
refactor(frontend): move stylings out of logic into centralized file
2026-03-03 20:28:23 +00:00
e2ac4495fd
chore(frontend): add extra types to frontend
2026-03-03 20:13:13 +00:00
f3b48525e2
feat(backend): increase default JWT expiration
2026-03-03 20:09:57 +00:00
55319461e5
feat: add "My Datasets" page
2026-03-03 19:52:12 +00:00
531ddb0467
fix(frontend): incorrect URLs in stats page
2026-03-03 18:36:46 +00:00
d11c5acb77
refactor: calculation of document titles into another class
2026-03-03 18:18:05 +00:00
f63f4e5f10
feat(frontend): add status page for loading dataset
2026-03-03 18:13:51 +00:00
23c58e20ae
feat(frontend): add title to pages
2026-03-03 17:59:50 +00:00
207c4b67da
feat(frontend): add dataset name requirements to the upload page
2026-03-03 17:28:46 +00:00
772205d3df
feat(api): add ability to fetch all datasets by a user
2026-03-03 17:25:00 +00:00
b6de100a17
feat: overhaul upload page styling
2026-03-03 17:18:09 +00:00
5310568631
feat: add React layout and a topbar allowing for easy logins
2026-03-03 17:17:57 +00:00
4b33f17b4b
fix: inconsistent styling in login page
2026-03-03 17:07:31 +00:00
64783e764d
fix: remove unnecessary styling in index.css
2026-03-03 16:57:10 +00:00
8ac5207a11
feat: add login page
2026-03-03 15:55:01 +00:00
090a57f4dd
build: add frontend to docker
2026-03-03 15:29:21 +00:00
c1a0324a03
build: add example env
2026-03-03 15:21:28 +00:00
cade2b1866
build: fix directories in docker compose
2026-03-03 15:19:57 +00:00
