Strong validation needed, otherwise data goes to Celery and crashes silently. In addition it checks if that specific source supports search or category.
Ideally category and search are fully optional, however some sites break if one or the other is not provided.
Unfortuntely `boards.ie` has a different page type for searches and I'm not bothered to implement a scraper from scratch.
In addition, removed comment limit options.
This is easier and quicker compared to deriving a topics list based on the dataset that has been scraped.
While using LLMs to create a personalised topic list based on the query, category or dataset itself would yield better results for most, it is beyond the scope of this project.
Idea is to have a "plugin-type" system, where new connectors can extend the `BaseConnector` class and implement the fetch posts method.
These are automatically detected by the registry, and automatically used in new Flask endpoints that give a list of possible sources.
Allows for an open-ended system where new data scrapers / API consumers can be added dynamically.
Capturing the instantiation of the database and dataset manager objects inside the try-catch will cause errors if something else fails.
If an exception occurs and the dataset_manager is not initialised, the code inside the catch block will fail.
Interaction graph was taking up too much space and was the only thing on the screen. Further statistics were added however these may be removed in favour of more informative statistics
Originally there was a simple "Loading" text, however this looked bad and might lead a user to think that the page had frozen.
There is now a more comprehensive loading animation which users might be happy to sit on for a few minutes.
This metric was never stastically significant and held no real value. It also so happened to hold accidental NaN values in the dataframe which broke the frontend.
Happy to remove.
