feat(connectors): add base connector and registry for detection

Idea is to have a "plugin-type" system, where new connectors can extend the `BaseConnector` class and implement the fetch posts method.

These are automatically detected by the registry, and automatically used in new Flask endpoints that give a list of possible sources.

Allows for an open-ended system where new data scrapers / API consumers can be added dynamically.
This commit is contained in:
2026-03-09 21:28:44 +00:00
parent 262a70dbf3
commit cc799f7368
5 changed files with 73 additions and 4 deletions

View File

@@ -111,6 +111,20 @@ def get_user_datasets():
current_user = int(get_jwt_identity())
return jsonify(dataset_manager.get_user_datasets(current_user)), 200
@app.route("/datasets/sources", methods=["GET"])
@jwt_required()
def get_dataset_sources():
return jsonify({""})
@app.route("/datasets/scrape", methods=["POST"])
@jwt_required()
def scrape_data():
if "sources" not in request.form:
return jsonify({"error": "Data source names are required."}), 400
sources = request.form.get("sources")
@app.route("/datasets/upload", methods=["POST"])
@jwt_required()
def upload_data():