\documentclass{article}[12pt]
\usepackage{graphicx}
\usepackage{setspace}
\usepackage{hyperref}

\begin{document}

\begin{titlepage}
    \centering
    \vspace*{3cm}

    {\Huge \textbf{Web-Based Tool for Observing and Analysing Online Communities} \par}
    
    \vspace{2cm}
    
    {\Large Dylan De Faoite \par}
    
    \vspace{0.5cm}
    
    {\large April 2026 \par}
    
    \vfill
    
    {\large
    Bachelor of Science in Computer Science \\
    University College Cork \\
    Supervisor: Paolo Palmeiri
    \par}
    
    \vspace{2cm}
    
\end{titlepage}
\section{Introduction}
This project presents the design and implementation of a web-based analytics engine for the exploration and analysis of online discussion data. Built using \textbf{Flask and Pandas}, and supplemented with \textbf{Natural Language Processing} (NLP) techniques, the system provides an API for extracting structural, temporal, linguistic, and emotional insights from social media posts. A React-based frontend delivers interactive visualizations and user controls, the backend architecture implements analytical pipeline for the data, including data parsing, manipulation and analysis.

\vspace{0.5cm}
Beyond its technical objectives, the system is conceptually informed by approaches from \textbf{digital ethnography} and computational social science. Traditional Ethnography is the practice of studying individual or group culture from the point of view of the subject of the study. Digital ethnography seeks to understand how social relations, topics and norms are constructed in online spaces.

\subsection{Motivation}
There are many beneficiaries of a digital ethnography analytic system: social scientists gain a deeper understanding of contemporary culture and online communities; businesses and marketers can better understand consumer behaviour and online engagement; educators and designers can improve digital learning environments and user experiences; and policymakers can make informed decisions regarding digital platforms, online safety, and community regulation.

\subsection{Goals \& Objectives}
\begin{itemize}
    \item \textbf{Collect data ethically}: enable users to link/upload text, and interaction data (messages etc) from specified online communities. Potentially an automated method for importing (using APIs or scraping techniques) could be included as well.
    \item \textbf{Organise content}: Store gathered material in a structured database with tagging for themes, dates, and sources.
    \item \textbf{Analyse patterns}: Use natural language processing (NLP) to detect frequent keywords, sentiment, and interaction networks.
    \item \textbf{Visualise insights}: Present findings as charts, timelines, and network diagrams to reveal how conversations and topics evolve.
\end{itemize}

\subsection{The Cork Dataset}

A defining feature of this project is its focus on a geographically grounded dataset centred on \textbf{Cork, Ireland}. The system analyses publicly available discussions relating to Cork drawn from multiple online platforms:

\begin{itemize}
    \item The \textbf{r/Cork} subreddit
    \item The \textbf{r/Ireland} subreddit using a Cork-specific search filter
    \item \textbf{YouTube} videos retrieved using Cork-related search queries
    \item The \textbf{Boards.ie Cork section}
\end{itemize}

\newpage
\section{Background}
This section describes what digital ethnography is, how it stems from traditional ethnography and why it is useful.

\subsection{Digital Ethnography}
Digital Ethnography is the study of cultures and interactions in various online spaces, such as forums, posts and video comments. The goal is not only to describe the high-level statistics such as number of posts and posts per day, but also analyse people's behaviour at an interactional and cultural level, delving into common phrases, interactions patterns and common topics and entities.

There are multiple methods to carry out digital ethnography, such as online participant observation through automated or manual methods, digital interviews via text or video or tracing digital footprints. 

Compared to traditional ethnography, digital ethnography is usually faster and more cost-effective due to the availability of large swathes of data across social media sites such as Reddit, YouTube, and Facebook and lack of need to travel. Traditional ethnography often relied on in-person interviews and in-person observation of communities

\subsection{Traditional Ethnography}
Ethnography originated in the late nineteenth and early twentieth centuries as a method for understanding cultures through long-term fieldwork. The goal was not just to describe behaviour, but to show how people made sense of that world. Over time, ethnography grew beyond anthropology into sociology, media studies, education, and human computer interaction, becoming a broadly used qualitative research approach. Traditional ethnography was closely tied to physical locations: villages, workplaces or towns. However, as communication technologies developed and social life increasingly took place through technological mediums, it was no longer tied to a physical place. Researchers questioned whether social interactions could still be studied properly if they were no longer tied to physical places.

\subsection{Transition to Digital Spaces}
The rise of the internet in the late twentieth century massively changed social interaction. Online forums, emails, SMS and social media platforms became central to human communication. All types of groups and identities were constructed. As a result, ethnographic methods were adapted to study these emerging digital environments. Early work in this area was referred to as "virtual ethnography" or "digital ethnography", where online spaces began to mixed and intertwine with traditional cultural spaces.

Digital ethnography gives us new challenges to overcome in comparison to traditional ethnography. The field is distributed across platforms, devices and online-offline interactions. For example, a digital ethnographer studying influencer culture might examine Instagram posts, comment sections, private messages, algorithms, and also conduct interviews or observe offline events. This transitions requires flexibility, since researchers can no longer rely solely on face-to-face interactions.

\subsection{Online Communities}
There are many different types of online communities, often structured in various ways, with many different types of users, norms and power dynamics. These communities can range from large-scale social networking platforms and discussion forums to niche interest. Each type of community fosters different forms of interaction, participation, and identity construction.

Participation within these communities is usually not evenly distributed. The majority of users are passive consumers (lurkers), a smaller percentage contribute occasionally, and a very small core group produces most of the content. This uneven contribution structure has significant implications for digital ethnography, as visible discourse may disproportionately reflect the perspectives of highly active members rather than the broader community. This is particularly evident in some reputation-based systems such as Reddit, which allows for the opinions of a few to rise above the rest.

\subsection{Digital Ethnography Metrics}
This section describes common keywords and metrics use to measure and quantify online communities using digital ethnography.

\subsubsection{Sentiment Analysis}
Sentiment Analysis involves capturing the emotions associated with a specific post, topic or entity. This type of analysis can be as simple as classifying a post as "positive" or "negative", or classifying a post into a set of pre-existing emotions such as anger, joy or sadness.

\subsubsection{Active vs Passive Participation}
Not everyone in an online community participates in the same way. Some users post regularly, leave comments, and interact with others, while many more simply read content without ever contributing anything themselves. Some might only contribute occasionally.

This distinction between active and passive participation (passive users are often referred to as "lurkers") is an important one in digital ethnography, because looking only at posts and comments can give a misleading picture of how large or engaged a community actually is. 

\subsubsection{Temporal Activity Patterns}
Looking at when a community is active can reveal quite a lot about its nature and membership. A subreddit that peaks at 2am UTC might have a mostly American userbase, while one that is consistently active across all hours could suggest a more globally distributed community. Beyond timezones, temporal patterns can also capture things like how a community responds to external events, like a sudden spike in posting activity often corresponds to something newsworthy happening that is relevant to the community.

\subsubsection{Cultural Markers}
Cultural markers are the words, phrases, memes, and behaviours that are specific to a particular community and signal that someone is a member of it. These might include in-jokes, niche slang, recurring references, or even particular ways of formatting posts. In the context of digital ethnography, identifying these markers is useful because they reveal how communities build a shared identity and distinguish themselves from outsiders.

\subsection{Natural Language Processing}
\textbf{Natural Language Processing} is a branch of artificial intelligence that allows machines to interpret, analyse and generate human language. The aim of NLP models is not only to understand single words individually, but to be able to understand the context of those words in a broader paragraph or story.

NLP can carry out many different types of tasks, such as classifying sentences or paragraphs, generating text content, extracting answers from text or even speech recognition in audio. However, even with the advances in NLP models, many challenges and limitations remain. These include understanding ambiguity, cultural context, sarcasm, and humour.

\subsubsection{Why Natural Language Processing?}
Digital ethnography traditionally relied on manual reading of texts and interviews. These approaches are valuable for deep interpretive analysis, but they do not scale well to the volume of data generated in online communities. A single subreddit might contain hundreds of thousands of posts and comments, far beyond what any single researcher could read and analyse by hand.

NLP techniques can be used to automatically process and analyse large volumes and applying ethnographic methods at scale. For example, NLP can be used to identify common themes and topics in a subreddit, track how these themes evolve over time, and even detect the emotional tone of discussions. This allows researchers to gain insights into the dynamics of online communities that would be impossible to achieve through manual analysis alone.

\subsubsection{Sentiment Analysis}
\textbf{Sentiment Analysis} involves determining the emotional tone behind a piece of text. It is commonly used to classify text as positive, negative, or neutral. This technique is widely applied in areas such as customer feedback analysis, social media monitoring, and market research. More advanced sentiment analysis models can detect nuanced emotions, such as frustration, satisfaction, or sarcasm, although accurately identifying these emotions remains a challenge.

\subsubsection{Named Entity Recognition}
\textbf{Named Entity Recognition (NER)} is the process of identifying and classifying key entities within a text into predefined categories like names of people, organisations, locations, or dates. NER is essential for structuring unstructured text data and is often used in information extraction, search engines, and question-answering systems. Despite its usefulness, NER can struggle with ambiguous entities or context-dependent meanings.

\subsubsection{Topic Modelling}
\textbf{Topic Modelling} is a technique used in NLP to identify the main themes or topics present in a collection of text. Instead of analysing each sentence individually, topic modelling looks for patterns of words that frequently appear together, allowing it to group documents based on similar themes.

This method is often used to organise lots of unstructured data, such as news articles, research papers, or social media posts.

\subsection{Limits of Computation Analysis}
While computational methods enable large-scale observation and analysis of online communities, there are many limitations that must be acknowledged. Many limitations come from NLP techniques and the practical boundaries of computational resources.

Natural Language Processors will be central to many aspects of the virtual ethnography, such as emotional and topic classification. While these models are strong and have shown results in many areas, they are imperfect and may produce inaccurate or misleading results.

One key limitation is how the models will likely find it difficult to interpret context-dependent language. Online communities will often use sarcasm, irony or culturally specific references, all of which will be challenging to for NLP models to correctly interpret. For example, a sarcastic comment might be incorrectly classified as positive, despite conveying negativity.

In addition, the simplification of complex human interactions and emotions into discrete categories like "happy" or "sad" will more than likely overlook some nuance and ambiguity, even if the model is not inherently "wrong". As a result, the outputs of NLP models should be interpreted as indicative patterns rather than definitive representations of user meaning.

\subsubsection{Computational Constraints}
The performance and speed of the system will be influenced by the computational resources available during development and execution. While the system will attempt to use GPU acceleration during NLP inference, these resource may not always be available or may not be particularly strong should they exist.

\subsection{Cork Dataset}
The Cork dataset serves as the foundation for this project, providing a geographically and culturally grounded corpus for analysis. Rather than examining a globally distributed or topic-neutral community, the dataset centres on a single city with Cork, Ireland which allows the system's analytical outputs to be interpreted against a known social and cultural context.

The dataset is drawn from four distinct online platforms, each of which represents a structurally different mode of online community participation:

\begin{itemize}
    \item \textbf{r/Cork} — a dedicated subreddit for Cork residents and diaspora, characterised by threaded discussion, upvote-based content curation, and an established community identity with its own norms and recurring topics.
    \item \textbf{r/Ireland} (Cork-filtered) — the broader Irish national subreddit, filtered by Cork-related keywords, capturing how Cork is discussed within a wider national discourse rather than within its own community space.
    \item \textbf{YouTube} — video comments retrieved via Cork-related search queries, representing a flatter, less threaded interaction model and a potentially more casual or emotionally expressive register than forum-style platforms.
    \item \textbf{Boards.ie Cork section} — an older Irish forum platform with a distinct demographic profile and lower volume compared to Reddit, providing a counterpoint to the Reddit-dominant data and representing a longer-established form of Irish online community.
\end{itemize}

Reddit's hierarchical comment threading enables deep conversational analysis and reply-chain metrics, whereas YouTube comments are largely flat and unthreaded. Boards.ie occupies a middle ground, with linear threads but a more intimate community character. Taken together, the four sources offer variation in interaction structure, community age, demographic composition, and linguistic register, all of which are factors that the system's analytical modules are designed to detect and compare.

Collecting data across multiple platforms also introduces the challenge of normalisation. Posts, comments, and metadata fields differ in schema and semantics across sources. A core design requirement of the system is the normalisation of these inputs into a unified event-based internal representation, allowing the same analytical pipeline to operate uniformly regardless of the source.

\newpage
\section{Analysis}

\subsection{Goals \& Objectives}
The objective of this project is to provide a tool that can assist social scientists, digital ethnographers, and researchers to observing and interpret online communities and the interactions between them. Rather than replacing the study of digital ethnography or the related fields, this tool aims to aid researchers analyse communities.

Specifically, the system aims to: 

\begin{itemize}
    \item \textbf{Enable scalable observation}: Provide researchers with the ability to process and explore large volumes of online discussion data that would be impractical to analyse manually.

    \item \textbf{Support mixed-method research}: Bridge quantitative computational analysis and qualitative ethnographic interpretation by presenting statistically derived insights that can inform deeper contextual study.

    \item \textbf{Reveal structural dynamics}: Expose interaction patterns such as reply networks, participation inequality, conversation depth, and influential contributors within a community.

    \item \textbf{Identify thematic structures}: Detect dominant topics, recurring phrases, and emerging themes using Natural Language Processing techniques.

    \item \textbf{Track emotional and cultural signals}: Analyse sentiment and emotion distributions across posts, users, and topics to better understand the affective tone of discussions and how it evolves over time.

    \item \textbf{Examine temporal evolution}: Provide time-series analysis of activity levels, topic trends, and emotional shifts, enabling longitudinal observation of community development.

    \item \textbf{Promote ethical data practices}: Restrict analysis to publicly available data, provide opt-out mechanisms for computationally intensive processing, and ensure responsible handling of user-generated content.
\end{itemize}

Ultimately, the project seeks to demonstrate how computational systems can aid and augment social scientists and digital ethnographers toolkits.

\subsection{Feasibility Analysis}
\subsubsection{NLP Limitations}
Online communities often use sarcasm, irony or context-specific references, all of which will be challenging for NLP models, especially weaker ones, to correctly interpret. In a Cork-specific dataset, this will be especially apparent due to the use of regional slang or informal grammar. 

Therefore, the outputs of the model for any single event should not be considered as definitive, but rather as an indicative pattern that is more likely to be correct when aggregated across the entire dataset. For example, while a single comment about a specific topic might be misclassified as positive, the overall sentiment of that topic across thousands of comments is more likely to reflect the true emotional tone of the community.

To account for NLP limitations, the system will:
\begin{itemize}
    \item Rely on \textbf{aggregated results} rather than individual classifications.
    \item Provide \textbf{context for outputs}, such as confidence scores where available.
    \item Allow \textbf{access to original text} behind each NLP result.
\end{itemize}

Overall, while NLP provides powerful tools for analysing large datasets, its limitations must be acknowledged and mitigated through careful design and interpretation of results.

\subsubsection{Data Normalisation}
Different social media platforms will produce data in many different formats. For example, Reddit data will have a much different reply structure to a forum-based platform like Boards.ie where there are no nested replies. Therefore, a core design requirement of the system is to normalise all incoming data into a single unified internal data model. This allows the same analytical functions to be applied across all data sources, regardless of their original structure.

Posts and comments are two different types of user-generated content, however when it comes to ethnographic analysis, they are both just "events" or information that is being shared by a user. From an ethnographic perspective, the distinction between a post and a comment is not particularly important, since they both represent user-generated content that contributes to the community discourse. Therefore, the system will normalise all posts and comments into a single "event" data model, which will allow the same analytical functions to be applied uniformly across all content. This also simplifies the data model and reduces the complexity of the analytical pipeline, since there is no need to maintain separate processing paths for posts and comments.

Though separate processing paths are not needed, the system will still retain metadata that indicates whether an event was originally a post or a comment, as well as any relevant structural information (e.g., parent-child relationships in Reddit threads).

\subsubsection{Scalability Constraints}
This system should be scalable enough to handle large datasets, but there are practical limits to how much data can be processed within reasonable timeframes, especially given the computational demands of NLP models.

Some of the data can be precomputed during the data ingestion phases, such as datetime column derivations and NLP outputs, which can speed up and make queries more efficient. However, the initial processing time of large datasets will still be significant, especially if the dataset contains hundreds of thousands of posts and comments.

To mitigate this, the system will:
\begin{itemize}
    \item Utilise GPU acceleration where available for NLP inference.
    \item Pre-compute some analytical results during data ingestion to speed up subsequent queries.
    \item Store NLP outputs in the database to avoid redundant processing.
    \item Implement asynchronous processing for long-running tasks.
\end{itemize}

Overall, while the system is designed to be scalable, it is important to set realistic expectations regarding performance and processing times, especially for very large datasets.

\subsection{Ethics}
The system will process only publicly available data, and will not attempt to access private or restricted content.

\subsubsection{Automated Data Collection}
The system will provide an option for users to automatically fetch datasets from social media sites filtered for keywords or categories. Therefore, it's important to ensure that this data collection is done ethically.

The system will:
\begin{itemize}
    \item Respect rate limits by implementing an exponential backoff strategy for API requests.
    \item Only collect data that is publicly available and does not require authentication or violate platform terms of service.
    \item Provide user-agent headers that identify the system and its purposes
    \item Allow users the option to upload their own datasets instead of automated collection.
    \item For websites without an API, the \texttt{robots.txt} file will be examined to ensure compliance with platform guidelines.
    \item Data volume limits of up to 1000 posts per source will be enforced server-side to prevent excessive data collection.
\end{itemize}

Some platforms provide APIs that allow for easy and ethical data collection, such as YouTube and Reddit. These APIs have clear guidelines and rate limits that the system will adhere to. 

\paragraph{Reddit (API)}
Reddit provides a public API that allows for the retrieval of posts, comments, and metadata from subreddits. The system will use the official Reddit API with proper authentication via OAuth2 and access tokens. 

In November 2025, Reddit introduced a new approval process for API access, which requires developers to apply for access and specify their intended use case. While the public unauthenticated endpoints are still accessible, they have far stricter rate limits (100 requests every 10 minutes) compared to authenticated access (100 requests per minute). Therefore, the system shall allow for authenticated access to the Reddit API to speed up data retrival.

Unauthenticated access will still be available as a fallback if client credentials are not provided on the backend, but this will massively slow the data retreival process, and this will still only fetch public posts and comments.

From reddit, the system will collect posts, comments and all replies to comments, as well as metadata such as the author name and timestamp.

\paragraph{Boards.ie (Web Scraping)}
Boards.ie is an Irish discussion forum with no public API, so the system will use web scraping instead. The platforms \texttt{robots.txt} will be used to ensure compliance with the site's guidelines for automated access. The boards.ie \texttt{robots.txt} file contains the following information:

\begin{verbatim}
Sitemap: https://www.boards.ie/sitemapindex.xml
User-agent: *
Disallow: /entry/
Disallow: /messages/
Disallow: /profile/comments/
Disallow: /profile/discussions/
Disallow: /search/
Disallow: /sso/
Disallow: /sso
\end{verbatim}

Public discussion threads are allowed to be automatically crawled, while user profiles, private messages, and authentication endpoints are not allowed. The system will respect these boundaries and will not attempt to access any restricted path.

\paragraph{YouTube (Data API v3)}
YouTube is supported via the official YouTube Data API v3, provided by Google. The API exposes structured endpoints for querying videos, comments, channels, and playlists, making it well-suited for collecting public discourse around specific topics or keywords.

Authentication is handled through an API key issued via the Google Cloud Console. The API enforces a quota system rather than a traditional rate limit: each project is allocated 10,000 quota units per day by default, with different operations consuming different amounts.

In addition, comment retreival can be disabled by the video uploader, so the system will handle this case by skipping videos where comments are not accessible.

\subsubsection{Data Storage \& Retention}
All data fetched from social media sites are stored locally in a PostgreSQL database. The system will not share or expose any of this data to third parties beyond the users of this application. Raw API responses are discarded once the relevant information is extracted. 

All datasets are associated with one and only one user account, and the users themselves are responsible for uploading or fetching the data, analysing the data, and deleting the data when they are done. The system will not retain any data beyond what is necessary for the end-user to carry out their analysis, and users will have the option to delete their datasets at any time.

The system will not store any personally identifiable information beyond what is necessary for the analysis, which includes only usernames and timestamps. The system will not attempt to de-anonymise content creators or link data across platforms.

\subsubsection{User Security}
Standard security practices will be followed to protect user data and prevent unauthorized access. This includes:
\begin{itemize}
    \item The hashing of all user passwords and no storage of plaintext passwords.
    \item The use of JWTs for session management, with secure signing and an expiration time of 24 hours.
    \item Access control on all analysis API endpoints to ensure that end-users can only access their own datasets and results.
    \item Parameterised queries for all database interactions to prevent SQL injection attacks.
\end{itemize}

\subsection{Requirements}

The following requirements are derived from the backend architecture, NLP processing pipeline, and the React-based frontend interface.

\subsubsection{Functional Requirements}

\paragraph{Data Ingestion and Preparation}
\begin{itemize}
    \item The system shall accept social media data in \texttt{.jsonl} format containing posts and nested comments.
    \item The system shall validate uploaded files and return structured error responses for invalid formats or malformed data.
    \item The system shall normalise posts and comments into a unified event-based dataset.
    \item The system shall give the user the option to automatically fetch datasets from social media sites filtered for specific keywords or categories.
    \item The system shall provide a loading screen with a progress bar after the dataset is uploaded.
\end{itemize}

\paragraph{Dataset Management}
\begin{itemize}
    \item The system shall utilise Natural Language Processing models to generate average emotions per event.
    \item The system shall utilise Natural Language Processing models to classify each event into a topic.
    \item The system shall utilise Natural Language Processing models to identify entities in each event.
    \item The system shall allow the users to view the raw dataset.
    \item The system shall return detailed endpoints that return calculated statistics grouped into themes.
\end{itemize}

\paragraph{Filtering and Search}
\begin{itemize}
    \item The system shall support keyword-based filtering across content, author, and optionally title fields.
    \item The system shall support filtering by start and end date ranges.
    \item The system shall support filtering by one or more data sources.
    \item The system shall allow multiple filters to be applied simultaneously.
    \item The system shall return a filtered dataset reflecting all active filters.
\end{itemize}

\paragraph{Temporal Analysis}
\begin{itemize}
    \item The system shall compute event frequency per day.
    \item The system shall generate weekday--hour heatmap data representing activity distribution.
\end{itemize}

\paragraph{Linguistic Analysis}
\begin{itemize}
    \item The system shall compute word frequency statistics excluding standard and domain-specific stopwords.
    \item The system shall extract common bi-grams and tri-grams from textual content.
    \item The system shall compute lexical diversity metrics for the dataset.
\end{itemize}

\paragraph{Emotional Analysis}
\begin{itemize}
    \item The system shall compute average emotional distribution per topic.
    \item The system shall compute overall average emotional distribution across the dataset.
    \item The system shall determine dominant emotion distributions.
    \item The system shall compute emotional distribution grouped by data source.
\end{itemize}

\paragraph{User Analysis}
\begin{itemize}
    \item The system shall identify top users based on activity.
    \item The system shall compute per-user activity and behavioural metrics.
\end{itemize}

\paragraph{Interaction Analysis}
\begin{itemize}
    \item The system shall compute average conversation thread depth.
    \item The system shall identify top interaction pairs between users.
    \item The system shall generate an interaction graph based on user relationships.
    \item The system shall compute conversation concentration metrics.
\end{itemize}

\paragraph{Cultural Analysis}
\begin{itemize}
    \item The system shall identify identity-related linguistic markers.
    \item The system shall detect stance-related linguistic markers.
    \item The system shall compute average emotional expression per detected entity.
\end{itemize}

\paragraph{Frontend}
\begin{itemize}
    \item The system shall provide a frontend UI to accommodate all of the above functions
    \item The system shall provide a tab for each endpoint in the frontend
\end{itemize}

\subsubsection{Non-Functional Requirements}

\paragraph{Performance}
\begin{itemize}
    \item The system shall utilise GPU acceleration where available for NLP.
    \item The system shall utilise existing React libraries for visualisations.
\end{itemize}

\paragraph{Scalability}
\begin{itemize}
    \item The system shall utilise cookies and session tracking for multi-user support.
    \item NLP models shall be cached to prevent redundant loading.
\end{itemize}

\paragraph{Reliability and Robustness}
\begin{itemize}
    \item The system shall implement structured exception handling.
    \item The system shall return meaningful JSON error responses for invalid requests.
    \item The dataset reset functionality shall preserve data integrity.
\end{itemize}

\newpage
\section{Design}
\subsection{System Architecture}
\begin{figure}[h]
    \centering
    \includegraphics[width=1.0\textwidth]{img/architecture.png}
    \caption{System Architecture Diagram}
    \label{fig:architecture}
\end{figure}

\begin{figure}[h]
    \centering
    \includegraphics[width=1.0\textwidth]{img/schema.png}
    \caption{System Schema}
    \label{fig:schema}
\end{figure}

\subsection{Data Pipeline}
As this project is focused on the collection and analysis of online community data, the primary component that must be well-designed is the data pipeline, which encompasses the processes of data ingestion, normalisation, enrichment, storage, and retrieval for analysis.

A unified data model is used to represent all incoming data, regardless of its original source or structure. This ensures that the same pipeline works across YouTube, Reddit and boards.ie data, and can be easily extended to new sources in the future.



\subsection{Connector Abstraction}
While the system is designed around a Cork-based dataset, it is intentionally source-agnostic, meaning that additional data sources could be added in the future without changes to the core analytical pipeline.

\textbf{Data Connectors} are components responsible for fetching and normalising data from specific sources. Each connector implements a standard interface for data retrieval, such as:
\begin{itemize}
    \item \texttt{get\_new\_posts()} — retrieves raw data from the source, either through API calls or web scraping.
\end{itemize}

Creating a base interface for what a connector should look like allows for the easy addition of new data sources in the future. For example, if a new social media platform becomes popular, a new connector can be implemented to fetch data from that platform without needing to modify the existing data pipeline or analytical modules.

The connector registry is designed so that any new connector implementing \texttt{BaseConnector} is automatically discovered and registered at runtime, without requiring changes to any existing code. This allows for a modular and extensible architecture where new data sources can be integrated with minimal effort.

\subsection{Client-Server Architecture}
The system will follow a client-server architecture, with a Flask-based backend API and a React-based frontend interface. The backend will handle data processing, NLP analysis, and database interactions, while the frontend will provide an interactive user interface for data exploration and visualization.

The reasoning behind this architecture is that it allows the analytics to be aggregated and computed on the server side using Pandas which is much faster than doing it on the client frontend. The frontend will focus on rendering and visualising the data.

\subsubsection{Flask API}
The Flask backend will expose a RESTful API with endpoints for dataset management, authentication and user management, and analytical queries. Flask will call on backend components for data parsing, normalisation, NLP processing and database interfacing. 

Flask was chosen for its simplicity, familiarity and speed of development. It also has many extensions that can be used for authentication (Flask-Bcrypt, Flask-Login).

\subsubsection{React Frontend}
React was chosen for the frontend due to its massive library of pre-built components with efficient rendering capabilities and ability to display many different types of data. The frontend will be structured around a tabbed interface, with each tab corresponding to a different analytical endpoint (e.g., temporal analysis, linguistic analysis, emotional analysis). Each tab will fetch data from the backend API and render it using appropriate visualisation libraries (react-wordcloud for word clouds, react-chartjs-2 for charts, etc). The frontend will also include controls for filtering the dataset based on keywords, date ranges, and data sources.


\subsection{Database vs On-Disk Storage}
Originally, the system was designed to store \texttt{json} datasets on disk and load them into memory for processing. This was simple and time-efficient for early development and testing. However, as the functionality of the system expanded, it become clear that a more persistent and scalable storage solution was needed.

Storing datasets in a database allows for more efficient querying, filtering, and updating of data without needing to reload entire datasets into memory. However the priamry benefit of using a database is support for \textbf{ multiple users and multiple datasets per user}. 

An additional benefit of using a database was that it allowed the NLP processing to be done once, with the NLP results stored alongside the original data in the database. This meant that the system could avoid redundant NLP processing on the same data, which was a significant performance improvement.

\texttt{PostgreSQL} was chosen as the database solution due to its robustness, support for complex queries, and compatibility with Python through \texttt{psycopg2}. PostgreSQL's support for JSONB fields allows for storage of unstructured NLP outputs, which alternatives like SQLite does not support.

\subsection{Asynchronous Processing}
The usage of NLP models for tasks such as sentiment analysis, topic classification, and entity recognition can be computationally intensive, especially for large datasets. To prevent the Flask API from blocking while these tasks are being processed, an asynchronous processing queue will be implemented using \textbf{Redis} and \textbf{Celery}.

When NLP processing is triggered or data is being fetched from social media APIs, a task will be added to the Redis queue. Celery workers will then pop tasks off the Redis queue and process these tasks in the background, which ensures the API to remain responsive to user requests. This approach also allows for better scalability, as additional workers can be added to handle increased load.

Some of the these tasks, like fetching data from social media APIs are very long-running tasks that can take hours to complete. By using asynchronous processing that updates the database with progress updates, users can see the status of their data fetching through the frontend.

\subsection{Docker Deployment}
Docker Compose will be used to containerise the entire application, including:
\begin{itemize}
    \item The Flask backend API
    \item The React frontend interface
    \item The PostgreSQL database
    \item The Redis server for task queuing
    \item Celery workers for asynchronous processing
    \item NLP model caching and management
\end{itemize}

In addition, the source code for the backend and frontend will be mounted as volumes within the containers to allow for live code updates during development, which will speed up the process.

Enviornment variables, such as database credentials and social media API keys, will be managed through an \texttt{.env} file that is passed into the Docker containers through \texttt{docker-compose.yml}.


\newpage
\section{Implementation}

\newpage
\section{Evaluation}

\newpage
\section{Conclusions}

\end{document}
