Add comment limit to _parse_comments method in BoardsAPI
Some boards.ie threads have thousands of comments which is slow to fetch with pagination
This commit is contained in:
@@ -105,11 +105,11 @@ class BoardsAPI:
|
||||
|
||||
return post
|
||||
|
||||
def _parse_comments(self, url: str, post_id: str) -> list[Comment]:
|
||||
def _parse_comments(self, url: str, post_id: str, comment_limit: int = 500) -> list[Comment]:
|
||||
comments = []
|
||||
current_url = url
|
||||
|
||||
while current_url:
|
||||
while current_url and len(comments) < comment_limit:
|
||||
html = self._fetch_page(current_url)
|
||||
page_comments = self._parse_page_comments(html, post_id)
|
||||
comments.extend(page_comments)
|
||||
|
||||
Reference in New Issue
Block a user