fix(connectors): category / search fields breaking
Ideally category and search are fully optional, however some sites break if one or the other is not provided. Unfortuntely `boards.ie` has a different page type for searches and I'm not bothered to implement a scraper from scratch. In addition, removed comment limit options.
This commit is contained in:
@@ -46,8 +46,7 @@ def fetch_and_process_dataset(self,
|
||||
raw_posts = connector.get_new_posts_by_search(
|
||||
search=search,
|
||||
category=category,
|
||||
post_limit=limit,
|
||||
comment_limit=limit
|
||||
post_limit=limit
|
||||
)
|
||||
posts.extend(post.to_dict() for post in raw_posts)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user