Scrapers

Scrapers are used to get articles information on your website. They need to be configured to know how to get the information.

You can manage (add, edit, delete) all you scrapers from the 'Scrapers' tab in the dashboard.

Configuration

Here is the interface to manage your scrapers.

Scrapers config

There are a lot of parameters here. Let's see how to configure them:

  • Name : The name of the scraper, to identify it.
  • Rescrap frequency : To precise when do you want your webpages to be rescraped, to get updated data (in days).
  • Whitelists - Blacklists : Are used to define rules to tell the scrapers which element to scrape or not, depending on their URL.
    For example you can define the following rule : Don't (blacklist) scrape article which URL ends with ".com"
  • Metadata & Categories extraction : Used to specify the request to scrap a metadata/categories

WARNING

Be careful with these rules, keep in mind that you can have conflicts between them, resulting in the scrapers scraping nothing.

Finally, your scrapers are listed above.