Programming Guide

This section of the documentation is for people who want to use the cmoncrawl library to create their own extraction pipeline in python. It allows use to take full advatange of the cmoncrawl library unlike the command line utility which is limited to a few options.

Contents:

• How to extract from Common Crawl (theory)
  • 1. Querying CommonCrawl
  • 2. Downloading a file
  • 3. Choose extractor
  • 4. Filtering out the web page
  • 5. Extract fields from the page
  • 6. File saving
• How to extract from Common Crawl (practice)
  • Pipeline
  • Simulatenous querying and extracting
  • Query records and then extract
  • Distributed Simulatenous high-throughput querying and extracting
  • Be cooperative