cmoncrawl.processor.dao.api

Classes

class cmoncrawl.processor.dao.api.CCAPIGatewayDAO(base_url: str = 'https://data.commoncrawl.org/')

This class represents a DAO (Data Access Object) for interacting with the Common Crawl API Gateway. It provides methods for opening and closing a connection, fetching data for a given domain record, and handling errors related to downloading data.

Parameters:

base_url (str) – The base URL of the Common Crawl API Gateway. Defaults to “https://data.commoncrawl.org/”.

aopen()

Asynchronously opens a connection to the API Gateway.

aclose()

Asynchronously closes the connection to the API Gateway.

fetch()

Asynchronously fetches data for a given domain record.

Example usage:

>>> dao = CCAPIGatewayDAO()
>>> async with dao:
>>>     data = await dao.fetch(domain_record)