How to retrieve the HTML of a page from CommonCrawl?

Question

Assuming I have:

• the link of the CC*.warc file (and the file itself, if it helps);
• offset; and
• length

How can I get the HTML content of that page?

Thanks for your time and attention.

Answer 1

Using warcio it would be simply:

warcio extract --payload <file.warc.gz> <offset>

Alternatively, fetch the WARC record using the HTTP range request and then extract the payload at offset 0:

curl -s -r331727487-$((331727487+6613-1)) \
   https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2020-40/segments/1600400203096.42/warc/CC-MAIN-20200922031902-20200922061902-00310.warc.gz \
   >warc_temp.warc.gz
warcio extract --payload warc_temp.warc.gz 0

The range starts at offset and ends at offset+length-1. See also getting WARC file

Answer 2

The below command worked for me.

warcio extract --payload CC-MAIN-20230928130936-20230928160936-00336.warc.gz 988155426 > nutsaboutmoney_from_warcio.html

The html file can then be read by a browser.

This was run in a virtual environment using python 3.11.6. warcio had been installed in the virtual environment with: pip instal warcio