docs: Add guide about creating and using WARC files with Crawlee
#1273
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pijukatel
added
documentation
Mantisus
approved these changes
Jun 27, 2025
barjin
reviewed
Jun 30, 2025
docs/guides/code_examples/creating_web_archive/manual_archiving_playwright_crawler.py
Outdated
Show resolved
Hide resolved
Co-authored-by: Jindřich Bär <jindrichbar@gmail.com>
barjin
reviewed
Jun 30, 2025
docs/guides/code_examples/creating_web_archive/manual_archiving_playwright_crawler.py
Show resolved
Hide resolved
barjin
approved these changes
Jul 1, 2025
vdusek
reviewed
Jul 2, 2025
docs/guides/creating_web_archive.mdx
Outdated
| - Make the crawler use the proxy server. | ||
| - Deal with the [pywb Certificate Authority](https://pywb.readthedocs.io/en/latest/manual/configuring.html#https-proxy-and-pywb-certificate-authority). | ||
|
|
||
| For example, in `PlaywrightCrawler`, this is the simplest setup, which takes the shortcut and ignores the CA-related errors: |
There was a problem hiding this comment.
Maybe PlaywrightCrawler could be API link here.
| # it is needed to scroll the page to load all content. | ||
| # It slows down the crawling, but ensures that all content is loaded. | ||
| await context.infinite_scroll() | ||
| await context.enqueue_links() |
There was a problem hiding this comment.
Maybe specify explicitly that we want to enqueue links only from the same domain?
| async def request_handler(context: ParselCrawlingContext) -> None: | ||
| context.log.info(f'Archiving {context.request.url} ...') | ||
| archive_response(context=context, writer=writer) | ||
| await context.enqueue_links() |
There was a problem hiding this comment.
Maybe specify explicitly that we want to enqueue links only from the same domain?
| # it is needed to scroll the page to load all content. | ||
| # It slows down the crawling, but ensures that all content is loaded. | ||
| await context.infinite_scroll() | ||
| await context.enqueue_links() |
There was a problem hiding this comment.
Maybe specify explicitly that we want to enqueue links only from the same domain?
|
|
||
| Previous commands start the wayback server that allows crawler requests to be served from the archived pages in the `example-collection` instead of sending requests to the real website. This is again [proxy mode of the wayback server](https://pywb.readthedocs.io/en/latest/manual/usage.html#http-s-proxy-mode-access), but without recording capability. Now you need to [configure your crawler](#configure-the-crawler) to use this proxy server, which was already described above. Once everything is finished, you can just run your crawler, and it will crawl the offline archived version of the website from your WARC file. | ||
|
|
||
| You can also manually browse the archived pages in the wayback server by going to the locally hosted server and entering the collection and URL of the archived page, for example: `http://localhost:8080/example-collection/https:/crawlee.dev/`. The wayback server will serve the page from the WARC file if it exists, or it will return a 404 error if it does not. For more detail about the server please refer to the [pywb documentation](https://pywb.readthedocs.io/en/latest/manual/usage.html#getting-started). |
There was a problem hiding this comment.
Please add a conclusion at the end with a summary and a call to action to join Discord and check out our Github. See https://crawlee.dev/python/docs/guides/storages#conclusion for example.
The final sentence could be the same:
If you have questions or need assistance, feel free to reach out on our GitHub or join our Discord community. Happy scraping!
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add guide about creating and using WARC files with
CrawleeIssues