Learn more about Crawlee for Python from the creators of Crawlee. Join us on August 5th at 9 AM EST.Sign up now!
Crawlee covers your crawling and scraping end-to-end andhelps you build reliable scrapers. Fast.
🚀 Crawlee for Python is open to early adopters!
Your crawlers will appear almost human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data and persistently store it in machine-readable formats, without having to worry about the technical details. And thanks to rich configuration options, you can tweak almost any aspect of Crawlee to suit your project's needs if the default settings don't cut it.
👉View full documentation, guides and examples on theCrawlee project website👈
We also have a TypeScript implementation of the Crawlee, which you can explore and utilize for your projects. Visit our GitHub repository for more informationCrawlee for JS/TS on GitHub.
We recommend visiting theIntroduction tutorialin Crawlee documentation for more information.
Crawlee is available as thecrawlee
PyPI package.
pip install crawlee
Additional, optional dependencies unlocking more features are shipped as package extras.
If you plan to parse HTML and use CSS selectors, installcrawlee
withbeautifulsoup
extra:
pip install'crawlee[beautifulsoup]'
If you plan to use a (headless) browser, installcrawlee
with theplaywright
extra:
pip install'crawlee[playwright]'
Then, install the Playwright dependencies:
playwright install
You can install multiple extras at once by using a comma as a separator:
pip install'crawlee[beautifulsoup,playwright]'
Verify that Crawlee is successfully installed:
python -c'import crawlee; print(crawlee.__version__)'
The quickest way to get started with Crawlee is by using the Crawlee CLI and selecting one of the prepared templates. First, ensure you havePipxinstalled:
pipx --help
Then, run the CLI and choose from the available templates:
pipx run crawlee create my-crawler
If you already havecrawlee
installed, you can spin it up by running:
crawlee create my-crawler
Here are some practical examples to help you get started with different types of crawlers in Crawlee. Each example demonstrates how to set up and run a crawler for specific use cases, whether you need to handle simple HTML pages or interact with JavaScript-heavy sites. A crawler run will create astorage/
directory in your current working directory.
TheBeautifulSoupCrawler
downloads web pages using an HTTP library and provides HTML-parsed content to the user. It usesHTTPXfor HTTP communication andBeautifulSoupfor parsing HTML. It is ideal for projects that require efficient extraction of data from HTML content. This crawler has very good performance since it does not use a browser. However, if you need to execute client-side JavaScript, to get your content, this is not going to be enough and you will need to use PlaywrightCrawler. Also if you want to use this crawler, make sure you installcrawlee
withbeautifulsoup
extra.
importasyncio
fromcrawlee.beautifulsoup_crawlerimportBeautifulSoupCrawler,BeautifulSoupCrawlingContext
asyncdefmain()->None:
crawler=BeautifulSoupCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
asyncdefrequest_handler(context:BeautifulSoupCrawlingContext)->None:
context.log.info(f'Processing{context.request.url}...')
# Extract data from the page.
data={
'url':context.request.url,
'title':context.soup.title.stringifcontext.soup.titleelseNone,
}
# Push the extracted data to the default dataset.
awaitcontext.push_data(data)
# Enqueue all links found on the page.
awaitcontext.enqueue_links()
# Run the crawler with the initial list of URLs.
awaitcrawler.run(['https://crawlee.dev'])
if__name__=='__main__':
asyncio.run(main())
ThePlaywrightCrawler
uses a headless browser to download web pages and provides an API for data extraction. It is built onPlaywright,an automation library designed for managing headless browsers. It excels at retrieving web pages that rely on client-side JavaScript for content generation, or tasks requiring interaction with JavaScript-driven content. For scenarios where JavaScript execution is unnecessary or higher performance is required, consider using theBeautifulSoupCrawler
.Also if you want to use this crawler, make sure you installcrawlee
withplaywright
extra.
importasyncio
fromcrawlee.playwright_crawlerimportPlaywrightCrawler,PlaywrightCrawlingContext
asyncdefmain()->None:
crawler=PlaywrightCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
)
# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
asyncdefrequest_handler(context:PlaywrightCrawlingContext)->None:
context.log.info(f'Processing{context.request.url}...')
# Extract data from the page.
data={
'url':context.request.url,
'title':awaitcontext.page.title(),
}
# Push the extracted data to the default dataset.
awaitcontext.push_data(data)
# Enqueue all links found on the page.
awaitcontext.enqueue_links()
# Run the crawler with the initial list of requests.
awaitcrawler.run(['https://crawlee.dev'])
if__name__=='__main__':
asyncio.run(main())
Explore ourExamplespage in the Crawlee documentation for a wide range of additional use cases and demonstrations.
Why Crawlee is the preferred choice for web scraping and crawling?
- Unified interface forHTTP & headless browsercrawling.
- Automaticparallel crawlingbased on available system resources.
- Written in Python withtype hints- enhances DX (IDE autocompletion) and reduces bugs (static type checking).
- Automaticretrieson errors or when you’re getting blocked.
- Integratedproxy rotationand session management.
- Configurablerequest routing- direct URLs to the appropriate handlers.
- Persistentqueue for URLsto crawl.
- Pluggablestorageof both tabular data and files.
- Robusterror handling.
- Crawlee has out-of-the-box support forheadless browsercrawling (Playwright).
- Crawlee has aminimalistic & elegant interface- Set up your scraper with fewer than 10 lines of code.
- Completetype hintcoverage.
- Based on standardAsyncio.
Crawlee is open-source and runs anywhere, but since it's developed byApify,it's easy to set up on the Apify platform and run in the cloud. Visit theApify SDK websiteto learn more about deploying Crawlee to the Apify platform.
If you find any bug or issue with Crawlee, pleasesubmit an issue on GitHub.For questions, you can ask onStack Overflow,in GitHub Discussions or you can join ourDiscord server.
Your code contributions are welcome, and you'll be praised for eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, seeCONTRIBUTING.md.
This project is licensed under the Apache License 2.0 - see theLICENSEfile for details.