The gist

Scrapling is an open-source web scraping framework for Python, created by Karim Shoair. It's designed to handle everything from single requests to large-scale, concurrent crawls. The library solves common scraping challenges by automatically adapting to website structure changes, bypassing anti-bot systems like Cloudflare Turnstile, and managing complex crawling logic like proxy rotation and session management. It provides a robust, developer-friendly toolkit for reliable data extraction from modern websites.

What it does

Crawl websites using a Scrapy-like spider framework with concurrency controls.
Bypass anti-bot systems, including Cloudflare Turnstile, with advanced stealth fetchers.
Adapt to website layout changes by automatically relocating target elements.
Manage complex crawls with features like pause/resume, proxy rotation, and session support.
Fetch dynamically loaded content using integrated headless browser automation.
Extract data with a flexible parser supporting CSS selectors, XPath, and text search.

How it works

Scrapling is an open-source Python library installed via pip. Developers import its classes to write scripts that define crawlers or fetch web pages. Users provide start URLs and selectors (CSS or XPath) to target specific data. The library processes the pages, returning extracted content as Python objects or saving it directly to JSON/JSONL files. It can be used as an imported library, a command-line tool, or a self-hosted Docker container.

Best for

This framework is best for Python developers who need to build scrapers for complex, modern websites that employ anti-bot protections or frequently change their layout. It's ideal for projects requiring both simple data fetching and full-scale, resilient crawling.

Watch out for

The base installation only includes the core parser. To use web fetching, browser automation, or CLI features, you must install optional dependencies and then run a separate command (scrapling install) to download browser binaries.