Use case · decision ranking
Web data collection
Collect data repeatedly from authorized web sources.
11 reviewed matches, ranked by fit and deterministic project health.
#1
Editorialscrapy/scrapy: Scrapy, a fast high-level web crawling & scraping framework for Python.
Editorial95% fitThe curated scraping category is a strong fit for repeatable collection of data from authorized web sources.Compare this repository → ★ 64KPythonBSD-3-Clauseweb-scraping
#2
Editorialfirecrawl/firecrawl: The context API to search, scrape, and interact with the web at scale. 🔥.
Editorial95% fitThe curated scraping category is a strong fit for repeatable collection of data from authorized web sources.Compare this repository → ★ 171.9KTypeScriptAGPL-3.0data-extractionweb-scraping
#3
Editorialapify/crawlee: Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Editorial95% fitThe curated scraping category is a strong fit for repeatable collection of data from authorized web sources.Compare this repository → ★ 25.5KTypeScriptApache-2.0web-scraping
#4
EditorialD4Vinci/Scrapling: 🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!.
Editorial95% fitThe curated scraping category is a strong fit for repeatable collection of data from authorized web sources.Compare this repository → ★ 76.3KPythonBSD-3-Clauseweb-scraping
#5
Editorialgetmaxun/maxun: 🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥.
Editorial95% fitThe curated scraping category is a strong fit for repeatable collection of data from authorized web sources.Compare this repository → ★ 17.3KTypeScriptAGPL-3.0web-scraping
#6
EditorialMontFerret/ferret: Declarative data automation language and Go runtime for structured extraction workflows.
Editorial95% fitThe curated scraping category is a strong fit for repeatable collection of data from authorized web sources.Compare this repository → ★ 6KGoApache-2.0data-extractionweb-scraping
#7
Editorialapify/crawlee-python: Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Editorial95% fitThe curated scraping category is a strong fit for repeatable collection of data from authorized web sources.Compare this repository → ★ 9.5KPythonApache-2.0web-scraping
#8
Editorialfirecrawl/firecrawl-mcp-server: 🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.
Editorial95% fitThe curated scraping category is a strong fit for repeatable collection of data from authorized web sources.Compare this repository → ★ 7.3KJavaScriptMITapiweb-scraping
#9
Editorialalirezamika/autoscraper: A Smart, Automatic, Fast and Lightweight Web Scraper for Python.
Editorial95% fitThe curated scraping category is a strong fit for repeatable collection of data from authorized web sources.Compare this repository → ★ 7.9KPythonMITdata-extractionweb-scraping
#10
Editorialcode4craft/webmagic: A scalable web crawler framework for Java.
Editorial95% fitThe curated scraping category is a strong fit for repeatable collection of data from authorized web sources.Compare this repository → ★ 11.7KJavaApache-2.0web-scraping
#11
Editorialbrightdata/cli: Official Bright Data CLI - scrape, search, and extract structured web data directly from your terminal.
Editorial95% fitThe curated scraping category is a strong fit for repeatable collection of data from authorized web sources.Compare this repository → ★ 6.4KTypeScriptMITdata-extractionweb-scraping