Research data preparation

firecrawl/firecrawl: The context API to search, scrape, and interact with the web at scale. 🔥.

soimort/you-get: :arrow_double_down: Dumb downloader that scrapes the web.

airbytehq/airbyte: Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

clips/pattern: Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

pathwaycom/pathway: Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

MontFerret/ferret: Declarative data automation language and Go runtime for structured extraction workflows.

adbar/trafilatura: Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML.

alirezamika/autoscraper: A Smart, Automatic, Fast and Lightweight Web Scraper for Python.

dzhng/deep-research: An AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models. The goal of this repo is to provide the simplest implementation of a deep research agent - e.g. an agent that can refine its research direction overtime and deep dive into a topic.

brightdata/cli: Official Bright Data CLI - scrape, search, and extract structured web data directly from your terminal.