Data Extraction

:arrow_double_down: Dumb downloader that scrapes the web

Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

An AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models. The goal of this repo is to provide the simplest implementation of a deep research agent - e.g. an agent that can refine its research direction overtime and deep dive into a topic.

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.