GitHub topic

crawler GitHub repositories

Explore 9 tracked GitHub repositories tagged crawler. Compare their recorded star activity and implementation languages, then follow related topics to find other projects.

Snapshot: 2026-10-28. The collection gained 507 stars since the previous daily snapshot.

Tracked repositories: 9
Total stars: 343,504
Stars gained · 7 days: +3,468
Stars gained · 30 days: +14,259

Compare star activity

Up to five projects on this page, ordered by total stars. Open a project for its description and recorded star history.

Swipe to compare all star periods →

Star activity for crawler repositories on this page
Repository	Total stars	Today	7 days	30 days
firecrawl/firecrawl	196.1K	+375	+2,589	+10,532
unclecode/crawl4ai	86.4K	+85	+480	+2,048
ScrapeGraphAI/Scrapegraph-ai	32.3K	+28	+243	+924
getmaxun/maxun	18.1K	+12	+103	+489
adbar/trafilatura	7.0K	+3	+30	+155

Explore repositories

Showing 1–9 of 9 tracked repositories.

firecrawl/firecrawl

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf

★ 86.4K+85 todayPython

llm ai

View ↗

ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

getmaxun/maxun

Turn any website into a structured API. Extract, automate, search and monitor the web.

★ 18.1K+12 todayTypeScript

agents self-hosted

View ↗

adbar/trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

fastcrw/crw

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

★ 1.2K+4 todayRust

llm ai

View ↗

codelibs/fess

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.

★ 1.1K+0 todayJava