Crawler.sh
Fast, local-first web crawler with SEO analysis and Markdown extraction. Run site audits from your terminal or desktop app.
What is Crawler.sh?
Crawler.sh is a local-first web crawler and AEO & SEO analysis tool built in Rust for speed and privacy. It crawls entire websites in seconds, runs 22 automated SEO checks per page, extracts readable content as clean Markdown, and exports results in JSON, CSV, or Sitemap XML formats. Available as both a CLI tool and native desktop app for macOS.
Best for: Local-first workflows · Privacy-sensitive projects · CLI automation · Markdown output
Developer Stack Fit
Quick read on where Crawler.sh fits in a software team's AI stack. Validate final fit against your codebase, data policy, and deployment model.
- Stack layer
- Secondary/general AI
- Deployment model
- Self-hosted or local option
- Open-source status
- Not confirmed
- API support
- Not a primary API tool
- MCP support
- No MCP signal found
- Security posture
- Stronger controls worth validating
- Best use case
- Local-first workflows
Key Features
- 01
Local-first crawling built in Rust for maximum speed
Crawl thousands of pages in seconds
- 02
22 automated SEO checks per page (titles, meta, noindex, thin content, URLs)
22 automated SEO/AEO checks
- 03
Clean Markdown extraction with word count and author byline
Clean Markdown extraction
- 04
Multiple export formats: NDJSON, JSON arrays, Sitemap XML, CSV, TXT
Local-first privacy-focused
- 05
CLI tool for terminal workflows and automation scripts
A core productivity capability that teams use daily.
- 06
Native macOS desktop app with visual dashboard
A core productivity capability that teams use daily.
- 07
Privacy-friendly — all processing runs locally
A core productivity capability that teams use daily.
- 08
Configurable concurrency, depth limits, and polite delays
A core productivity capability that teams use daily.
- 09
AEO (Answer Engine Optimization) analysis
A core productivity capability that teams use daily.
Pros & Cons
What stands out
- Very fast local crawling thanks to Rust implementation
- Keeps all data local by default for privacy
- Markdown extraction ideal for content reuse
- Multiple export formats for easy integration
- Both CLI and GUI accommodate all users
- Free tier includes core features
Watch outs
- Newly launched — cloud features still in development
- Limited support for heavily JavaScript-rendered sites
- Desktop app currently macOS only
- Enterprise integrations not yet available
Pricing Plans
Crawler.sh Pricing
Choose the perfect plan for your needs. All plans include our core features with different usage limits and advanced capabilities.
Free
Pro
Need a Custom Solution?
Looking for enterprise features or custom pricing? Contact Crawler.sh directly for tailored solutions.
Contact SalesMost teams land on the Free plan.
Alternatives
FAQ
What is Crawler.sh and how does it work?
Crawler.sh is a productivity tool that fast, local-first web crawler with seo analysis and markdown extraction. run site audits from your terminal or desktop app.. It uses AI to help users improve productivity through analyzing input and generating relevant output.
Is Crawler.sh free to use?
Crawler.sh offers a freemium model with a free tier. You can get started without paying anything.
Does Crawler.sh have a free trial?
Yes — Free to try with no time limit.
What can Crawler.sh do?
More productivity Tools
Moltbot (formerly Clawdbot)
The only AI assistant that messages YOU first. Self-hosted, open-source, proactive personal AI.
Read review →
Fathom
Free AI meeting assistant that automatically records, transcribes, highlights, and summarizes Zoom, Google Meet, and Teams calls.
Read review →
Notion Custom Agents
Autonomous AI teammates that run workflows 24/7 across Notion, Slack, Mail, Calendar, and connected tools—no manual prompting needed after setup.
Read review →Affiliate Disclosure: We may earn a commission when you purchase through links on our site. This doesn't affect our editorial independence or the price you pay.
Crawler.sh
Freemium