AnyCrawl

🚀 Scrape a single URL and extract content in various formats. AnyCrawl turns websites into LLM-ready structured data with high performance multi-threading. Best for: Extracting content from a known, single page (article, docs page, product page). Not recommended for: Broad discovery across many pages (use anycrawl_crawl); open-ended questions across the web (use anycrawl_search). Common mistakes: Using a headless engine unnecessarily (prefer cheerio for static pages); requesting heavy formats (screenshots/rawHtml) when not needed; setting large timeouts without cause. Prompt example: "Scrape this page and return clean markdown: https://example.com/blog/post" Engine guidance: Use cheerio for static HTML, playwright for dynamic apps, puppeteer for Chrome automation. Usage example without formats: { "name": "anycrawl_scrape", "arguments": { "url": "https://news.ycombinator.com", "engine": "cheerio" } } Usage example with formats: { "name": "anycrawl_scrape", "arguments": { "url": "https://example.com/docs/page", "engine": "playwright", "formats": ["markdown"], "wait_for": 1500 } } Returns: { url, status, jobId, title, html?, markdown?, metadata?, timestamp }

必填参数：url、engine

🌐 Start an asynchronous crawl job to scrape multiple pages from a website. Perfect for comprehensive site analysis, content aggregation, and bulk data collection with native multi-threading. Best for: Multi-page coverage of a site or section (docs, blogs, categories). Not recommended for: A single known page (use anycrawl_scrape); open-ended web-wide queries (use anycrawl_search). Common mistakes: Setting limit too high; using strategy="all" unintentionally; requesting heavy formats (e.g., full-page screenshots) across many pages; deep max_depth without need. Prompt example: "Crawl the docs section and return markdown for up to 100 pages." Strategy guidance: - same-domain (default) is safest; - same-hostname for subdomain specificity; - same-origin for strict protocol+domain; - all for external links (use cautiously). Usage example (basic): { "name": "anycrawl_crawl", "arguments": { "url": "https://docs.example.com/*", "engine": "cheerio", "limit": 100, "max_depth": 5 } } Usage example (with formats and filters): { "name": "anycrawl_crawl", "arguments": { "url": "https://example.com/blog/*", "engine": "cheerio", "formats": ["markdown"], "exclude_paths": ["/tags/*", "*.pdf"], "include_tags": ["article", "main"], "limit": 50 } } Returns: Job creation info { job_id, status, message }. Use anycrawl_crawl_status and anycrawl_crawl_results to monitor and retrieve data.

必填参数：url、engine

📊 Check the status of an asynchronous crawl job. Monitor progress, view statistics, and track completion status. Best for: Ongoing monitoring of a crawl created with anycrawl_crawl. Not recommended for: Fetching page content (use anycrawl_crawl_results). Usage example: { "name": "anycrawl_crawl_status", "arguments": { "job_id": "7a2e165d-8f81-4be6-9ef7-23222330a396" } } Returns: { job_id, status, start_time, expires_at, credits_used, total, completed, failed }

必填参数：job_id

📄 Get results from a completed or in-progress crawl job. Supports pagination for large crawls with thousands of pages. Best for: Retrieving crawled page data and metadata after or during a crawl. Common mistakes: Forgetting to paginate via skip when next is present; requesting extremely large pages of data. Usage example: { "name": "anycrawl_crawl_results", "arguments": { "job_id": "7a2e165d-8f81-4be6-9ef7-23222330a396", "skip": 0 } } Returns: { status, total, completed, creditsUsed, next?, data: Array<pageResult> } where next can be used as the next skip value for pagination.

必填参数：job_id

🛑 Cancel a pending or running crawl job. Useful for stopping long-running crawls or correcting configuration mistakes. Best for: Stopping crawls that are no longer needed or misconfigured. Not recommended for: Completed jobs (cancellation has no effect). Usage example: { "name": "anycrawl_cancel_crawl", "arguments": { "job_id": "7a2e165d-8f81-4be6-9ef7-23222330a396" } } Returns: Confirmation { job_id, status } indicating cancellation state.

必填参数：job_id

🔍 Search the web using AnyCrawl's powerful search engine integration. Get SERP (Search Engine Results Page) data with optional content scraping for comprehensive research. Best for: Finding specific information across multiple websites when you don't know which site has it; retrieving the most relevant content for an open-ended query. Not recommended for: Searching the filesystem; when you already know the exact website to extract (use anycrawl_scrape); when you need comprehensive coverage of a single site (use anycrawl_crawl). Common mistakes: Using crawl for open-ended questions; requesting heavy scrape_options (large formats/timeouts) unnecessarily. Prompt example: "Find the latest research papers on AI published in 2023." Sources: web (default). Image/news verticals are not yet supported in this tool. Scrape options: Only set scrape_options when absolutely necessary. Prefer small limits (≤5) and minimal formats (e.g., ["markdown"]) to avoid timeouts. Usage example without formats: { "name": "anycrawl_search", "arguments": { "query": "top AI companies", "limit": 5, "scrape_options": { "engine": "cheerio" } } } Usage example with formats: { "name": "anycrawl_search", "arguments": { "query": "latest AI research papers 2023", "limit": 5, "lang": "en", "country": "US", "scrape_options": { "engine": "cheerio", "formats": ["markdown"], "wait_for": 1000 } } } Returns: Array of search results with optional scraped content for each result URL.

必填参数：query、scrape_options