Skip to content

firecrawl/web-agent

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

538 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Firecrawl Web Agent

License: MIT

Firecrawl Agent

Firecrawl Agent Demo

Firecrawl runs a research-grade autonomous agent at firecrawl.dev/app/agent, powered by Spark 1 models optimized for structured web research. This repo gives you the open-source foundation to build your own β€” fork it, swap models, add skills, and deploy however you want.

Get started

# 1. Install the Firecrawl CLI and authenticate
npx -y firecrawl-cli@latest init -y --browser

# 2. Scaffold an agent project
firecrawl create agent -t next

Open Source

Each layer builds on the one below it. Start at the top for a ready-to-use app, or go lower in the stack for finer control over the primitives.

Layer Description Get started
Next.js Template Chat UI, streaming, Skills, Subagents, structured output firecrawl create agent -t next
Express Template API server with Skills, Subagents, structured output firecrawl create agent -t express
↑
Agent Core Orchestrator built on Deep Agents (LangChain). Skills, Subagents, structured output firecrawl create agent -t library
↑
Firecrawl AI SDK Search, Scrape, Interact as Vercel AI SDK tools npm i firecrawl-aisdk
↑
Firecrawl SDK Core API client for Scrape, Search, Crawl, Extract npm i @mendable/firecrawl-js
↑
API Reference REST API, use from any language docs.firecrawl.dev

Examples

Level Examples
Next.js Full template
Express API server
Agent Core Basic Β· Structured output Β· Parallel Subagents Β· With Skills Β· Streaming
Firecrawl AI SDK npmjs.com/package/firecrawl-aisdk

How it works

The agent combines web tools with an AI model in a loop β€” it plans, acts, observes, and repeats until the task is done. The harness is Deep Agents (from LangChain), which gives us the plan-act loop, parallel task sub-agent spawning, and on-demand SKILL.md loading out of the box. Our agent-core wires Firecrawl's tools into that runtime and layers on structured output and streaming.

  • Harness β€” Deep Agents. Provides the agent loop, sub-agent spawning, skills loading, and context management.
  • Tools β€” Search, Scrape, Interact (browser automation), bash. Powered by firecrawl-aisdk.
  • Skills β€” reusable SKILL.md playbooks. Auto-discovered from agent-core/src/skills/definitions/, loaded on demand via Deep Agents' skills middleware.
  • Subagents β€” parallel workers for independent tasks, spawned via Deep Agents' task tool. Each has its own tool set and session state (e.g. an isolated interact browser session).
  • Output β€” structured results via formatOutput (JSON) and data processing via bashExec, a set of bash tools powered by just-bash.

Project structure

Directory What's inside
agent-core/ Core agent logic, orchestrator, Skills, tools
agent-templates/ Deployment templates - Next.js, Express, Library

License

MIT

About

πŸ”₯ Open-source web data agent optimized for structured web research

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors