Convex Firecrawl Agent


Publishable Convex Component that wraps Firecrawl's async /v2/agent API as durable, reactive job resources.

src="./docs/assets/readme-screenshot.webp"
alt="Screenshot of the Convex Firecrawl Agent demo UI"
width="900"
/>

Turn long-running agent workflows into something you can treat like any other Convex data model: create once, subscribe reactively, and render status/results without writing your own polling + retry + storage plumbing.

What you get

- Durable jobs: Create once; observe status and results via reactive queries.
- Webhook-first architecture: Real-time status updates via Firecrawl webhooks, with bounded reconciliation fallback for reliability.
- Retry with backoff: Kickoff failures (429, 5xx, network errors) retry up to 5 times with exponential backoff.
- Cancellation: Cancel locally and (best-effort) remotely. If Firecrawl already completed, the job ends completed (completion wins).
- Cleanup: deleteJob removes terminal jobs (and their events, plus any stored result file) to keep UIs tidy and support retention policies.
- Large result handling: Results larger than 750 KiB are stored in Convex file storage with an inline preview; use getResultDownloadUrl when resultStorageId is present.
- Observability: Optional per-job event timeline for debugging and UI timelines.

Found a bug or have a feature request? Please open an issue: https://github.com/gitmaxd/convex-firecrawl-agent/issues

Model selection

The component supports Firecrawl's agent models via the optional model parameter:

| Model | Cost | Best for |
|-------|------|----------|
| spark-1-mini | 60% cheaper | Most tasks (default) |
| spark-1-pro | Standard | Complex multi-domain research |

Start with the default (spark-1-mini) for cost efficiency. Use spark-1-pro when accuracy is critical or tasks require advanced reasoning across multiple sources.

See Firecrawl model docs for detailed comparison.

Demo

See the example/ directory for a complete working demo (Convex backend + Vite React UI) that demonstrates the full job lifecycle: create → observe → cancel → delete, with live event streaming and result viewing.

Install

``sh
npm i @gitmaxd/convex-firecrawl-agent
`

Prerequisites

- Convex project: If you don't have one yet, run npx convex dev to create a new project (you'll be prompted to log in or create a free account).
- Firecrawl API key: Get one at firecrawl.dev, then set it in your Convex deployment:
`sh
npx convex env set FIRECRAWL_API_KEY ""
`

Add to your Convex app

`ts
// convex/convex.config.ts
import { defineApp } from "convex/server";
import convexFirecrawlAgent from "@gitmaxd/convex-firecrawl-agent/convex.config.js";

const app = defineApp();
app.use(convexFirecrawlAgent);

export default app;
`

Usage

$3

Your Convex app should not call component functions directly from the client. Instead, define app functions that:

- authenticate the caller
- map identity → userId
- delegate to the component

Use exposeApi to generate app-level queries/mutations with your auth mapping:

`ts
// convex/firecrawlAgent.ts
import { components } from "./_generated/api";
import { exposeApi } from "@gitmaxd/convex-firecrawl-agent";

export const {
createJob,
getJob,
listJobs,
cancelJob,
deleteJob,
listJobEvents,
} = exposeApi(components.convexFirecrawlAgent, {
auth: async (ctx) => {
const userId = (await ctx.auth.getUserIdentity())?.subject;
if (!userId) throw new Error("Unauthorized");
return userId;
},
includeEvents: true,
// Optional override for tests or custom limits:
// resultStorageThresholdBytes: 750 * 1024,
});
`

$3

From your client (via your app mutations/queries) or from other Convex functions:

`ts
// create
const jobId = await ctx.runMutation(api.firecrawlAgent.createJob, {
targetId: "doc:123",
prompt: "Extract a short summary",
urls: ["https://news.convex.dev"],
});

// observe (reactive when used in a client query)
const job = await ctx.runQuery(api.firecrawlAgent.getJob, { jobId });

// cancel (optional)
await ctx.runMutation(api.firecrawlAgent.cancelJob, { jobId });

// cleanup (recommended for terminal jobs)
await ctx.runMutation(api.firecrawlAgent.deleteJob, { jobId });
`

Guardrails & limits

To keep job documents durable and Firecrawl requests valid, the component enforces a few guardrails up front (the mutation throws and the job is not created if these fail):

- prompt length: maximum 10,000 characters. If exceeded: Prompt exceeds maximum length of 10000.
- schema must be JSON-serializable: the schema is stored on the job and forwarded as JSON, so it must round-trip through JSON.stringify(...) (no BigInt, functions, or circular references). If invalid: Schema must be JSON-serializable.
- Large-result storage threshold must be valid: resultStorageThresholdBytes (and the RESULT_STORAGE_THRESHOLD_BYTES env override used by the helper wrappers) must be a finite number ≥ 0.
- If you pass an invalid resultStorageThresholdBytes, the wrapper throws: resultStorageThresholdBytes must be a finite number >= 0.
- If RESULT_STORAGE_THRESHOLD_BYTES is set but invalid, it is ignored (treated as unset).
- 0 is valid and forces results to always be stored in file storage.
- maxCredits must be valid: when provided, must be a finite integer > 0. Invalid values (non-finite, non-integer, or ≤ 0) cause the mutation to throw and no job is created.
- model selection: optional model parameter accepts "spark-1-mini" or "spark-1-pro" to select the Firecrawl agent model.

Type-safe integration (single source of truth)

`ts
import { exposeApi } from "@gitmaxd/convex-firecrawl-agent";
import { components } from "./_generated/api";

const agentApi = exposeApi(components.convexFirecrawlAgent, {
auth: async (ctx, operation) => {
const userId = (await ctx.auth.getUserIdentity())?.subject;
if (!userId) throw new Error("Unauthorized");
return userId;
},
});

type AgentApi = typeof agentApi;
type AuthFn = Parameters[1]["auth"]; // (ctx, op) => Promise
type CreateJobReturn = Awaited>; // string jobId
`

Large result handling

If a job result exceeds 750 KiB, the component stores the full payload in Convex file storage and keeps an inline preview. Use getResultDownloadUrl to fetch a signed download URL when resultStorageId is present.

Observability: job events

Enable includeEvents: true in exposeApi to record a per-job event timeline. The demo UI uses this to show state transitions and retry/backoff behavior.

Troubleshooting

- Missing FIRECRAWL_API_KEY: set it with npx convex env set FIRECRAWL_API_KEY "...".
- Jobs stuck in processing: check the job's event timeline and fields like attempt, lastHttpStatus, firecrawlStatus, and error; verify your API key is valid.

See the full troubleshooting guide: docs/10-troubleshooting.md.

Documentation

- Start here: docs/02-getting-started.md
- Full doc index: docs/README.md

References

- Convex Components Authoring: https://docs.convex.dev/components/authoring
- Firecrawl Agent feature docs: https://docs.firecrawl.dev/features/agent

License

Apache-2.0. See LICENSE`.