Inside SearchFit Engineering: Why We Chose Trigger.dev v4

Introduction

If your platform speaks to LLMs at scale, running multi-agent content generation steps that span Claude, GPT-4o, and vector databases, you eventually hit a wall with background job orchestration unless using modern run engines like trigger dev v4. Not the fun kind of wall where you add a queue and call it a day, but the kind where straggling workflows eat resources, observability evaporates mid-pipeline, and retries spawn duplicate side effects that cost real money.

That’s where SearchFit was six months ago. Our content engine orchestrates thousands of answer engine optimization (AEO) audits and content generation runs daily, coordinating four specialized agents, Researcher, Writer, Validator, Scorer, across ChatGPT, Perplexity, Gemini, and Claude. We built the multi-agent architecture from the ground up, but the job runner was a Frankenstein of manual Redis queues and recursive Node.js worker loops. It worked until we tried to scale past 1,000 concurrent runs, then we hit phantom failures, inconsistent retries, and audit results that couldn’t be traced back to the job that triggered them.

This article is a step-by-step account of how we evaluated, migrated, and hardened our job orchestration onto Trigger.dev v4. It’s written for platform engineers making similar decisions, whether you’re running AI agents, content pipelines, or anything that chains dozens of async tasks with state. We won’t call Trigger.dev “the best” or “the leading” anything. Instead, we’ll show you the concrete numbers, the architectural trade-offs, and the operator-level tactics that convinced us to go all-in.

By the end, you’ll have a repeatable framework for evaluating job infrastructure, plus the specific patterns we used to maintain 99.7% delivery accuracy on AI-generated content while cutting orchestration overhead by 40%.

Prerequisites

Before diving into the migration process, make sure your team has these pieces in place. We’re assuming you’re already running some form of background work, if you’re starting from zero, the decision is simpler, but the chaos is still real.

• TypeScript proficiency: Trigger.dev v4 is built on TypeScript-first developer experience. Our existing codebase was TypeScript, but you’ll need at least intermediate TypeScript skills to model job payloads, infer types, and debug compile-time issues in workflow definitions.
• A running Node.js 20+ environment: Trigger.dev v4 requires Node.js 20 or later for its runtime. Our previous worker loop ran on Node 18; upgrading forced us to clean up deprecated assert usage and switch to native fetch, minor but necessary. Brush up on Node.js 20 changes if you’re behind.
• Knowledge of your current job topology: Before you evaluate any orchestration platform, map every job type, its dependencies, retry behavior, and average latency. If you can’t draw it on a whiteboard, you’re not ready. We spent two weeks profiling our content engine’s agent pipeline to get accurate numbers.
• A centralized secrets store (we use Vault, but AWS Secrets Manager or even environment variables work). Trigger.dev integrates with Vercel’s integration patterns, but you’ll need to securely inject API keys for up to 12 services (OpenAI, Anthropic, Perplexity, Pinecone, etc.). We leaned heavily on OAuth and API key management patterns to keep tokens scoped and rotated.
• Observability stack natively accepting structured logs: We use Grafana Loki + Tempo. Trigger.dev v4 emits OpenTelemetry spans; we’ll show how to route them into your existing tools.

Pro Tip: Before touching any code, run an audit of your current failure modes. We identified 17 distinct failure patterns in our old system (connection resets, LLM rate limits, dimension mismatches, etc.), each became a test case during migration.

Step 1: Diagnose the Orchestration Gap

You can’t fix what you can’t measure. The first step is to instrument your current job runner to expose its actual behavior under load. For us, that meant hooking into our worker pool and logging precise timestamps for every state transition.

1.1 Profile Job Execution Patterns

Our old architecture used a single Redis-backed BullMQ queue with 48 concurrent workers. Each worker picked up a job from the aeo-audit queue, executed a series of LLM calls, and wrote results to PostgreSQL. Simple, until we looked at tail latencies.

At 1,000 concurrent jobs, the 95th percentile latency for a full audit jumped from 18 seconds to 143 seconds. Worse, 4.2% of jobs failed silently, the LLM returned a payload that didn’t match our validation schema, but the worker itself never errored. That meant the job stayed “completed” while the downstream content was garbage. We discovered this only after a re-run of 10,000 AEO audits with Claude 4.7 showed a 6% discrepancy in citation accuracy from what the queue thought it had produced.

Run a similar profile: pick a representative week, export all job metadata, and calculate latency percentiles, failure rates by error type, and retry amplification. You’ll likely find what we did: a single monolithic queue becomes a bottleneck when jobs have heterogeneous durations (some audits take 10 seconds, pillar content generation takes 4 minutes). Long-running jobs starve short ones, and your retry logic doesn’t distinguish between transient and permanent errors.

1.2 Identify Idempotency and Side-Effect Risks

Ask yourself: “If this job retries after a partial completion, will I double-charge an LLM token budget or write duplicate rows?” Our old system had no true idempotency keys; retries spammed the OpenAI API and created orphaned embeddings in Pinecone. That’s when we knew we needed a platform with built-in idempotency and precise at-least-once semantics.

Document every external side effect your jobs produce, LLM API calls, database writes, outbound webhooks, file writes, and note whether they are idempotent-safe. If they aren’t, you’ll need to wrap them with idempotency keys or transactional outboxes. Trigger.dev v4 supports this natively, as we’ll detail.

Warning: Skipping this step leads to the worst kind of bug: duplicate content published under different IDs, polluting your citation monitoring data and wasting tokens.

Step 2: Map the Job Topology for Multi-Agent Workflows

Once you have the baseline, you can design a topology that matches your actual pipeline stages. SearchFit’s content engine isn’t a single job; it’s a directed acyclic graph (DAG) of agent tasks.

2.1 Decompose Into Distinct Jobs

For a single AEO audit, the old system treated the entire run as one job. We decomposed it into five discrete jobs, each with its own retry policy and timeout:

1. Discovery Job: Queries our keyword and competitor citation index, identifies gaps. Runs for 15-30 seconds, high parallelism.
2. Research Job: Fires multiple LLM calls to gather entity context. Susceptible to rate limits; needs exponential backoff.
3. Writing Job: Generates AI-optimized content block by block. Uses Claude 4.7 extended thinking for pillar pages; consumes the most tokens.
4. Validation Job: The validator agent checks hallucination, schema compliance, and citation accuracy. If it fails, we must roll back to Writing or Research without reprocessing everything.
5. Scoring Job: Computes predicted visibility scores across engines, updates dashboards.

Trigger.dev v4’s childWorkflows and step primitives map directly onto this DAG. Jobs can wait for children, pass partial results, and propagate failures upstream. That’s a night-and-day difference from a flat queue.

2.2 Decide on State Boundaries

How far can a job advance before you consider its output durable? We decided each step must checkpoint its partial output to a dedicated workflow_state table before invoking the next. If the next step fails, the preceding step’s checkpoint ensures we don’t replay expensive LLM calls. Trigger.dev v4’s built-in idempotencyKey and auto-persisted step output meant we could drop our hand-rolled checkpoint logic entirely.

Map your own boundaries with a diagram; we used the open-source Trigger.dev workflow definitions as a reference for how to structure step payloads. The official docs provide excellent examples of multi-step workflows with TypeScript generics that infer types across steps.

Step 3: Evaluate Trigger.dev v4’s Run Engine

This is the core decision. We evaluated three orchestration options: staying with BullMQ + custom sharding, moving to a managed queue like AWS SQS with Step Functions, or adopting Trigger.dev v4. The evaluation criteria were: determinism, observability, cost, and developer ergonomics for TypeScript-first teams.

3.1 Deterministic Execution Guarantees

The hardest bug we ever debugged was a flaky content generation job that produced different results on retry because the LLM temperature setting was re-read from a Redis cache that had expired. Traditional queues retry the entire function; if any part is non-deterministic, you get inconsistent outcomes.

Trigger.dev v4’s Run Engine guarantees that each step sees the same input on retry, it serializes step outputs and replays them. This is achieved through a combination of step-level caching and a global runId that ties the entire dataflow to an immutable snapshot. That means if your LLM call fails due to a temporary rate limit, the step retries with the exact prompt, temperature, and seed as before. For AI workloads, this is table-stakes.

We validated this by running 500 identical “GPT-4o summarization” jobs with a simulated 20% transient failure rate. On our old queue, 12% of final outputs diverged from the original intent. On Trigger.dev v4, zero diverged, the retries produced byte-exact matches because the input was frozen.

3.2 Observability Per Step

Our old system emitted a single “job completed” event. Trigger.dev v4 emits spans for every step, with automatic correlation to the parent workflow. We piped these into our existing Grafana stack using the OpenTelemetry exporter and immediately saw which steps were slowest, which LLM providers added latency, and where back pressure built.

For an engineering team, this matters because it lets you set SLOs per step: “Research must complete within 10 seconds for 95th percentile.” If Research chokes on Perplexity’s API, you can trigger a fallback to an alternative provider without retrying the entire pipeline. We discuss this pattern in detail in our agent memory and long-context research architecture.

3.3 Cost Efficiency

We priced Trigger.dev v4’s platform against our self-managed Redis and compute costs. The old system required three t3.xlarge instances (24 vCPUs) to handle peak loads, plus Redis replication and manual patching, roughly $3,200/month. Trigger.dev v4’s team plan for our volume (2.5M step invocations/month) comes in at $900/month, and we retired those three instances.

But the real cost win was token savings. Because deterministic retries eliminated duplicate LLM calls, our monthly OpenAI/Anthropic costs dropped 34%. That alone pays for the platform several times over. You can see the detailed breakdown in our Claude 4.7 extended thinking results.

Pro Tip: Trigger.dev v4 charges per step execution, not per workflow run. If you split coarse jobs into fine-grained steps, you might increase step count. But the cost is still lower than wasted compute, and the observability gain makes it worth it. We initially over-fragmented and then consolidated some trivial steps; measure, don’t guess.

Step 4: Migration Planning and Execution

The migration itself took three weeks from first commit to full cutover. We’ll outline the exact sequence, including the canary approach that kept production safe.

4.1 Set Up the Trigger.dev v4 Environment

We used the Trigger.dev v4 getting started guide video to spin up a development environment. The steps:

1. Install @trigger.dev/sdk v4 and @trigger.dev/build.
2. Initialize the project structure (trigger.config.ts) with our endpoint and project ID.
3. Create a jobs directory and define the first five job types (Discovery, Research, Writing, Validation, Scoring) as exported workflow functions.
4. Write integration tests that mock LLM calls using the @trigger.dev/testing utilities. The SDK’s createAppRoute helper lets you run local tests with the same deterministic engine that runs in production.

We hit one snag: our VSCode TypeScript server kept inferring any for step output types. The fix was to explicitly annotate return types on step functions, a minor discipline that improved code review.

4.2 Canary Migration With Dual Writes

We couldn’t afford to halt content generation. Instead, we ran both systems in parallel for ten days, with a feature flag routing 10% of new audit requests to Trigger.dev. The steps:

• Forked the production database so both systems read the same inputs without cross-contamination.
• Implemented a workflow_dispatcher service that checked a random percentage and enqueued to either BullMQ or Trigger.dev.
• Built a sidecar that compared the output of each system for the same input (same brand, same keyword) and logged differences.

Initially, we saw 3% output divergence, mostly because the new deterministic engine produced more consistent citations. The old engine sometimes varied LLM results because of cache-miss-induced temperature changes. After tuning our prompt seeds to be immutable, divergence dropped to 0.1%.

4.3 Gradual Rollout and Monitoring

Every 48 hours, we bumped the percentage: 10% → 30% → 70% → 100%. The monitoring dashboard tracked key metrics:

• Workflow completion rate: stayed at 99.7% throughout.
• P99 latency: dropped from 143s to 67s because short Discovery jobs no longer queued behind long Writing jobs.
• Duplicate content incidents: dropped from 17 per week to zero.

We used real-time citation monitoring APIs to verify that the output fed into our AI citation tracker correctly. The migration was complete when we turned off the BullMQ workers and archived their config.

Warning: During the 70% phase, a surge of 1,200 concurrent jobs briefly exhausted our LLM rate limits because the new system’s higher concurrency exposed a hard cap we’d never hit before. We fixed this by adding a rateLimit middleware in the Trigger.dev workflow configuration, a one-line change.

Step 5: Observability and Error Recovery Patterns

Post-migration, we discovered far more about our system because every step emits traces. Here are the patterns we use daily.

5.1 Per-Workflow Dashboards

We built Grafana dashboards that segment by workflow name. For example, the “Writing Job” dashboard shows:

• Average tokens consumed per engine (GPT-4o vs. Claude)
• Failure rate by error type (rate limit vs. schema validation)
• Step latency heatmap

This immediately revealed that Claude 4.7 writing steps for pillar content were taking 4x longer than expected due to an overly long system prompt. Trimming the prompt cut 22% off the step time without affecting quality.

5.2 Custom Error Handling With Steps

Trigger.dev v4’s step.run returns a union type you can match on. We defined an error taxonomy:

type LLMError = { type: 'rate_limit'; retryAfter: number } | { type: 'schema_mismatch' } | { type: 'timeout' };

In the Writing step, if the error is rate_limit, we throw a retryWithBackoff error that Trigger.dev’s SDK understands and reschedules. If it’s schema_mismatch, we log a structured event and skip to a fallback Writer. This kind of nuanced retry logic would have been a tangled mess in BullMQ.

5.3 Long-Running Workflows and Cancellation

Some pillar content jobs take 8+ minutes. Trigger.dev v4 supports heartbeat and cancellation signals. We use heartbeats to detect stalled jobs (LLM provider hangs) and cancel them before token budgets blow. The Trigger.dev v4 GA announcement details the new cancellation signal propagation, which works across child workflows too.

Step 6: Scaling and Cost Optimization

As our volume grew, we squeezed performance further.

6.1 Concurrency Control

Trigger.dev v4 lets you set maxConcurrency per workflow and per step. We set maxConcurrency: 200 on the Research step to avoid overwhelming Perplexity’s API. But we also set a global maxConcurrency: 500 on the entire project so that we never exceed our LLM provider agreement.

6.2 Batch Processing

For bulk AEO audits, we now batch 50 Discovery jobs into a parent workflow that fans out via childWorkflow. The parent aggregates results and emits a single completion event. This reduced API overhead and allowed us to process 10,000 audits in 4.3 hours instead of 11 hours.

6.3 Local Development and CI

We integrated Trigger.dev’s local development server into our CI pipeline. Every PR runs a suite of mock workflows that complete in under a minute, verifying that no step introduces breaking changes to the workflow topology. This caught a regression where a step’s output type changed, breaking downstream validation.

Conclusion and Key Takeaways

Moving to Trigger.dev v4 wasn’t a “lift and shift”, it forced us to decompose our monolith of a job queue into a well-structured DAG of observable, idempotent steps. The process taught us several hard lessons:

1. Profile before you migrate: Without measuring your current system’s failure modes and latencies, you’ll optimize the wrong things.
2. Determinism matters for AI pipelines: Non-deterministic retries produce divergent outputs that corrupt your data. Trigger.dev v4’s step caching eliminates that class of bug.
3. Step-level observability is a superpower: Knowing that “Validation fails 8% of the time for Perplexity-sourced entities because of JSON truncation” lets you fix the root cause instead of blindly retrying.
4. Co-locate business logic with orchestration: Writing jobs in TypeScript, with type-safe payloads that flow across steps, means your job runner is no longer a separate system that drifts from your application code.
5. Canary deployments with dual writes are non-negotiable: Without a side-by-side comparison, you’ll ship errors that only surface at scale.

Today, SearchFit’s multi-agent content engine processes over 50,000 jobs weekly, with a 99.7% completion rate and predictable costs. Our team spends zero time debugging queue state or chasing phantom retries. That’s not marketing hype, it’s what happens when your job infrastructure aligns with your agent architecture.

If you’re evaluating orchestration for AI workloads, start by reading the Trigger.dev v4 documentation and the official GA post. Then, follow the profiling steps we outlined, and don’t skip the idempotency check. For deeper integration patterns, check out our guides on building custom AEO workflows via API and integrating SearchFit with n8n for self-hosted environments.