Orchestrator Command

Overview

The Orchestrator Command (symphony orchestrator) starts the Symphony daemon as a standalone process. Unlike the comprehensive symphony start command which launches both web UI and orchestrator together, this command provides granular control—ideal for debugging, restarting the daemon independently, or containerized deployments.

The orchestrator is Symphony's heartbeat: it polls the database every 5 seconds, validates completion claims, dispatches agents to available slots, manages retries, and processes phase transitions.

How It Works

Three-Phase Initialization

1. Single-Instance Guard

When the orchestrator command is invoked, it checks for an existing orchestrator PID:

typescript

const existingPid = readOrchestratorPid(symphonyDir)
if (existingPid !== null) {
  if (isProcessRunning(existingPid)) {
    log.error('Already running. Kill it first or remove .symphony/orchestrator.pid')
    process.exit(1)
  } else {
    log.info('Stale PID file found. Cleaning up.')
    removeOrchestratorPid(symphonyDir)
  }
}
writeOrchestratorPid(symphonyDir)

The PID file (.symphony/orchestrator.pid) is the source of truth for single-instance detection. If a stale PID exists (process crashed), it's cleaned up automatically. The new PID is written before migrations to ensure cleanup on any subsequent failure.

2. FTS5 Learning Index Rebuild

The orchestrator rebuilds the learning search index at startup:

typescript

const allIssues = db.select(...).from(issues).all()
const parsed = allIssues.map(i => ({
  id: i.id,
  projectId: i.projectId,
  learnings: (i.learnings ?? []) as Learning[],
}))
rebuildLearningsFts(rawDb, parsed)

Why? Learning data is stored as JSON in the learnings column. The FTS5 virtual table enables fast full-text search without scanning the entire JSON blob. Rebuilding on startup ensures the index is consistent with the main table — critical after crashes or manual DB edits.

The learning FTS table has 4 searchable fields + 4 UNINDEXED metadata columns:

Searchable: pattern, context, applies_to
Metadata: learning_id, issue_id, project_id, learning_type (used for filtering/hydration)

3. Orchestrator Start + Signal Handlers

Once initialized, the Orchestrator instance is created and started:

typescript

const orchestrator = new Orchestrator(db, config, symphonyDir)

const shutdown = async () => {
  log.info('Shutdown signal received')
  orchestrator.stop()
  await orchestrator.awaitShutdown(30000)
  removeOrchestratorPid(symphonyDir)
  process.exit(0)
}

process.on('SIGINT', shutdown)  // Ctrl+C
process.on('SIGTERM', shutdown) // kill -TERM
process.on('exit', () => removeOrchestratorPid(symphonyDir))

orchestrator.start()
log.info('Running. Press Ctrl+C to stop.', { pid: process.pid })

Signal handlers ensure clean shutdown: the orchestrator stops accepting new dispatch, awaits in-flight agents for up to 30 seconds, removes the PID file, and exits.

Key Components

File	Responsibility
`cli/index.ts`	Command routing; routes 'orchestrator' to startOrchestrator()
`cli/commands/orchestrator.ts`	Thin CLI wrapper; initializes config/DB, calls orchestrator index
`server/orchestrator/index.ts`	Main entry point; PID management, FTS rebuild, shutdown handling
`server/orchestrator/orchestrator.ts`	Core Orchestrator class; implements tick loop (documented in tick-loop.md)
`server/database/migrate.ts`	Runs SQL migrations; ensures schema is current
`server/utils/config.ts`	Loads symphony.config.json + environment overrides
`server/utils/logger.ts`	Structured logging with level control (debug/info/warn/error)
`server/learning/search.ts`	FTS5 virtual table creation, rebuild logic, and search

Design Decisions

1. Separate Daemon Startup from Web UI

Choice: Independent symphony orchestrator command (vs integrated in Nuxt)
Rationale: Enables debugging workflows (restart orchestrator without rebuilding web assets); supports containerized deployments (separate container for daemon); simplifies process management
Trade-off: Adds manual coordination burden vs automatic co-management in start command

2. Single-Instance Guard with PID File

Choice: OS-level PID file + signal 0 liveness check (vs DB flag)
Rationale: Reliable across crashes; no DB dependency; fast detection; works across network boundaries
Trade-off: Requires cleanup on unclean shutdown; doesn't survive process manager restart without file cleanup

3. FTS5 Index Rebuild at Startup

Choice: Full index rebuild on every orchestrator start (vs incremental updates)
Rationale: Ensures consistency after crashes, manual edits, or schema changes; simple and correct; startup is infrequent (~once per dev session)
Trade-off: Slow on very large learning datasets (1000s of learnings); could optimize with delta tracking

4. 30-Second Graceful Shutdown Timeout

Choice: Wait up to 30s for in-flight agents before force-killing (vs immediate SIGKILL)
Rationale: Allows agents to commit work, record findings, and update status cleanly; 30s is user-acceptable for restarts
Trade-off: Long restarts if an agent hangs; could add timeout config flag

5. No Port Configuration for Orchestrator

Choice: Orchestrator accepts no --port or other flags (vs parameterizable start)
Rationale: Orchestrator doesn't listen on a port; it's purely event-driven; simplifies CLI surface
Trade-off: Can't customize timeouts or concurrency at CLI level; must edit symphony.config.json

6. Auto-Cleanup of Stale PID

Choice: Detect stale PID and remove automatically (vs failing with error)
Rationale: Reduces operator friction after crashes; enables restart without manual intervention
Trade-off: Hides potential issues (daemon hung but not truly dead); assumes OS signal 0 check is reliable

7. Structured Logging with Configurable Level

Choice: Emit structured logs; level controlled via config.log_level (default 'info')
Rationale: Production visibility; operators can enable debug logs via symphony.config.json without code changes
Trade-off: Adds log volume; requires operators to know about log_level config

8. FTS Index Metadata Columns UNINDEXED

Choice: Store metadata (learning_id, issue_id, etc.) in FTS5 table as UNINDEXED (vs separate normalization)
Rationale: Enables filtering and hydration without joining; data duplication negligible at scale
Trade-off: Slightly larger FTS table; simpler than normalization

Orchestrator Tick Loop (docs/server/orchestrator/tick-loop.md) — Core dispatch cycle (5-second polling, phase transitions, retries)
Learning System (docs/server/mcp/mcp-tool-registry.md) — How findings/learnings are recorded
Start Command (docs/cli/commands/start-command.md) — Comprehensive startup (web + orchestrator together)
Configuration (symphony.config.json) — Runtime config for log_level, concurrency, timeouts

Known Gaps

Orphaned PID File Cleanup — If process manager kills orchestrator with SIGKILL, PID file remains; next start detects stale PID and cleans automatically, but manual removal may be needed in pathological cases
No Startup Validation — Orchestrator doesn't probe dependencies (DB, file system, config) before starting; first error may come mid-tick
FTS Index Performance at Scale — Rebuilding learnings index is O(learnings), can take seconds with 10k+ learnings; no progress indicator
No Metrics Export — Orchestrator doesn't expose Prometheus metrics or similar; status visibility limited to symphony status CLI
Signal Handler Race Conditions — SIGINT during FTS rebuild or migrations could leave DB in partial state; relies on SQLite WAL mode recovery
Logging to stdout Only — No file rotation, no syslog integration; operators must use process manager to capture logs (systemd journal, Docker logs, etc.)
No Hot Reload — Config changes require orchestrator restart; no SIGHUP support for graceful reload
Stale PID Detection Gaps — Process might appear dead to signal 0 but still actively running (e.g., zombie state, zombie parent); heuristic is probabilistic

Example Usage

Start Orchestrator Standalone

bash

symphony orchestrator
# Output:
# ╭─────────────────────────────────╮
# │ Starting Symphony Orchestrator  │
# ╰─────────────────────────────────╯
#
# [Orchestrator] FTS5 learning index rebuilt (2341 learnings)
# [Orchestrator] Running. Press Ctrl+C to stop. (pid: 12345)

Start in Background (systemd, tmux, etc.)

bash

# systemd
systemctl start symphony-orchestrator

# tmux
tmux new-session -d -s symphony-orchestrator 'symphony orchestrator'

# nohup
nohup symphony orchestrator > /var/log/symphony.log 2>&1 &

Check Orchestrator Status

bash

symphony status
# Shows: Orchestrator Running (PID 12345)

Graceful Restart

bash

kill -TERM $(cat .symphony/orchestrator.pid)
symphony orchestrator

Force Kill (Unclean)

bash

kill -9 $(cat .symphony/orchestrator.pid)
rm .symphony/orchestrator.pid  # Manual cleanup
symphony orchestrator           # Starts fresh

Integration with `symphony start`

The comprehensive start command launches both web UI and orchestrator:

bash

symphony start --port 3333
# Spawns two processes via ProcessManager:
# 1. Nuxt dev server (port 3333)
# 2. Orchestrator daemon (stdio inherited)

The orchestrator command is used when you need isolated daemon control without the web UI (debugging, CI/CD, containerized deployments).

Orchestrator Command ​

Overview ​

How It Works ​

Three-Phase Initialization ​

1. Single-Instance Guard ​

2. FTS5 Learning Index Rebuild ​

3. Orchestrator Start + Signal Handlers ​

Key Components ​

Design Decisions ​

1. Separate Daemon Startup from Web UI ​

2. Single-Instance Guard with PID File ​

3. FTS5 Index Rebuild at Startup ​

4. 30-Second Graceful Shutdown Timeout ​

5. No Port Configuration for Orchestrator ​

6. Auto-Cleanup of Stale PID ​

7. Structured Logging with Configurable Level ​

8. FTS Index Metadata Columns UNINDEXED ​

Related Docs ​

Known Gaps ​

Example Usage ​

Start Orchestrator Standalone ​

Start in Background (systemd, tmux, etc.) ​

Check Orchestrator Status ​

Graceful Restart ​

Force Kill (Unclean) ​

Integration with symphony start ​

Orchestrator Command

Overview

How It Works

Three-Phase Initialization

1. Single-Instance Guard

2. FTS5 Learning Index Rebuild

3. Orchestrator Start + Signal Handlers

Key Components

Design Decisions

1. Separate Daemon Startup from Web UI

2. Single-Instance Guard with PID File

3. FTS5 Index Rebuild at Startup

4. 30-Second Graceful Shutdown Timeout

5. No Port Configuration for Orchestrator

6. Auto-Cleanup of Stale PID

7. Structured Logging with Configurable Level

8. FTS Index Metadata Columns UNINDEXED

Related Docs

Known Gaps

Example Usage

Start Orchestrator Standalone

Start in Background (systemd, tmux, etc.)

Check Orchestrator Status

Graceful Restart

Force Kill (Unclean)

Integration with `symphony start`