[FEATURE] add cooperative cycling and Storyline content ingestion engine

[github-actions[bot]]

Originally posted as issue #2032 by @thedotmack
#2032

Type: feature / enhancement request (re-filed from maintainer's own PR during backlog cleanup)

Summary

• Cooperative cycling: Replace abort-based session cycling with MCP custom tools (cycle_status, complete_cycle) that let the agent decide when to cycle based on context pressure
• ContentIngestionRunner: Sequential file-by-file content ingestion orchestrator with SQLite progress tracking, content hash dedup, and cancellation support
• StorylineRoutes: HTTP API (POST /run, GET /status, POST /cancel) for the Storyline skill to drive content ingestion from the engine

New files

File	Purpose
EndlessModeToolServer.ts	MCP tool server with cycle_status and complete_cycle for cooperative cycling
ContentIngestionRunner.ts	File-by-file ingestion orchestrator with observer pipeline integration
StorylineRepository.ts	SQLite CRUD for storyline_runs and storyline_files tables
StorylineRoutes.ts	Express route handlers for /api/storyline/* endpoints
migrations.ts (008)	Schema for storyline tracking tables

Modified files

File	Change
EndlessRunner.ts	Refactored from abort-based to cooperative cycling via MCP tools
worker-service.ts	Register StorylineRoutes
logger.ts	Add STORYLINE and STORYLINE_DB component types

Architecture

Storyline Skill (separate repo)
  → POST /api/storyline/run (files + mode + goal)
  → Engine: ContentIngestionRunner reads files sequentially
  → Engine: Observer captures structured observations per file
  → Engine: Progress tracked in SQLite
  → Storyline polls GET /api/storyline/status
  → Observations searchable via mem-search

ContentIngestionRunner composes shared utilities (findClaudeExecutable, getModelId, buildIsolatedEnv) — it does NOT extend EndlessRunner.

Test plan

• bun test passes (1181 pass, 0 fail, 3 skip)
• Manual E2E: create inbox with 5 markdown files, POST /run, poll /status, verify observations are searchable
• Test cancellation mid-flight via POST /cancel
• Test content hash dedup on re-run (same files skipped)

🤖 Generated with Claude Code

---

Re-filed as an issue during the April 2026 backlog cleanup. Original: #1577