ATOM Documentation

Historical Sync Backfill — Complete Fix Summary

Problem

Outlook memory ingestion backfill jobs were stuck in paused/failed/cancelled states with 0 entities and 0 neural links. Workers were being killed mid-processing, and semantic extraction was failing with 401 authentication errors.

Root Causes (9 bugs found and fixed)

Known Issues (Not Yet Fixed)

Additional Improvements

Performance

• **Chunk size**: 1000→100 records to avoid Microsoft Graph 504 errors
• **Progress calculation**: Uses has_more flag instead of per-page total_count (fixed 150% display bug)
• **Email body**: Full content (up to 10KB) instead of 500-char bodyPreview

Architecture

• **Dual ingestion paths**:
• **Semantic memory** (LanceDB): Vector embeddings for all tenants, no API key required
• **Structured memory** (GraphRAG): Entity extraction for tenants with BYO key
• **Worker VM**: 2GB memory (1-core) for GraphRAG ingestion headroom
• **Autoscaling**: Multi-worker (1-3 machines) via Fly Machines API

Deployment Timeline

• **2026-05-01 00:00-01:00 UTC** - Initial fixes (timeout, 504 retry, worker shutdown)
• **2026-05-01 01:00-02:00 UTC** - Session fixes (re-query, asyncio, BYOK key)
• **2026-05-01 02:00-03:00 UTC** - Email body + worker process fixes
• **2026-05-01 03:00-04:00 UTC** - Reaper timeout + dual-memory architecture

Architecture

Email record (100 per chunk)
    │
    ├─→ _extract_structured_entities() → DiscoveredEntity → Postgres
    │   └─→ Rule-based: from, to, subject, content fields
    │
    ├─→ graphrag.ingest_document() → GraphNode/GraphEdge → Postgres
    │   └─→ LLM-based: people, orgs, topics from email body
    │   └─→ Requires: Tenant BYOK key (OPENAI_API_KEY)
    │
    └─→ lancedb.add_document() × 100 → LanceDB
        └─→ Full text + subject → Vector embeddings
        └─→ Requires: None (runs for all tenants)

Deployment Commands

Deploy Latest Changes

git pull origin main
fly deploy -a atom-saas --strategy immediate

Verify Deployment

# Check machines started
fly status -a atom-saas

# Check worker logs
fly logs -a atom-saas --machine 2861d27a3414e8 | grep -E "(ROLE|Starting|Loaded service)"

Verification

Check Recent Jobs

SELECT
    id,
    status,
    records_processed,
    entities_extracted,
    relationships_extracted,
    created_at,
    completed_at
FROM historical_sync_jobs
WHERE created_at > NOW() - INTERVAL '1 hour'
ORDER BY created_at DESC
LIMIT 5;

Monitor Worker Activity

# Real-time worker logs
fly logs -a atom-saas --machine 2861d27a3414e8 | grep -E "(dequeue|Fetched|Persisting|entities|relationships|LanceDB)"

# Check for errors
fly logs -a atom-saas --machine 2861d27a3414e8 | grep -E "(error|Error|ERROR|failed|Failed)"

Verify Semantic Extraction

# Check LanceDB documents stored
# (Requires access to production database)

# Check GraphRAG entities created
SELECT COUNT(*) FROM graph_nodes
WHERE tenant_id = '31c06fc4-db22-4740-83ea-48ac14f25810'
  AND created_at > NOW() - INTERVAL '1 hour';

Expected Performance

Per 100-record chunk:

• **Fetch from Outlook**: ~2-5 seconds
• **LanceDB embeddings**: ~30-60 seconds (synchronous, one-at-a-time)
• **GraphRAG entity extraction**: ~5-15 minutes (if BYOK key available)
• **Total time**: ~8-10 minutes (LanceDB only) or ~15-20 minutes (both paths)

**Note**: LanceDB calls are currently synchronous (not batched with asyncio.gather()). This is a future optimization opportunity.

Current Status

✅ Working

• Worker stays alive and processes jobs continuously
• Fetches 100 emails per chunk without 504 errors
• Full email body (10KB) available for processing
• No premature reaper kills (30-minute threshold)
• LanceDB embeddings created for all tenants
• BYOK tenants get GraphRAG entity extraction

⏳ Pending

• **Transaction error fix** needed for GraphRAG path
• **LanceDB batching** optimization (future)
• Production backfill completion metrics

🔧 Known Limitations

• GraphRAG path blocked by transaction errors (Bug #10)
• Single-core worker limits concurrent processing
• Synchronous LanceDB calls add ~30-60 seconds per chunk

Next Steps

1. **Fix transaction errors** in GraphRAG ingestion path
2. **Verify production backfill** completes successfully
3. **Optimize LanceDB** with batched concurrent calls
4. **Monitor reaper** to ensure 30-minute threshold is sufficient