ATOM Documentation

Production Cleanup & Test Data Prevention Summary

**Date:** 2026-04-09

**Purpose:** Brennan incident recovery - Remove all test data except Brennan tenant

**Author:** Claude Code

---

🚨 EXECUTIVE SUMMARY

Your production database contains **1,829 tenants** with **3,658-18,290 agents**, most of which are test data. This document provides the tools and procedures to clean up the database and prevent future test data contamination.

**Brennan Tenant (Keep):**

• ID: 31c06fc4-db22-4740-83ea-48ac14f25810
• Name: Brennan Machinery
• Subdomain: brennan

---

📋 TABLE OF CONTENTS

1. Immediate Actions
2. Cleanup Procedure
3. Prevention Strategy
4. Diagnostic Tools
5. Safety Checks

---

🎯 IMMEDIATE ACTIONS

Step 1: Run Diagnostic Script

# Set database connection
export DATABASE_URL="postgresql://user:pass@host/db?sslmode=require"

# Run diagnostic
cd backend-saas
python3 scripts/diagnostic_prevent_test_data.py

**What it does:**

• Detects tenants with test_tenant=true flag
• Identifies suspicious name patterns (test, demo, sample)
• Flags suspicious email domains (test.com, example.com)
• Detects bulk creation patterns
• Finds abandoned/unused tenants
• Provides cleanup recommendations

**Output:** JSON report saved to /tmp/production_diagnostic_*.json

---

🧹 CLEANUP PROCEDURE

Step 2: Execute Cleanup Script

⚠️ **CRITICAL:** Read the safety guide before running!

# Set environment variables
export DATABASE_URL="postgresql://user:pass@host/db?sslmode=require"
export REDIS_URL="rediss://quiet-monkey-21389.upstash.io:6379"
export KEEP_TENANT_ID="31c06fc4-db22-4740-83ea-48ac14f25810"
export CONFIRM_CLEANUP="YES_I_UNDERSTAND_THIS_WILL_DELETE_ALL_TENANTS_EXCEPT_KEEP_TENANT"

# Run cleanup
cd backend-saas
python3 scripts/cleanup_all_tenants_except_brennan.py

**What it deletes (in order):**

1. **Level 1: Dependent Data**

• Episodes, feedback, skills, capabilities
• Agent usage records, metrics
• Conversations, messages, sessions
• Canvas audits, snapshots, skills
• Integration connections, webhooks, credentials
• Marketplace submissions, reviews
• Usage records, quotas
• Proposals, supervisor assignments
• Workspace invitations, members
• Tenant settings

1. **Level 2: Core Entities**

• Agents
• Workspaces

1. **Level 3: Tenant Records**

• Tenants (tenant records themselves)

1. **Redis Cleanup**

• All tenant-specific keys cleared

**Safety Features:**

✅ Uses environment variables (no hardcoded IDs)

✅ Creates backup before deletion

✅ Respects foreign key constraints

✅ Batch commits (every 10 tenants)

✅ Detailed logging

✅ Verification after completion

**Estimated Runtime:** 15-30 minutes

---

🛡️ PREVENTION STRATEGY

Step 3: Deploy Prevention Service

**New File:** backend-saas/core/test_data_prevention_service.py

**Features:**

• Blocks tenant creation with test keywords (test, demo, sample)
• Blocks suspicious email domains (test.com, example.com)
• Detects bulk creation patterns (3+ tenants/hour from same email)
• Logs all blocked requests for monitoring
• Environment-aware (only enforces in production)

**Integration Example:**

from core.test_data_prevention_service import TestDataPreventionService, get_test_data_prevention_service

@router.post("/tenants")
async def create_tenant(
    request: Request,
    tenant_data: TenantCreate,
    prevention: TestDataPreventionService = Depends(get_test_data_prevention_service)
):
    # Check for test data patterns
    is_suspicious, reason = prevention.check_tenant_creation(
        name=tenant_data.name,
        subdomain=tenant_data.subdomain,
        email=tenant_data.email
    )

    if is_suspicious:
        # Log the attempt
        prevention.log_suspicious_request(
            endpoint="/tenants",
            data=tenant_data.dict(),
            reason=reason,
            ip_address=request.client.host
        )

        # Block the request
        raise HTTPException(
            status_code=400,
            detail=f"Suspicious request detected: {reason}"
        )

    # Proceed with normal tenant creation
    ...

**Database Migration:**

# Apply migration for prevention logs table
cd backend-saas
alembic upgrade head

**Migration:** 20260409_141941_8b8ea176d100.py

• Creates test_data_prevention_logs table
• Tracks all blocked test data attempts
• Useful for monitoring and analysis

---

🔍 DIAGNOSTIC TOOLS

Script 1: Production Diagnostic

**Location:** backend-saas/scripts/diagnostic_prevent_test_data.py

**Checks:**

1. Test tenant flag (test_tenant=true)
2. Suspicious name patterns
3. Suspicious email addresses
4. Bulk creation patterns
5. Inactive/abandoned tenants
6. API abuse patterns
7. Data quality issues

**Usage:**

export DATABASE_URL="postgresql://user:pass@host/db"
python3 backend-saas/scripts/diagnostic_prevent_test_data.py

Script 2: Quick Tenant Count

**Quick check of database state:**

-- Total tenants
SELECT COUNT(*) FROM tenants;

-- Test tenants
SELECT COUNT(*) FROM tenants WHERE test_tenant = true;

-- Suspicious names
SELECT COUNT(*) FROM tenants
WHERE LOWER(name) LIKE ANY('%test%', '%demo%', '%sample%');

-- Empty tenants (no agents)
SELECT COUNT(*) FROM tenants t
LEFT JOIN agents a ON a.tenant_id = t.id
WHERE a.id IS NULL;

Script 3: Verify Brennan Tenant

-- Should return 1
SELECT COUNT(*) FROM tenants WHERE id = '31c06fc4-db22-4740-83ea-48ac14f25810';

-- Should show Brennan Machinery
SELECT name, subdomain, created_at FROM tenants
WHERE id = '31c06fc4-db22-4740-83ea-48ac14f25810';

---

✅ SAFETY CHECKS

Pre-Cleanup Checklist

• [ ] Backup solution is available
• [ ] Database admin is aware
• [ ] You have authorization to proceed
• [ ] This is production but all test data
• [ ] No real customers will be affected
• [ ] You understand this is PERMANENT deletion
• [ ] DATABASE_URL environment variable is set correctly
• [ ] REDIS_URL environment variable is set correctly
• [ ] KEEP_TENANT_ID environment variable is set to Brennan's ID
• [ ] CONFIRM_CLEANUP environment variable is set to confirmation phrase

Post-Cleanup Verification

-- Should return 1
SELECT COUNT(*) FROM tenants;

-- Should return Brennan Machinery
SELECT name FROM tenants;

-- Should return Brennan's ID
SELECT id FROM tenants WHERE subdomain = 'brennan';

-- Should return 0 (no test tenants)
SELECT COUNT(*) FROM tenants WHERE test_tenant = true;

Rollback Procedure

If something goes wrong:

1. **Stop the script immediately**

1. **Restore from backup**

• Backup location: /tmp/brennan_tenant_backup_*.sql
• Use database admin tools for restore

1. **Contact database admin**

• For point-in-time recovery
• NeonDB supports PITR

---

📊 EXPECTED RESULTS

Before Cleanup

• Tenants: 1,829
• Agents: 3,658-18,290
• Test tenants: ~1,800+
• Redis keys: Thousands

After Cleanup

• Tenants: 1 (Brennan Machinery only)
• Agents: (Brennan's agents only)
• Test tenants: 0
• Redis keys: (Brennan's keys only)

---

🚀 NEXT STEPS

1. **Run diagnostic** to confirm test data extent
2. **Review cleanup guide** at backend-saas/scripts/CLEANUP_GUIDE.md
3. **Execute cleanup script** with proper environment variables
4. **Verify results** with post-cleanup SQL queries
5. **Deploy prevention service** to block future test data
6. **Monitor logs** for any blocked test data attempts

---

📞 SUPPORT

If you encounter issues:

1. Check script output for error messages
2. Review backup file location
3. Verify database connectivity
4. Check Redis connection
5. Review foreign key constraints if deletion fails

**Script Locations:**

• Cleanup: backend-saas/scripts/cleanup_all_tenants_except_brennan.py
• Diagnostic: backend-saas/scripts/diagnostic_prevent_test_data.py
• Prevention Service: backend-saas/core/test_data_prevention_service.py
• Guide: backend-saas/scripts/CLEANUP_GUIDE.md

---

⚠️ CRITICAL REMINDERS

1. **This operation is irreversible** - Proceed with caution
2. **Always use environment variables** - Never hardcode tenant IDs
3. **Test in development first** - Before running in production
4. **Have backup ready** - In case something goes wrong
5. **Monitor the process** - Don't leave it unattended
6. **Verify after completion** - Ensure only Brennan tenant remains

---

**Generated:** 2026-04-09

**Status:** Ready for execution

**Priority:** CRITICAL