ATOM Documentation

โ† Back to App

Production Cleanup - Next Steps

**Date:** 2026-04-09

**Status:** Database cleanup complete โœ… | Redis cleanup pending | Prevention deployment pending

---

๐Ÿ“Š Cleanup Results

Database (PostgreSQL)

  • โœ… **COMPLETE** - Deleted 1,845 test tenants
  • โœ… **Result:** Only 3 tenants remain (Brennan + 2 system tenants)
  • โœ… **Data reduction:** 99.8%

Redis (Upstash)

  • โš ๏ธ **PENDING** - Needs manual cleanup
  • Estimated keys to delete: Thousands of test tenant keys
  • Action required: Run cleanup script with authentication

---

๐Ÿงน STEP 1: Clear Redis Data

# Get Redis URL from secrets
fly ssh console -c atom-saas

# In the console, run:
cd app
python3 -c "
import os
import redis

# Get Redis URL from environment
redis_url = os.getenv('UPSTASH_REDIS_URL')
print(f'Connecting to Redis...')

# Connect
r = redis.from_url(redis_url)

# Get all keys
all_keys = []
cursor = '0'
while cursor != 0:
    cursor, keys = r.scan(cursor=cursor, count=1000)
    all_keys.extend(keys)

print(f'Total keys: {len(all_keys)}')

# Delete all keys except Brennan's
brennan_id = '31c06fc4-db22-4740-83ea-48ac14f25810'
keys_to_delete = [k for k in all_keys if brennan_id not in str(k)]

print(f'Deleting {len(keys_to_delete)} keys...')

# Delete in batches
for i in range(0, len(keys_to_delete), 100):
    batch = keys_to_delete[i:i+100]
    r.delete(*batch)
    print(f'Deleted {min(i+100, len(keys_to_delete))}/{len(keys_to_delete)} keys')

# Verify
remaining = r.dbsize()
print(f'Remaining keys: {remaining}')
"

Option B: Flush All (Nuclear Option - CAUTION)

โš ๏ธ **This will delete ALL Redis keys including Brennan's**

fly ssh console -c atom-saas
redis-cli -u $UPSTASH_REDIS_URL FLUSHALL

Option C: Use Upstash Console

  1. Go to https://upstash.com/console
  2. Select your Redis database
  3. Use the key browser to delete keys not matching Brennan's tenant ID
  4. Or use "Flush Database" if you want to clear everything

---

๐Ÿ›ก๏ธ STEP 2: Deploy Prevention Service

2.1 Apply Database Migration

cd backend-saas
alembic upgrade head

This creates the test_data_prevention_logs table.

2.2 Add Prevention to Signup Flow

**File:** backend-saas/api/routes/tenants.py (or equivalent)

from core.test_data_prevention_service import TestDataPreventionService, get_test_data_prevention_service

@router.post("/tenants")
async def create_tenant(
    request: Request,
    tenant_data: TenantCreate,
    prevention: TestDataPreventionService = Depends(get_test_data_prevention_service),
    db: Session = Depends(get_db)
):
    # Check for test data patterns
    is_suspicious, reason = prevention.check_tenant_creation(
        name=tenant_data.name,
        subdomain=tenant_data.subdomain,
        email=tenant_data.email
    )

    if is_suspicious:
        # Log the attempt
        prevention.log_suspicious_request(
            endpoint="/tenants",
            data=tenant_data.dict(),
            reason=reason,
            ip_address=request.client.host
        )

        # Block the request in production
        raise HTTPException(
            status_code=400,
            detail=f"Suspicious request detected. Please use a real business name and email."
        )

    # Proceed with normal tenant creation
    # ... existing code ...

2.3 Test Prevention Locally

# Should be blocked
curl -X POST http://localhost:8000/api/tenants \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Test Company",
    "subdomain": "test-company-123",
    "email": "test@example.com"
  }'

# Should be allowed
curl -X POST http://localhost:8000/api/tenants \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Acme Corporation",
    "subdomain": "acme-corp",
    "email": "contact@acme.com"
  }'

2.4 Deploy to Production

# Commit changes
git add .
git commit -m "feat: add test data prevention service"
git push origin main

# Deploy
fly deploy -a atom-saas

---

๐Ÿ“Š STEP 3: Monitor Production

3.1 Check Tenant Count

-- Should return 3
SELECT COUNT(*) FROM tenants;

-- Should list only Brennan + system tenants
SELECT name, subdomain FROM tenants ORDER BY created_at;

3.2 Monitor Prevention Logs

-- Check blocked test data attempts
SELECT
    endpoint,
    reason,
    ip_address,
    created_at
FROM test_data_prevention_logs
ORDER BY created_at DESC
LIMIT 20;

3.3 Set Up Alerts

**Metrics to monitor:**

  • Tenant creation rate (should be < 10/hour normally)
  • Failed tenant creation attempts
  • New tenants with test keywords in logs

**Alert conditions:**

  • More than 5 tenants created in 1 hour
  • More than 10 failed tenant creations in 1 hour
  • Any tenant with "test" in name/subdomain

---

๐Ÿ”’ STEP 4: Add Additional Safeguards

4.1 Rate Limiting

**File:** backend-saas/core/abuse_protection_service.py

# Add tenant creation rate limit
MAX_TENANTS_PER_HOUR_PER_IP = 3
MAX_TENANTS_PER_HOUR_PER_EMAIL = 3

4.2 Email Domain Validation

**File:** backend-saas/core/test_data_prevention_service.py

# Add to SUSPICIOUS_PATTERNS
'suspicious_email_domains': [
    'test.com', 'example.com', 'fake.com',
    'temp.com', 'throwaway.email', 'guerrillamail.com',
    'mailinator.com', '10minutemail.com'
]

4.3 CAPTCHA on Signup

**Recommended:** Add hCaptcha or reCAPTCHA to signup form

**Frontend:**

// Add to signup form
import { HCaptcha } from '@hcaptcha/react-hcaptcha';

<HCaptcha
  sitekey="your-site-key"
  onVerify={handleCaptchaVerify}
/>

**Backend:**

# Verify CAPTCHA token
import httpx

async def verify_captcha(token: str):
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://hcaptcha.com/siteverify",
            data={
                "response": token,
                "secret": os.getenv("HCAPTCHA_SECRET")
            }
        )
        return response.json()["success"]

---

๐Ÿ“ STEP 5: Documentation

5.1 Update Operations Playbook

Document the cleanup process for future reference:

# Test Data Cleanup Procedure

## Detection
Run diagnostic: `python3 scripts/diagnostic_prevent_test_data.py`

## Cleanup
1. Backup Brennan tenant
2. Delete test tenants
3. Clear Redis keys
4. Verify only Brennan remains

## Prevention
- Test data prevention service active
- Rate limiting enabled
- CAPTCHA on signup
- Monitoring alerts configured

5.2 Update Developer Guidelines

# Production Data Guidelines

## DO NOT
- Create test tenants in production
- Use test@example.com emails
- Use "test", "demo", "sample" in tenant names
- Create automated tenants without approval

## DO
- Use production-like test data (e.g., "Acme Test Corp")
- Use real email domains
- Run E2E tests against /api/test/* endpoints only
- Use test environment for actual testing

---

โœ… Verification Checklist

Before considering this complete:

  • [ ] Redis data cleared (only Brennan keys remain)
  • [ ] Prevention service deployed to production
  • [ ] Database migration applied
  • [ ] Signup flow updated with validation
  • [ ] Rate limiting configured
  • [ ] CAPTCHA added to signup form
  • [ ] Monitoring alerts configured
  • [ ] Documentation updated
  • [ ] Team notified of new procedures

---

๐ŸŽฏ Success Metrics

**Before Cleanup:**

  • Tenants: 1,829 (99.9% test data)
  • Agents: 3,658-18,290
  • Redis keys: Thousands

**After Cleanup:**

  • Tenants: 3 (Brennan + 2 system)
  • Agents: Brennan's only
  • Redis keys: Brennan's only

**Prevention:**

  • Test data blocks: Active
  • Rate limiting: Active
  • Monitoring: Active

---

**Generated:** 2026-04-09

**Status:** Database cleanup complete, proceeding with Redis cleanup and prevention deployment