Production Cleanup - Next Steps
**Date:** 2026-04-09
**Status:** Database cleanup complete โ | Redis cleanup pending | Prevention deployment pending
---
๐ Cleanup Results
Database (PostgreSQL)
- โ **COMPLETE** - Deleted 1,845 test tenants
- โ **Result:** Only 3 tenants remain (Brennan + 2 system tenants)
- โ **Data reduction:** 99.8%
Redis (Upstash)
- โ ๏ธ **PENDING** - Needs manual cleanup
- Estimated keys to delete: Thousands of test tenant keys
- Action required: Run cleanup script with authentication
---
๐งน STEP 1: Clear Redis Data
Option A: Run Cleanup Script (Recommended)
# Get Redis URL from secrets
fly ssh console -c atom-saas
# In the console, run:
cd app
python3 -c "
import os
import redis
# Get Redis URL from environment
redis_url = os.getenv('UPSTASH_REDIS_URL')
print(f'Connecting to Redis...')
# Connect
r = redis.from_url(redis_url)
# Get all keys
all_keys = []
cursor = '0'
while cursor != 0:
cursor, keys = r.scan(cursor=cursor, count=1000)
all_keys.extend(keys)
print(f'Total keys: {len(all_keys)}')
# Delete all keys except Brennan's
brennan_id = '31c06fc4-db22-4740-83ea-48ac14f25810'
keys_to_delete = [k for k in all_keys if brennan_id not in str(k)]
print(f'Deleting {len(keys_to_delete)} keys...')
# Delete in batches
for i in range(0, len(keys_to_delete), 100):
batch = keys_to_delete[i:i+100]
r.delete(*batch)
print(f'Deleted {min(i+100, len(keys_to_delete))}/{len(keys_to_delete)} keys')
# Verify
remaining = r.dbsize()
print(f'Remaining keys: {remaining}')
"Option B: Flush All (Nuclear Option - CAUTION)
โ ๏ธ **This will delete ALL Redis keys including Brennan's**
fly ssh console -c atom-saas
redis-cli -u $UPSTASH_REDIS_URL FLUSHALLOption C: Use Upstash Console
- Go to https://upstash.com/console
- Select your Redis database
- Use the key browser to delete keys not matching Brennan's tenant ID
- Or use "Flush Database" if you want to clear everything
---
๐ก๏ธ STEP 2: Deploy Prevention Service
2.1 Apply Database Migration
cd backend-saas
alembic upgrade headThis creates the test_data_prevention_logs table.
2.2 Add Prevention to Signup Flow
**File:** backend-saas/api/routes/tenants.py (or equivalent)
from core.test_data_prevention_service import TestDataPreventionService, get_test_data_prevention_service
@router.post("/tenants")
async def create_tenant(
request: Request,
tenant_data: TenantCreate,
prevention: TestDataPreventionService = Depends(get_test_data_prevention_service),
db: Session = Depends(get_db)
):
# Check for test data patterns
is_suspicious, reason = prevention.check_tenant_creation(
name=tenant_data.name,
subdomain=tenant_data.subdomain,
email=tenant_data.email
)
if is_suspicious:
# Log the attempt
prevention.log_suspicious_request(
endpoint="/tenants",
data=tenant_data.dict(),
reason=reason,
ip_address=request.client.host
)
# Block the request in production
raise HTTPException(
status_code=400,
detail=f"Suspicious request detected. Please use a real business name and email."
)
# Proceed with normal tenant creation
# ... existing code ...2.3 Test Prevention Locally
# Should be blocked
curl -X POST http://localhost:8000/api/tenants \
-H "Content-Type: application/json" \
-d '{
"name": "Test Company",
"subdomain": "test-company-123",
"email": "test@example.com"
}'
# Should be allowed
curl -X POST http://localhost:8000/api/tenants \
-H "Content-Type: application/json" \
-d '{
"name": "Acme Corporation",
"subdomain": "acme-corp",
"email": "contact@acme.com"
}'2.4 Deploy to Production
# Commit changes
git add .
git commit -m "feat: add test data prevention service"
git push origin main
# Deploy
fly deploy -a atom-saas---
๐ STEP 3: Monitor Production
3.1 Check Tenant Count
-- Should return 3
SELECT COUNT(*) FROM tenants;
-- Should list only Brennan + system tenants
SELECT name, subdomain FROM tenants ORDER BY created_at;3.2 Monitor Prevention Logs
-- Check blocked test data attempts
SELECT
endpoint,
reason,
ip_address,
created_at
FROM test_data_prevention_logs
ORDER BY created_at DESC
LIMIT 20;3.3 Set Up Alerts
**Metrics to monitor:**
- Tenant creation rate (should be < 10/hour normally)
- Failed tenant creation attempts
- New tenants with test keywords in logs
**Alert conditions:**
- More than 5 tenants created in 1 hour
- More than 10 failed tenant creations in 1 hour
- Any tenant with "test" in name/subdomain
---
๐ STEP 4: Add Additional Safeguards
4.1 Rate Limiting
**File:** backend-saas/core/abuse_protection_service.py
# Add tenant creation rate limit
MAX_TENANTS_PER_HOUR_PER_IP = 3
MAX_TENANTS_PER_HOUR_PER_EMAIL = 34.2 Email Domain Validation
**File:** backend-saas/core/test_data_prevention_service.py
# Add to SUSPICIOUS_PATTERNS
'suspicious_email_domains': [
'test.com', 'example.com', 'fake.com',
'temp.com', 'throwaway.email', 'guerrillamail.com',
'mailinator.com', '10minutemail.com'
]4.3 CAPTCHA on Signup
**Recommended:** Add hCaptcha or reCAPTCHA to signup form
**Frontend:**
// Add to signup form
import { HCaptcha } from '@hcaptcha/react-hcaptcha';
<HCaptcha
sitekey="your-site-key"
onVerify={handleCaptchaVerify}
/>**Backend:**
# Verify CAPTCHA token
import httpx
async def verify_captcha(token: str):
async with httpx.AsyncClient() as client:
response = await client.post(
"https://hcaptcha.com/siteverify",
data={
"response": token,
"secret": os.getenv("HCAPTCHA_SECRET")
}
)
return response.json()["success"]---
๐ STEP 5: Documentation
5.1 Update Operations Playbook
Document the cleanup process for future reference:
# Test Data Cleanup Procedure
## Detection
Run diagnostic: `python3 scripts/diagnostic_prevent_test_data.py`
## Cleanup
1. Backup Brennan tenant
2. Delete test tenants
3. Clear Redis keys
4. Verify only Brennan remains
## Prevention
- Test data prevention service active
- Rate limiting enabled
- CAPTCHA on signup
- Monitoring alerts configured5.2 Update Developer Guidelines
# Production Data Guidelines
## DO NOT
- Create test tenants in production
- Use test@example.com emails
- Use "test", "demo", "sample" in tenant names
- Create automated tenants without approval
## DO
- Use production-like test data (e.g., "Acme Test Corp")
- Use real email domains
- Run E2E tests against /api/test/* endpoints only
- Use test environment for actual testing---
โ Verification Checklist
Before considering this complete:
- [ ] Redis data cleared (only Brennan keys remain)
- [ ] Prevention service deployed to production
- [ ] Database migration applied
- [ ] Signup flow updated with validation
- [ ] Rate limiting configured
- [ ] CAPTCHA added to signup form
- [ ] Monitoring alerts configured
- [ ] Documentation updated
- [ ] Team notified of new procedures
---
๐ฏ Success Metrics
**Before Cleanup:**
- Tenants: 1,829 (99.9% test data)
- Agents: 3,658-18,290
- Redis keys: Thousands
**After Cleanup:**
- Tenants: 3 (Brennan + 2 system)
- Agents: Brennan's only
- Redis keys: Brennan's only
**Prevention:**
- Test data blocks: Active
- Rate limiting: Active
- Monitoring: Active
---
**Generated:** 2026-04-09
**Status:** Database cleanup complete, proceeding with Redis cleanup and prevention deployment