sasa-maillist/SES_BOUNCE_TESTING_GUIDE.md

# Testing Bounces from SES Sandbox

AWS SES provides **built-in bounce simulator addresses** that work even in sandbox mode. This guide shows you how to use them to test your bounce handling.

## Quick Answer

Send email to these special AWS addresses to simulate different bounce types:

### Hard Bounce (Permanent - Invalid Address)
```bash
bounce@simulator.amazonses.com
```

### Soft Bounce (Temporary - Mailbox Full)
```bash
ooto@simulator.amazonses.com
```

### Complaint (Spam Report)
```bash
complaint@simulator.amazonses.com
```

### Successful Delivery (No Bounce)
```bash
success@simulator.amazonses.com
```

## Step-by-Step Testing Guide

### Option 1: Using Your Mailing Lists (Recommended)

This tests the complete flow: Postfix → SES → SNS → API

1. **Add the simulator address as a member:**
   ```bash
   # Using the API
   curl -X POST http://localhost:8000/members \
     -H "Authorization: Bearer $API_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
       "name": "Bounce Test",
       "email": "bounce@simulator.amazonses.com",
       "active": true
     }'
   ```

   Or use the Web UI:
   - Go to http://localhost:3000
   - Members tab → Add Member
   - Name: `Bounce Test`
   - Email: `bounce@simulator.amazonses.com`

2. **Subscribe to a test list:**
   - Click "Subscriptions" button for the test member
   - Toggle on one of your mailing lists

3. **Send email to the list:**
   ```bash
   # From inside Postfix container
   sudo docker compose exec postfix bash
   echo "This will bounce" | mail -s "Test Bounce" community@lists.sasalliance.org
   exit
   ```

   Replace `community@lists.sasalliance.org` with your actual list email.

4. **Wait 30-60 seconds** for:
   - Email to be sent via SES
   - SES to process the bounce
   - SNS to send notification to your webhook

5. **Check the results:**

   **Watch API logs in real-time:**
   ```bash
   sudo docker compose logs api -f
   ```
   
   You should see:
   ```
   ============================================================
   SNS Webhook Request Received
   ============================================================
   Content-Type: text/plain; charset=UTF-8
   User-Agent: Amazon Simple Notification Service Agent
   
   ✓ Notification Received
     Notification Type: Bounce
   
   ✓ Processing Bounce
     Bounce Type: Permanent
     Recipients: ['bounce@simulator.amazonses.com']
     ✓ Bounce processed successfully
   ============================================================
   ```

   **Check the database:**
   ```bash
   # View bounce logs
   sudo docker compose exec mysql mysql -u maillist -pmaillist maillist \
     -e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;"
   
   # Check member status
   sudo docker compose exec mysql mysql -u maillist -pmaillist maillist \
     -e "SELECT email, active, bounce_count, bounce_status FROM members WHERE email='bounce@simulator.amazonses.com';"
   ```

   **View in Web UI:**
   - Open http://localhost:3000
   - Go to Members tab
   - Find "Bounce Test" member
   - Should show: ❌ Inactive (red), bounce badge, last bounce timestamp
   - Click "Bounces" button to see detailed history

### Option 2: Direct Email via Postfix (Simpler)

Send directly to the simulator without going through your list:

```bash
# Enter Postfix container
sudo docker compose exec postfix bash

# Send test email
echo "Testing hard bounce" | mail -s "Hard Bounce Test" bounce@simulator.amazonses.com

# Or soft bounce
echo "Testing soft bounce" | mail -s "Soft Bounce Test" ooto@simulator.amazonses.com

# Exit container
exit
```

### Option 3: Using AWS SES Console (For Non-Sandbox Testing)

If you have SES production access:

1. Go to AWS SES Console
2. Click "Send test email"
3. To: `bounce@simulator.amazonses.com`
4. From: Your verified email/domain
5. Subject: "Bounce test"
6. Body: "Testing bounce handling"
7. Click "Send test email"

## Testing Different Bounce Types

### 1. Hard Bounce (Permanent Failure)
```bash
echo "Test" | mail -s "Test" bounce@simulator.amazonses.com
```

**Expected Result:**
- Member marked as `hard_bounce`
- Member deactivated (`active = 0`)
- `bounce_count` incremented
- Entry in `bounce_logs` table

### 2. Soft Bounce (Transient Failure)
```bash
echo "Test" | mail -s "Test" ooto@simulator.amazonses.com
```

**Expected Result:**
- Member marked as `clean` (first time)
- After 3 soft bounces → `soft_bounce` status
- `bounce_count` incremented
- Member stays active

### 3. Complaint (Spam Report)
```bash
echo "Test" | mail -s "Test" complaint@simulator.amazonses.com
```

**Expected Result:**
- API receives complaint notification
- Currently logged but not processed (you can extend the handler)

## Monitoring the Test

### Real-Time Monitoring (Recommended)

Open 3 terminal windows:

**Terminal 1 - API Logs:**
```bash
sudo docker compose logs api -f
```

**Terminal 2 - Postfix Logs:**
```bash
sudo docker compose logs postfix -f
```

**Terminal 3 - Send Test Email:**
```bash
sudo docker compose exec postfix bash
echo "Test" | mail -s "Bounce Test" bounce@simulator.amazonses.com
```

### Timeline

Here's what happens and when:

- **T+0s**: Email sent to Postfix
- **T+1-3s**: Postfix relays to SES
- **T+5-10s**: SES processes and generates bounce
- **T+10-30s**: SNS sends notification to your webhook
- **T+30-60s**: API processes bounce and updates database

## Verifying the Complete Flow

### 1. Check Postfix Logs
```bash
sudo docker compose logs postfix | grep bounce@simulator
```

Should show:
```
postfix/smtp[xxx]: ... to=<bounce@simulator.amazonses.com>, relay=email-smtp.eu-west-2.amazonaws.com[...], ... status=sent
```

### 2. Check SNS Subscription Status
- Go to AWS SNS Console
- Find your topic
- Check "Subscriptions" tab
- Status should be "Confirmed"
- Messages delivered should be > 0

### 3. Check API Logs
```bash
sudo docker compose logs api | grep -A 20 "SNS Webhook"
```

Should show successful processing.

### 4. Check Database
```bash
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist <<EOF
-- Show recent bounces
SELECT 
    b.bounce_id,
    m.email,
    b.bounce_type,
    b.diagnostic_code,
    b.timestamp
FROM bounce_logs b
JOIN members m ON b.member_id = m.member_id
ORDER BY b.timestamp DESC
LIMIT 5;

-- Show members with bounces
SELECT 
    email,
    active,
    bounce_count,
    bounce_status,
    last_bounce_at
FROM members
WHERE bounce_count > 0;
EOF
```

## Troubleshooting

### "Email not sent" or "Relay access denied"

**Problem**: Postfix not configured to relay via SES

**Check**:
```bash
sudo docker compose exec postfix postconf relayhost
sudo docker compose exec postfix postconf smtp_sasl_auth_enable
```

Should show:
```
relayhost = [email-smtp.eu-west-2.amazonaws.com]:587
smtp_sasl_auth_enable = yes
```

### "No bounce received after 5 minutes"

**Possible causes**:

1. **SNS subscription not confirmed**
   - Check AWS SNS console
   - Status should be "Confirmed", not "Pending"

2. **SNS topic not configured in SES**
   - Check SES → Configuration Sets or Verified Identities → Notifications
   - Bounce notifications should point to your SNS topic

3. **Webhook endpoint not accessible**
   - SNS requires HTTPS
   - Test: `curl https://your-domain.com:8000/health`

4. **API container not running**
   ```bash
   sudo docker compose ps api
   ```

### "Bounce received but not in database"

**Check API logs for errors**:
```bash
sudo docker compose logs api | grep -i error
```

**Check database tables exist**:
```bash
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e "SHOW TABLES;"
```

Should include: `bounce_logs`, `members`

## Testing Multiple Bounces

To test the "3 soft bounces = soft_bounce status" logic:

```bash
sudo docker compose exec postfix bash

# Send 3 emails to soft bounce simulator
for i in {1..3}; do
    echo "Soft bounce test $i" | mail -s "Test $i" ooto@simulator.amazonses.com
    sleep 70  # Wait between sends for SNS processing
done
```

After the 3rd bounce:
- Member's `bounce_status` should change from `clean` to `soft_bounce`
- `bounce_count` should be 3

## Cleanup After Testing

Remove test bounce data:

```bash
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist <<EOF
-- Delete test bounce logs
DELETE FROM bounce_logs WHERE email IN ('bounce@simulator.amazonses.com', 'ooto@simulator.amazonses.com');

-- Reset test member
UPDATE members 
SET bounce_count = 0, 
    bounce_status = 'clean', 
    last_bounce_at = NULL,
    active = 1
WHERE email IN ('bounce@simulator.amazonses.com', 'ooto@simulator.amazonses.com');
EOF
```

Or use the Web UI:
- Go to Members tab
- Find the test member
- Click "Bounces" button
- Click "Reset Bounce Status"

## Alternative: Simulate Bounces Without SES

If SNS isn't set up yet, use the included script to simulate bounces directly in the database:

```bash
./simulate_bounce.sh
```

This is useful for:
- Testing the UI without AWS
- Development environments
- Demonstrating bounce handling to stakeholders

## Next Steps

Once bounce handling is working:

1. **Remove simulator addresses** from your member list
2. **Monitor real bounces** in production
3. **Set up alerts** for high bounce rates
4. **Review bounced members** regularly and update/remove invalid addresses
5. **Consider complaint handling** (similar to bounces, for spam reports)

## Summary Commands

**Quick test sequence**:
```bash
# 1. Watch logs
sudo docker compose logs api -f &

# 2. Send test bounce
echo "Test" | sudo docker compose exec -T postfix mail -s "Bounce Test" bounce@simulator.amazonses.com

# 3. Wait 60 seconds, then check database
sleep 60
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 1;"
```

That's it! The bounce simulator is the easiest way to test your bounce handling without needing real bounced emails.