sasa-maillist/BOUNCE_HANDLING_SETUP.md

# SES SNS Bounce Handling Setup

This document describes how to configure AWS SES and SNS to handle email bounces automatically in the Mail List Manager.

## Overview

The system uses AWS Simple Notification Service (SNS) to receive real-time bounce notifications from AWS Simple Email Service (SES). When an email bounces:

1. SES sends a notification to an SNS topic
2. SNS forwards the notification to your webhook endpoint
3. The API processes the notification and updates the database
4. Members with hard bounces are automatically deactivated
5. Bounce history is tracked and displayed in the UI

## Bounce Status Types

- **Clean**: No bounces recorded
- **Soft Bounce**: Temporary delivery issues (e.g., mailbox full, temporary server issues)
  - After 3 soft bounces, the member is marked with soft bounce status
- **Hard Bounce**: Permanent delivery failure (e.g., invalid email address, domain doesn't exist)
  - Member is automatically deactivated and cannot receive emails

## Setup Instructions

### 1. Prerequisites

- AWS account with SES configured and verified
- Your Mail List Manager deployed and accessible via HTTPS (required for SNS webhook)
- Domain or subdomain for webhook (e.g., `https://lists.yourdomain.com`)

### 2. Create SNS Topic

1. Log in to AWS Console and navigate to SNS
2. Click "Create topic"
3. Choose "Standard" topic type
4. Name: `ses-bounce-notifications` (or your preferred name)
5. Display name: `SES Bounce Notifications`
6. Click "Create topic"
7. **Save the Topic ARN** (you'll need it in step 4) arn:aws:sns:eu-west-2:827164363113:ses-bounces

### 3. Subscribe Your Webhook to SNS Topic

1. In the SNS topic details, click "Create subscription"
2. Protocol: `HTTPS`
3. Endpoint: `https://yourdomain.com:8000/webhooks/sns`
   - Replace `yourdomain.com` with your actual domain
   - The API must be accessible via HTTPS (SNS doesn't support HTTP)
4. Enable raw message delivery: **Unchecked**
5. Click "Create subscription"
6. The subscription will be in "PendingConfirmation" status

### 4. Confirm SNS Subscription

When you create the subscription, SNS will send a `SubscriptionConfirmation` request to your webhook endpoint. The Mail List Manager API automatically confirms this subscription.

1. Check your API logs:
   ```bash
   sudo docker-compose logs -f api
   ```
2. You should see a log entry indicating the subscription was confirmed
3. In the AWS SNS console, refresh the subscriptions list
4. The status should change from "PendingConfirmation" to "Confirmed"

### 5. Configure SES to Send Bounce Notifications

1. Navigate to AWS SES console
2. Go to "Configuration Sets" (or "Verified identities" > select your domain > "Notifications")
3. For configuration sets:
   - Create a new configuration set or select existing
   - Add "Event destination"
   - Event types: Select **"Bounce"** (and optionally "Complaint")
   - Destination: SNS topic
   - Select your SNS topic created in step 2
4. For verified identities:
   - Select your sending domain/email
   - Click "Edit" in the "Notifications" section
   - Bounce feedback: Select your SNS topic
   - Include original headers: Enabled (optional)
   - Click "Save changes"

### 6. Verify Setup

#### Test with a Bounce Simulator

AWS SES provides bounce simulator addresses:

```bash
# From inside Postfix container
docker-compose exec postfix bash
echo "Test bounce" | mail -s "Test" bounce@simulator.amazonses.com
```

Or send to your mailing list with a test recipient:

1. Add `bounce@simulator.amazonses.com` as a member
2. Subscribe to a test list
3. Send an email to the list

#### Check the Results

1. Wait a few minutes for SES to process and send the notification
2. Check API logs:
   ```bash
   sudo docker-compose logs api | grep -i bounce
   ```
3. Log in to the web UI
4. Go to Members tab
5. Find the test member and click the "Bounces" button
6. You should see the bounce event recorded

### 7. Security Considerations

#### SNS Signature Verification

The webhook endpoint automatically verifies SNS message signatures to ensure notifications are genuine AWS messages. This prevents unauthorized parties from sending fake bounce notifications.

#### HTTPS Requirement

SNS requires HTTPS for webhooks. You'll need:
- Valid SSL/TLS certificate for your domain
- Reverse proxy (e.g., Nginx, Apache) in front of the API container
- Or use AWS API Gateway as a proxy

#### Example Nginx Configuration

```nginx
server {
    listen 443 ssl http2;
    server_name lists.yourdomain.com;
    
    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;
    
    # Webhook endpoint
    location /webhooks/sns {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
    
    # Optional: proxy API for web UI
    location /api {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}
```

### 8. Managing Bounces in the UI

#### View Bounce Status

In the Members tab, bounced emails are indicated with:
- Warning badge showing bounce count
- Color-coded status (yellow for soft bounce, red for hard bounce)
- Last bounce timestamp

#### View Bounce History

1. Click the "Bounces" button next to a member
2. View detailed bounce history including:
   - Bounce type (Permanent, Transient, Undetermined)
   - Bounce subtype
   - Diagnostic code from the receiving mail server
   - Timestamp of each bounce

#### Reset Bounce Status

If a member's email has been corrected or verified:

1. Open the bounce history modal
2. Click "Reset Bounce Status"
3. Confirm the action
4. The member's bounce count is cleared and they can receive emails again

**Note**: Only users with write access (administrators and operators) can reset bounce status.

### 9. Monitoring and Maintenance

#### Check Bounce Logs

```bash
# View all bounces in database
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT * FROM bounce_logs ORDER BY timestamp DESC LIMIT 20;"

# Count bounces by type
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT bounce_type, COUNT(*) as count FROM bounce_logs GROUP BY bounce_type;"

# Find members with bounces
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT name, email, bounce_count, bounce_status FROM members WHERE bounce_count > 0;"
```

#### API Health Check

```bash
# Check if webhook is accessible
curl -X POST https://yourdomain.com:8000/webhooks/sns \
  -H "Content-Type: application/json" \
  -d '{"Type":"test"}'
```

#### Clean Up Old Bounce Records

Periodically review and clean up old bounce records:

```sql
-- Delete bounce logs older than 90 days
DELETE FROM bounce_logs WHERE created_at < DATE_SUB(NOW(), INTERVAL 90 DAY);
```

## Troubleshooting

### SNS Subscription Not Confirming

- Ensure the API container is running and accessible via HTTPS
- Check API logs for errors
- Verify firewall rules allow HTTPS traffic to port 8000
- Test the endpoint manually: `curl https://yourdomain.com:8000/health`

### Bounces Not Being Recorded

1. Verify SNS topic is receiving messages:
   - Check SNS topic metrics in AWS Console
2. Verify subscription is active:
   - Check subscription status in SNS console
3. Check API logs for webhook errors:
   ```bash
   sudo docker-compose logs api | grep -i "sns\|bounce"
   ```
4. Test signature verification:
   - Temporarily add debug logging to the webhook endpoint

### Members Not Being Deactivated

- Check if bounce type is "Permanent"
- Review member's bounce_status in database:
  ```bash
  sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT * FROM members WHERE email='problem@example.com';"
  ```
- Verify bounce processing logic in API logs

### SSL Certificate Issues

If using self-signed certificates, SNS will reject the webhook. You must use:
- Valid certificate from a trusted CA (Let's Encrypt, etc.)
- Or use AWS Certificate Manager with API Gateway

## Additional Resources

- [AWS SES Bounce Handling](https://docs.aws.amazon.com/ses/latest/dg/event-publishing-retrieving-sns.html)
- [AWS SNS HTTPS Subscriptions](https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html)
- [SES Bounce Types](https://docs.aws.amazon.com/ses/latest/dg/notification-contents.html#bounce-types)