SES SNS Bounce Handling
This commit is contained in:
James Pattinson
255
BOUNCE_HANDLING_SETUP.md
Normal file
255
BOUNCE_HANDLING_SETUP.md
Normal file
@@ -0,0 +1,255 @@
|
||||
# SES SNS Bounce Handling Setup
|
||||
|
||||
This document describes how to configure AWS SES and SNS to handle email bounces automatically in the Mail List Manager.
|
||||
|
||||
## Overview
|
||||
|
||||
The system uses AWS Simple Notification Service (SNS) to receive real-time bounce notifications from AWS Simple Email Service (SES). When an email bounces:
|
||||
|
||||
1. SES sends a notification to an SNS topic
|
||||
2. SNS forwards the notification to your webhook endpoint
|
||||
3. The API processes the notification and updates the database
|
||||
4. Members with hard bounces are automatically deactivated
|
||||
5. Bounce history is tracked and displayed in the UI
|
||||
|
||||
## Bounce Status Types
|
||||
|
||||
- **Clean**: No bounces recorded
|
||||
- **Soft Bounce**: Temporary delivery issues (e.g., mailbox full, temporary server issues)
|
||||
- After 3 soft bounces, the member is marked with soft bounce status
|
||||
- **Hard Bounce**: Permanent delivery failure (e.g., invalid email address, domain doesn't exist)
|
||||
- Member is automatically deactivated and cannot receive emails
|
||||
|
||||
## Setup Instructions
|
||||
|
||||
### 1. Prerequisites
|
||||
|
||||
- AWS account with SES configured and verified
|
||||
- Your Mail List Manager deployed and accessible via HTTPS (required for SNS webhook)
|
||||
- Domain or subdomain for webhook (e.g., `https://lists.yourdomain.com`)
|
||||
|
||||
### 2. Create SNS Topic
|
||||
|
||||
1. Log in to AWS Console and navigate to SNS
|
||||
2. Click "Create topic"
|
||||
3. Choose "Standard" topic type
|
||||
4. Name: `ses-bounce-notifications` (or your preferred name)
|
||||
5. Display name: `SES Bounce Notifications`
|
||||
6. Click "Create topic"
|
||||
7. **Save the Topic ARN** (you'll need it in step 4)
|
||||
|
||||
### 3. Subscribe Your Webhook to SNS Topic
|
||||
|
||||
1. In the SNS topic details, click "Create subscription"
|
||||
2. Protocol: `HTTPS`
|
||||
3. Endpoint: `https://yourdomain.com:8000/webhooks/sns`
|
||||
- Replace `yourdomain.com` with your actual domain
|
||||
- The API must be accessible via HTTPS (SNS doesn't support HTTP)
|
||||
4. Enable raw message delivery: **Unchecked**
|
||||
5. Click "Create subscription"
|
||||
6. The subscription will be in "PendingConfirmation" status
|
||||
|
||||
### 4. Confirm SNS Subscription
|
||||
|
||||
When you create the subscription, SNS will send a `SubscriptionConfirmation` request to your webhook endpoint. The Mail List Manager API automatically confirms this subscription.
|
||||
|
||||
1. Check your API logs:
|
||||
```bash
|
||||
sudo docker-compose logs -f api
|
||||
```
|
||||
2. You should see a log entry indicating the subscription was confirmed
|
||||
3. In the AWS SNS console, refresh the subscriptions list
|
||||
4. The status should change from "PendingConfirmation" to "Confirmed"
|
||||
|
||||
### 5. Configure SES to Send Bounce Notifications
|
||||
|
||||
1. Navigate to AWS SES console
|
||||
2. Go to "Configuration Sets" (or "Verified identities" > select your domain > "Notifications")
|
||||
3. For configuration sets:
|
||||
- Create a new configuration set or select existing
|
||||
- Add "Event destination"
|
||||
- Event types: Select **"Bounce"** (and optionally "Complaint")
|
||||
- Destination: SNS topic
|
||||
- Select your SNS topic created in step 2
|
||||
4. For verified identities:
|
||||
- Select your sending domain/email
|
||||
- Click "Edit" in the "Notifications" section
|
||||
- Bounce feedback: Select your SNS topic
|
||||
- Include original headers: Enabled (optional)
|
||||
- Click "Save changes"
|
||||
|
||||
### 6. Verify Setup
|
||||
|
||||
#### Test with a Bounce Simulator
|
||||
|
||||
AWS SES provides bounce simulator addresses:
|
||||
|
||||
```bash
|
||||
# From inside Postfix container
|
||||
docker-compose exec postfix bash
|
||||
echo "Test bounce" | mail -s "Test" bounce@simulator.amazonses.com
|
||||
```
|
||||
|
||||
Or send to your mailing list with a test recipient:
|
||||
|
||||
1. Add `bounce@simulator.amazonses.com` as a member
|
||||
2. Subscribe to a test list
|
||||
3. Send an email to the list
|
||||
|
||||
#### Check the Results
|
||||
|
||||
1. Wait a few minutes for SES to process and send the notification
|
||||
2. Check API logs:
|
||||
```bash
|
||||
sudo docker-compose logs api | grep -i bounce
|
||||
```
|
||||
3. Log in to the web UI
|
||||
4. Go to Members tab
|
||||
5. Find the test member and click the "Bounces" button
|
||||
6. You should see the bounce event recorded
|
||||
|
||||
### 7. Security Considerations
|
||||
|
||||
#### SNS Signature Verification
|
||||
|
||||
The webhook endpoint automatically verifies SNS message signatures to ensure notifications are genuine AWS messages. This prevents unauthorized parties from sending fake bounce notifications.
|
||||
|
||||
#### HTTPS Requirement
|
||||
|
||||
SNS requires HTTPS for webhooks. You'll need:
|
||||
- Valid SSL/TLS certificate for your domain
|
||||
- Reverse proxy (e.g., Nginx, Apache) in front of the API container
|
||||
- Or use AWS API Gateway as a proxy
|
||||
|
||||
#### Example Nginx Configuration
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name lists.yourdomain.com;
|
||||
|
||||
ssl_certificate /path/to/cert.pem;
|
||||
ssl_certificate_key /path/to/key.pem;
|
||||
|
||||
# Webhook endpoint
|
||||
location /webhooks/sns {
|
||||
proxy_pass http://localhost:8000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
|
||||
# Optional: proxy API for web UI
|
||||
location /api {
|
||||
proxy_pass http://localhost:8000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 8. Managing Bounces in the UI
|
||||
|
||||
#### View Bounce Status
|
||||
|
||||
In the Members tab, bounced emails are indicated with:
|
||||
- Warning badge showing bounce count
|
||||
- Color-coded status (yellow for soft bounce, red for hard bounce)
|
||||
- Last bounce timestamp
|
||||
|
||||
#### View Bounce History
|
||||
|
||||
1. Click the "Bounces" button next to a member
|
||||
2. View detailed bounce history including:
|
||||
- Bounce type (Permanent, Transient, Undetermined)
|
||||
- Bounce subtype
|
||||
- Diagnostic code from the receiving mail server
|
||||
- Timestamp of each bounce
|
||||
|
||||
#### Reset Bounce Status
|
||||
|
||||
If a member's email has been corrected or verified:
|
||||
|
||||
1. Open the bounce history modal
|
||||
2. Click "Reset Bounce Status"
|
||||
3. Confirm the action
|
||||
4. The member's bounce count is cleared and they can receive emails again
|
||||
|
||||
**Note**: Only users with write access (administrators and operators) can reset bounce status.
|
||||
|
||||
### 9. Monitoring and Maintenance
|
||||
|
||||
#### Check Bounce Logs
|
||||
|
||||
```bash
|
||||
# View all bounces in database
|
||||
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT * FROM bounce_logs ORDER BY timestamp DESC LIMIT 20;"
|
||||
|
||||
# Count bounces by type
|
||||
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT bounce_type, COUNT(*) as count FROM bounce_logs GROUP BY bounce_type;"
|
||||
|
||||
# Find members with bounces
|
||||
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT name, email, bounce_count, bounce_status FROM members WHERE bounce_count > 0;"
|
||||
```
|
||||
|
||||
#### API Health Check
|
||||
|
||||
```bash
|
||||
# Check if webhook is accessible
|
||||
curl -X POST https://yourdomain.com:8000/webhooks/sns \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"Type":"test"}'
|
||||
```
|
||||
|
||||
#### Clean Up Old Bounce Records
|
||||
|
||||
Periodically review and clean up old bounce records:
|
||||
|
||||
```sql
|
||||
-- Delete bounce logs older than 90 days
|
||||
DELETE FROM bounce_logs WHERE created_at < DATE_SUB(NOW(), INTERVAL 90 DAY);
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### SNS Subscription Not Confirming
|
||||
|
||||
- Ensure the API container is running and accessible via HTTPS
|
||||
- Check API logs for errors
|
||||
- Verify firewall rules allow HTTPS traffic to port 8000
|
||||
- Test the endpoint manually: `curl https://yourdomain.com:8000/health`
|
||||
|
||||
### Bounces Not Being Recorded
|
||||
|
||||
1. Verify SNS topic is receiving messages:
|
||||
- Check SNS topic metrics in AWS Console
|
||||
2. Verify subscription is active:
|
||||
- Check subscription status in SNS console
|
||||
3. Check API logs for webhook errors:
|
||||
```bash
|
||||
sudo docker-compose logs api | grep -i "sns\|bounce"
|
||||
```
|
||||
4. Test signature verification:
|
||||
- Temporarily add debug logging to the webhook endpoint
|
||||
|
||||
### Members Not Being Deactivated
|
||||
|
||||
- Check if bounce type is "Permanent"
|
||||
- Review member's bounce_status in database:
|
||||
```bash
|
||||
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT * FROM members WHERE email='problem@example.com';"
|
||||
```
|
||||
- Verify bounce processing logic in API logs
|
||||
|
||||
### SSL Certificate Issues
|
||||
|
||||
If using self-signed certificates, SNS will reject the webhook. You must use:
|
||||
- Valid certificate from a trusted CA (Let's Encrypt, etc.)
|
||||
- Or use AWS Certificate Manager with API Gateway
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [AWS SES Bounce Handling](https://docs.aws.amazon.com/ses/latest/dg/event-publishing-retrieving-sns.html)
|
||||
- [AWS SNS HTTPS Subscriptions](https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html)
|
||||
- [SES Bounce Types](https://docs.aws.amazon.com/ses/latest/dg/notification-contents.html#bounce-types)
|
||||
Reference in New Issue
Block a user