Email bounce handling with postfix
This commit is contained in:
James Pattinson
243
EMAIL_BOUNCE_HANDLING_SETUP.md
Normal file
243
EMAIL_BOUNCE_HANDLING_SETUP.md
Normal file
@@ -0,0 +1,243 @@
|
||||
# Email-Based Bounce Handling Setup
|
||||
|
||||
This document explains the email-based bounce handling system implemented as an alternative to SNS webhooks for environments without SES production access.
|
||||
|
||||
## Overview
|
||||
|
||||
The system processes email bounces directly within the Postfix container by:
|
||||
1. Rewriting return paths to direct bounces to a processing address
|
||||
2. Processing bounce emails via Python script
|
||||
3. Updating member bounce statistics in MySQL database
|
||||
4. Automatically disabling members with excessive bounces
|
||||
2. Setting up an alias that pipes bounce emails to a Python processing script
|
||||
3. The script parses bounce emails, extracts bounced addresses, and updates the database
|
||||
4. Members with hard bounces are automatically deactivated
|
||||
5. Bounce history is tracked and displayed in the UI (same as SNS method)
|
||||
|
||||
## Advantages
|
||||
|
||||
- **Works with SES Sandbox**: No production SES access required
|
||||
- **No External Dependencies**: Doesn't require SNS, webhooks, or HTTPS domains
|
||||
- **Self-Contained**: All processing happens within the existing containers
|
||||
- **Real-time Processing**: Bounces are processed as soon as emails arrive
|
||||
- **Compatible**: Uses the same database schema and UI as SNS bounce handling
|
||||
|
||||
## Configuration
|
||||
|
||||
### 1. Enable Email Bounce Processing
|
||||
|
||||
In your `.env` file:
|
||||
|
||||
```bash
|
||||
# Enable email-based bounce processing
|
||||
ENABLE_EMAIL_BOUNCE_PROCESSING=true
|
||||
|
||||
# This will automatically enable bounce handling features
|
||||
ENABLE_BOUNCE_HANDLING=true # Automatically set to true when email processing is enabled
|
||||
```
|
||||
|
||||
### 2. Restart the System
|
||||
|
||||
```bash
|
||||
sudo docker-compose down
|
||||
sudo docker-compose up --build -d
|
||||
```
|
||||
|
||||
### 3. Verify Configuration
|
||||
|
||||
Check that email bounce processing is enabled:
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:8000/config | jq .
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```json
|
||||
{
|
||||
"bounce_handling_enabled": true,
|
||||
"sns_webhooks_enabled": false,
|
||||
"email_bounce_processing_enabled": true
|
||||
}
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
### Postfix Configuration
|
||||
|
||||
The system configures Postfix with:
|
||||
|
||||
- `bounce_notice_recipient = bounces@lists.sasalliance.org`
|
||||
- `2bounce_notice_recipient = bounces@lists.sasalliance.org`
|
||||
- `error_notice_recipient = bounces@lists.sasalliance.org`
|
||||
|
||||
### Aliases Configuration
|
||||
|
||||
The `bounces` address is configured to pipe emails to the processing script:
|
||||
|
||||
```
|
||||
bounces: "|/usr/local/bin/process-bounce.py"
|
||||
```
|
||||
|
||||
### Bounce Processing Script
|
||||
|
||||
The Python script (`/usr/local/bin/process-bounce.py`):
|
||||
|
||||
1. **Reads bounce emails** from stdin (via pipe)
|
||||
2. **Parses email content** using multiple regex patterns to extract bounced addresses
|
||||
3. **Analyzes bounce type** based on SMTP error codes:
|
||||
- 5xx codes = Permanent bounces
|
||||
- 4xx codes = Transient bounces
|
||||
- Unknown = Undetermined
|
||||
4. **Updates database** using the same schema as SNS bounce handling:
|
||||
- Logs bounce in `bounce_logs` table
|
||||
- Updates member `bounce_count`, `bounce_status`, and `last_bounce_at`
|
||||
- Deactivates members with permanent bounces
|
||||
- Marks members with soft bounce status after 3 transient bounces
|
||||
|
||||
## Testing
|
||||
|
||||
### Test the Processing Script
|
||||
|
||||
You can test the bounce processing script in test mode:
|
||||
|
||||
```bash
|
||||
# Test with sample bounce email
|
||||
sudo docker-compose exec postfix /usr/local/bin/process-bounce.py --test
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
2025-10-14 15:49:16,041 - bounce-processor - INFO - Starting bounce processing
|
||||
2025-10-14 15:49:16,041 - bounce-processor - INFO - Running in test mode with sample bounce email
|
||||
2025-10-14 15:49:16,050 - bounce-processor - INFO - Extracted addresses: ['testuser@example.com']
|
||||
2025-10-14 15:49:16,050 - bounce-processor - INFO - Test mode - would process 1 bounce(s):
|
||||
2025-10-14 15:49:16,050 - bounce-processor - INFO - {'email': 'testuser@example.com', 'bounce_type': 'Permanent', 'bounce_subtype': 'General', 'diagnostic_code': '', 'timestamp': '2025-10-14 15:49:16'}
|
||||
```
|
||||
|
||||
### Test with Real Bounce Email
|
||||
|
||||
To test with a real bounce:
|
||||
|
||||
1. Send an email to a non-existent address via your mailing list
|
||||
2. Wait for the bounce to be processed
|
||||
3. Check the database for bounce logs:
|
||||
|
||||
```bash
|
||||
# Check bounce logs
|
||||
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;"
|
||||
|
||||
# Check member bounce status
|
||||
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT member_id, email, bounce_count, bounce_status, last_bounce_at FROM members WHERE bounce_count > 0;"
|
||||
```
|
||||
|
||||
### View in Web Interface
|
||||
|
||||
1. Open http://localhost:3000
|
||||
2. Navigate to the Members tab
|
||||
3. Look for bounce badges and bounce counts next to member names
|
||||
4. Click the "Bounces" button next to a member to view bounce history
|
||||
|
||||
## Supported Bounce Email Formats
|
||||
|
||||
The processing script recognizes these bounce patterns:
|
||||
|
||||
### SMTP Error Codes
|
||||
- 550, 551, 553, 552, 554 (permanent failures)
|
||||
- 450, 451, 452 (temporary failures)
|
||||
|
||||
### Delivery Status Notification (DSN)
|
||||
- `Final-Recipient:` headers
|
||||
- `Original-Recipient:` headers
|
||||
|
||||
### Common Bounce Messages
|
||||
- "user unknown"
|
||||
- "does not exist"
|
||||
- "not found"
|
||||
- "mailbox unavailable"
|
||||
- "recipient rejected"
|
||||
|
||||
## Monitoring
|
||||
|
||||
### View Processing Logs
|
||||
|
||||
```bash
|
||||
# View bounce processing logs
|
||||
sudo docker-compose logs postfix | grep bounce-processor
|
||||
|
||||
# Follow logs in real-time
|
||||
sudo docker-compose logs -f postfix | grep bounce-processor
|
||||
```
|
||||
|
||||
### Check Aliases Configuration
|
||||
|
||||
```bash
|
||||
# Verify aliases are configured correctly
|
||||
sudo docker-compose exec postfix cat /etc/aliases
|
||||
|
||||
# Check alias database
|
||||
sudo docker-compose exec postfix postmap -q bounces /etc/aliases
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Bounces Not Being Processed
|
||||
|
||||
1. **Check aliases configuration:**
|
||||
```bash
|
||||
sudo docker-compose exec postfix cat /etc/aliases
|
||||
```
|
||||
|
||||
2. **Verify script permissions:**
|
||||
```bash
|
||||
sudo docker-compose exec postfix ls -la /usr/local/bin/process-bounce.py
|
||||
```
|
||||
|
||||
3. **Test script manually:**
|
||||
```bash
|
||||
sudo docker-compose exec postfix /usr/local/bin/process-bounce.py --test
|
||||
```
|
||||
|
||||
4. **Check Postfix logs:**
|
||||
```bash
|
||||
sudo docker-compose logs postfix | grep -i bounce
|
||||
```
|
||||
|
||||
### Database Connection Issues
|
||||
|
||||
1. **Check environment variables:**
|
||||
```bash
|
||||
sudo docker-compose exec postfix env | grep MYSQL
|
||||
```
|
||||
|
||||
2. **Test database connection:**
|
||||
```bash
|
||||
sudo docker-compose exec postfix python3 -c "import pymysql; print('PyMySQL available')"
|
||||
```
|
||||
|
||||
### Script Errors
|
||||
|
||||
View detailed error logs:
|
||||
```bash
|
||||
sudo docker-compose logs postfix | grep -A 10 -B 10 "bounce-processor.*ERROR"
|
||||
```
|
||||
|
||||
## Comparison with SNS Webhooks
|
||||
|
||||
| Feature | Email-Based | SNS Webhooks |
|
||||
|---------|-------------|--------------|
|
||||
| **SES Requirement** | Sandbox OK | Production access required |
|
||||
| **External Dependencies** | None | SNS, HTTPS domain |
|
||||
| **Processing Speed** | Real-time | Real-time |
|
||||
| **Setup Complexity** | Low | High |
|
||||
| **Reliability** | High | High |
|
||||
| **Bounce Detection** | Regex-based | AWS-provided |
|
||||
| **Cost** | Free | SNS charges apply |
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Monitor bounce processing** to ensure it's working correctly
|
||||
2. **Review bounce patterns** in the logs to improve detection if needed
|
||||
3. **Set up bounce notification alerts** (optional)
|
||||
4. **Consider upgrading to SNS webhooks** when SES production access is available
|
||||
|
||||
Email-based bounce handling provides a robust alternative that works immediately with any SES configuration while providing the same bounce management features as the SNS webhook method.
|
||||
Reference in New Issue
Block a user