sasa-maillist/EMAIL_BOUNCE_HANDLING_SETUP.md

Email-Based Bounce Handling Setup

This document explains the email-based bounce handling system implemented as an alternative to SNS webhooks for environments without SES production access.

Overview

The system processes email bounces directly within the Postfix container by:

1. Rewriting return paths to direct bounces to a processing address
2. Processing bounce emails via Python script
3. Updating member bounce statistics in MySQL database
4. Automatically disabling members with excessive bounces
5. Setting up an alias that pipes bounce emails to a Python processing script
6. The script parses bounce emails, extracts bounced addresses, and updates the database
7. Members with hard bounces are automatically deactivated
8. Bounce history is tracked and displayed in the UI (same as SNS method)

Advantages

• Works with SES Sandbox: No production SES access required
• No External Dependencies: Doesn't require SNS, webhooks, or HTTPS domains
• Self-Contained: All processing happens within the existing containers
• Real-time Processing: Bounces are processed as soon as emails arrive
• Compatible: Uses the same database schema and UI as SNS bounce handling

Configuration

1. Enable Email Bounce Processing

In your .env file:

# Enable email-based bounce processing
ENABLE_EMAIL_BOUNCE_PROCESSING=true

# This will automatically enable bounce handling features
ENABLE_BOUNCE_HANDLING=true  # Automatically set to true when email processing is enabled

2. Restart the System

sudo docker-compose down
sudo docker-compose up --build -d

3. Verify Configuration

Check that email bounce processing is enabled:

curl -s http://localhost:8000/config | jq .

Expected output:

{
  "bounce_handling_enabled": true,
  "sns_webhooks_enabled": false,
  "email_bounce_processing_enabled": true
}

How It Works

Postfix Configuration

The system configures Postfix with:

• bounce_notice_recipient = bounces@lists.sasalliance.org
• 2bounce_notice_recipient = bounces@lists.sasalliance.org
• error_notice_recipient = bounces@lists.sasalliance.org

Aliases Configuration

The bounces address is configured to pipe emails to the processing script:

bounces: "|/usr/local/bin/process-bounce.py"

Bounce Processing Script

The Python script (/usr/local/bin/process-bounce.py):

1. Reads bounce emails from stdin (via pipe)
2. Parses email content using multiple regex patterns to extract bounced addresses
3. Analyzes bounce type based on SMTP error codes:
   • 5xx codes = Permanent bounces
   • 4xx codes = Transient bounces
   • Unknown = Undetermined
4. Updates database using the same schema as SNS bounce handling:
   • Logs bounce in bounce_logs table
   • Updates member bounce_count, bounce_status, and last_bounce_at
   • Deactivates members with permanent bounces
   • Marks members with soft bounce status after 3 transient bounces

Testing

Test the Processing Script

You can test the bounce processing script in test mode:

# Test with sample bounce email
sudo docker-compose exec postfix /usr/local/bin/process-bounce.py --test

Expected output:

2025-10-14 15:49:16,041 - bounce-processor - INFO - Starting bounce processing
2025-10-14 15:49:16,041 - bounce-processor - INFO - Running in test mode with sample bounce email
2025-10-14 15:49:16,050 - bounce-processor - INFO - Extracted addresses: ['testuser@example.com']
2025-10-14 15:49:16,050 - bounce-processor - INFO - Test mode - would process 1 bounce(s):
2025-10-14 15:49:16,050 - bounce-processor - INFO -   {'email': 'testuser@example.com', 'bounce_type': 'Permanent', 'bounce_subtype': 'General', 'diagnostic_code': '', 'timestamp': '2025-10-14 15:49:16'}

Test with Real Bounce Email

To test with a real bounce:

1. Send an email to a non-existent address via your mailing list
2. Wait for the bounce to be processed
3. Check the database for bounce logs:

# Check bounce logs
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;"

# Check member bounce status
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT member_id, email, bounce_count, bounce_status, last_bounce_at FROM members WHERE bounce_count > 0;"

View in Web Interface

1. Open http://localhost:3000
2. Navigate to the Members tab
3. Look for bounce badges and bounce counts next to member names
4. Click the "Bounces" button next to a member to view bounce history

Supported Bounce Email Formats

The processing script recognizes these bounce patterns:

SMTP Error Codes

• 550, 551, 553, 552, 554 (permanent failures)
• 450, 451, 452 (temporary failures)

Delivery Status Notification (DSN)

• Final-Recipient: headers
• Original-Recipient: headers

Common Bounce Messages

• "user unknown"
• "does not exist"
• "not found"
• "mailbox unavailable"
• "recipient rejected"

Monitoring

View Processing Logs

# View bounce processing logs
sudo docker-compose logs postfix | grep bounce-processor

# Follow logs in real-time
sudo docker-compose logs -f postfix | grep bounce-processor

Check Aliases Configuration

# Verify aliases are configured correctly
sudo docker-compose exec postfix cat /etc/aliases

# Check alias database
sudo docker-compose exec postfix postmap -q bounces /etc/aliases

Troubleshooting

Bounces Not Being Processed

1. Check aliases configuration:

   sudo docker-compose exec postfix cat /etc/aliases
   

2. Verify script permissions:

   sudo docker-compose exec postfix ls -la /usr/local/bin/process-bounce.py
   

3. Test script manually:

   sudo docker-compose exec postfix /usr/local/bin/process-bounce.py --test
   

4. Check Postfix logs:

   sudo docker-compose logs postfix | grep -i bounce
   

Database Connection Issues

1. Check environment variables:

   sudo docker-compose exec postfix env | grep MYSQL
   

2. Test database connection:

   sudo docker-compose exec postfix python3 -c "import pymysql; print('PyMySQL available')"
   

Script Errors

View detailed error logs:

sudo docker-compose logs postfix | grep -A 10 -B 10 "bounce-processor.*ERROR"

Comparison with SNS Webhooks

Feature	Email-Based	SNS Webhooks
SES Requirement	Sandbox OK	Production access required
External Dependencies	None	SNS, HTTPS domain
Processing Speed	Real-time	Real-time
Setup Complexity	Low	High
Reliability	High	High
Bounce Detection	Regex-based	AWS-provided
Cost	Free	SNS charges apply

Next Steps

1. Monitor bounce processing to ensure it's working correctly
2. Review bounce patterns in the logs to improve detection if needed
3. Set up bounce notification alerts (optional)
4. Consider upgrading to SNS webhooks when SES production access is available

Email-based bounce handling provides a robust alternative that works immediately with any SES configuration while providing the same bounce management features as the SNS webhook method.