SES SNS Bounce Handling

This commit is contained in:
James Pattinson
2025-10-13 15:05:42 +00:00
parent ac23638125
commit 72f3297a80
12 changed files with 1276 additions and 3 deletions

255
BOUNCE_HANDLING_SETUP.md Normal file
View File

@@ -0,0 +1,255 @@
# SES SNS Bounce Handling Setup
This document describes how to configure AWS SES and SNS to handle email bounces automatically in the Mail List Manager.
## Overview
The system uses AWS Simple Notification Service (SNS) to receive real-time bounce notifications from AWS Simple Email Service (SES). When an email bounces:
1. SES sends a notification to an SNS topic
2. SNS forwards the notification to your webhook endpoint
3. The API processes the notification and updates the database
4. Members with hard bounces are automatically deactivated
5. Bounce history is tracked and displayed in the UI
## Bounce Status Types
- **Clean**: No bounces recorded
- **Soft Bounce**: Temporary delivery issues (e.g., mailbox full, temporary server issues)
- After 3 soft bounces, the member is marked with soft bounce status
- **Hard Bounce**: Permanent delivery failure (e.g., invalid email address, domain doesn't exist)
- Member is automatically deactivated and cannot receive emails
## Setup Instructions
### 1. Prerequisites
- AWS account with SES configured and verified
- Your Mail List Manager deployed and accessible via HTTPS (required for SNS webhook)
- Domain or subdomain for webhook (e.g., `https://lists.yourdomain.com`)
### 2. Create SNS Topic
1. Log in to AWS Console and navigate to SNS
2. Click "Create topic"
3. Choose "Standard" topic type
4. Name: `ses-bounce-notifications` (or your preferred name)
5. Display name: `SES Bounce Notifications`
6. Click "Create topic"
7. **Save the Topic ARN** (you'll need it in step 4)
### 3. Subscribe Your Webhook to SNS Topic
1. In the SNS topic details, click "Create subscription"
2. Protocol: `HTTPS`
3. Endpoint: `https://yourdomain.com:8000/webhooks/sns`
- Replace `yourdomain.com` with your actual domain
- The API must be accessible via HTTPS (SNS doesn't support HTTP)
4. Enable raw message delivery: **Unchecked**
5. Click "Create subscription"
6. The subscription will be in "PendingConfirmation" status
### 4. Confirm SNS Subscription
When you create the subscription, SNS will send a `SubscriptionConfirmation` request to your webhook endpoint. The Mail List Manager API automatically confirms this subscription.
1. Check your API logs:
```bash
sudo docker-compose logs -f api
```
2. You should see a log entry indicating the subscription was confirmed
3. In the AWS SNS console, refresh the subscriptions list
4. The status should change from "PendingConfirmation" to "Confirmed"
### 5. Configure SES to Send Bounce Notifications
1. Navigate to AWS SES console
2. Go to "Configuration Sets" (or "Verified identities" > select your domain > "Notifications")
3. For configuration sets:
- Create a new configuration set or select existing
- Add "Event destination"
- Event types: Select **"Bounce"** (and optionally "Complaint")
- Destination: SNS topic
- Select your SNS topic created in step 2
4. For verified identities:
- Select your sending domain/email
- Click "Edit" in the "Notifications" section
- Bounce feedback: Select your SNS topic
- Include original headers: Enabled (optional)
- Click "Save changes"
### 6. Verify Setup
#### Test with a Bounce Simulator
AWS SES provides bounce simulator addresses:
```bash
# From inside Postfix container
docker-compose exec postfix bash
echo "Test bounce" | mail -s "Test" bounce@simulator.amazonses.com
```
Or send to your mailing list with a test recipient:
1. Add `bounce@simulator.amazonses.com` as a member
2. Subscribe to a test list
3. Send an email to the list
#### Check the Results
1. Wait a few minutes for SES to process and send the notification
2. Check API logs:
```bash
sudo docker-compose logs api | grep -i bounce
```
3. Log in to the web UI
4. Go to Members tab
5. Find the test member and click the "Bounces" button
6. You should see the bounce event recorded
### 7. Security Considerations
#### SNS Signature Verification
The webhook endpoint automatically verifies SNS message signatures to ensure notifications are genuine AWS messages. This prevents unauthorized parties from sending fake bounce notifications.
#### HTTPS Requirement
SNS requires HTTPS for webhooks. You'll need:
- Valid SSL/TLS certificate for your domain
- Reverse proxy (e.g., Nginx, Apache) in front of the API container
- Or use AWS API Gateway as a proxy
#### Example Nginx Configuration
```nginx
server {
listen 443 ssl http2;
server_name lists.yourdomain.com;
ssl_certificate /path/to/cert.pem;
ssl_certificate_key /path/to/key.pem;
# Webhook endpoint
location /webhooks/sns {
proxy_pass http://localhost:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Optional: proxy API for web UI
location /api {
proxy_pass http://localhost:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
```
### 8. Managing Bounces in the UI
#### View Bounce Status
In the Members tab, bounced emails are indicated with:
- Warning badge showing bounce count
- Color-coded status (yellow for soft bounce, red for hard bounce)
- Last bounce timestamp
#### View Bounce History
1. Click the "Bounces" button next to a member
2. View detailed bounce history including:
- Bounce type (Permanent, Transient, Undetermined)
- Bounce subtype
- Diagnostic code from the receiving mail server
- Timestamp of each bounce
#### Reset Bounce Status
If a member's email has been corrected or verified:
1. Open the bounce history modal
2. Click "Reset Bounce Status"
3. Confirm the action
4. The member's bounce count is cleared and they can receive emails again
**Note**: Only users with write access (administrators and operators) can reset bounce status.
### 9. Monitoring and Maintenance
#### Check Bounce Logs
```bash
# View all bounces in database
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT * FROM bounce_logs ORDER BY timestamp DESC LIMIT 20;"
# Count bounces by type
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT bounce_type, COUNT(*) as count FROM bounce_logs GROUP BY bounce_type;"
# Find members with bounces
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT name, email, bounce_count, bounce_status FROM members WHERE bounce_count > 0;"
```
#### API Health Check
```bash
# Check if webhook is accessible
curl -X POST https://yourdomain.com:8000/webhooks/sns \
-H "Content-Type: application/json" \
-d '{"Type":"test"}'
```
#### Clean Up Old Bounce Records
Periodically review and clean up old bounce records:
```sql
-- Delete bounce logs older than 90 days
DELETE FROM bounce_logs WHERE created_at < DATE_SUB(NOW(), INTERVAL 90 DAY);
```
## Troubleshooting
### SNS Subscription Not Confirming
- Ensure the API container is running and accessible via HTTPS
- Check API logs for errors
- Verify firewall rules allow HTTPS traffic to port 8000
- Test the endpoint manually: `curl https://yourdomain.com:8000/health`
### Bounces Not Being Recorded
1. Verify SNS topic is receiving messages:
- Check SNS topic metrics in AWS Console
2. Verify subscription is active:
- Check subscription status in SNS console
3. Check API logs for webhook errors:
```bash
sudo docker-compose logs api | grep -i "sns\|bounce"
```
4. Test signature verification:
- Temporarily add debug logging to the webhook endpoint
### Members Not Being Deactivated
- Check if bounce type is "Permanent"
- Review member's bounce_status in database:
```bash
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT * FROM members WHERE email='problem@example.com';"
```
- Verify bounce processing logic in API logs
### SSL Certificate Issues
If using self-signed certificates, SNS will reject the webhook. You must use:
- Valid certificate from a trusted CA (Let's Encrypt, etc.)
- Or use AWS Certificate Manager with API Gateway
## Additional Resources
- [AWS SES Bounce Handling](https://docs.aws.amazon.com/ses/latest/dg/event-publishing-retrieving-sns.html)
- [AWS SNS HTTPS Subscriptions](https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html)
- [SES Bounce Types](https://docs.aws.amazon.com/ses/latest/dg/notification-contents.html#bounce-types)

View File

@@ -317,6 +317,7 @@ docker-compose up -d # Reinitializes from schema.sql
- ✅ **REST API** - Complete programmatic access with token auth
- ✅ **Sender Whitelist** - Only authorized domains can send to lists
- ✅ **SES Integration** - Reliable email delivery through Amazon SES
- ✅ **Bounce Handling** - Automatic tracking and management of email bounces via SNS
- ✅ **Secure** - Private Docker network, token auth, environment-based credentials
- ✅ **Flexible** - Manage via web, API, or direct database access
- ✅ **Scalable** - Database-driven architecture supports many lists and members
@@ -326,6 +327,7 @@ docker-compose up -d # Reinitializes from schema.sql
- **Web Interface**: See `web/README.md` for frontend features and usage
- **REST API**: See `api/README.md` for complete API reference
- **Database**: See `database/README.md` for schema and SQL examples
- **Bounce Handling**: See `BOUNCE_HANDLING_SETUP.md` for SNS configuration
- **AI Agents**: See `.github/copilot-instructions.md` for development guidance
## Roadmap
@@ -336,6 +338,7 @@ docker-compose up -d # Reinitializes from schema.sql
- [x] Multi-service Docker Compose architecture
- [x] REST API with authentication
- [x] Sender whitelist for authorized domains
- [x] Bounce handling with SES SNS integration
- [ ] Email verification workflow for new members
- [ ] Subscription confirmation (double opt-in)
- [ ] List archive functionality

View File

@@ -5,6 +5,7 @@ FastAPI-based REST API for managing mailing lists and members
from fastapi import FastAPI, HTTPException, Depends, Header, status, Request
from fastapi.middleware.cors import CORSMiddleware
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from fastapi.responses import PlainTextResponse
from pydantic import BaseModel, EmailStr
from typing import List, Optional, Annotated
import mysql.connector
@@ -19,6 +20,15 @@ import bcrypt
from jose import JWTError, jwt
from passlib.context import CryptContext
from enum import Enum
import json
import base64
from urllib.parse import urlparse
import httpx
from cryptography import x509
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives.serialization import load_pem_public_key
# Configuration
API_TOKEN = os.getenv('API_TOKEN', 'change-this-token') # Keep for backward compatibility during transition
@@ -229,6 +239,9 @@ class Member(BaseModel):
name: str
email: EmailStr
active: bool = True
bounce_count: Optional[int] = 0
last_bounce_at: Optional[datetime] = None
bounce_status: Optional[str] = 'clean'
class MemberUpdate(BaseModel):
name: Optional[str] = None
@@ -808,6 +821,257 @@ async def bulk_import_members(bulk_request: BulkImportRequest, current_user: Cur
cursor.close()
raise HTTPException(status_code=500, detail=f"Bulk import failed: {str(e)}")
# SNS Webhook for Bounce Handling
async def verify_sns_signature(request: Request) -> dict:
"""Verify SNS message signature"""
try:
body = await request.body()
message = json.loads(body.decode('utf-8'))
# For SubscriptionConfirmation and UnsubscribeConfirmation, we don't validate signature
# AWS will send a URL to confirm
if message.get('Type') in ['SubscriptionConfirmation', 'UnsubscribeConfirmation']:
return message
# Get certificate URL and download certificate
cert_url = message.get('SigningCertURL')
if not cert_url:
raise HTTPException(status_code=400, detail="Missing SigningCertURL")
# Verify certificate URL is from AWS
parsed_url = urlparse(cert_url)
if not parsed_url.hostname.endswith('.amazonaws.com'):
raise HTTPException(status_code=400, detail="Invalid certificate URL")
# Download certificate
async with httpx.AsyncClient() as client:
cert_response = await client.get(cert_url)
cert_response.raise_for_status()
cert_pem = cert_response.content
# Load certificate and extract public key
cert = x509.load_pem_x509_certificate(cert_pem, default_backend())
public_key = cert.public_key()
# Build string to sign
if message.get('Type') == 'Notification':
string_to_sign = (
f"Message\n{message['Message']}\n"
f"MessageId\n{message['MessageId']}\n"
)
if 'Subject' in message:
string_to_sign += f"Subject\n{message['Subject']}\n"
string_to_sign += (
f"Timestamp\n{message['Timestamp']}\n"
f"TopicArn\n{message['TopicArn']}\n"
f"Type\n{message['Type']}\n"
)
else:
string_to_sign = (
f"Message\n{message['Message']}\n"
f"MessageId\n{message['MessageId']}\n"
f"SubscribeURL\n{message['SubscribeURL']}\n"
f"Timestamp\n{message['Timestamp']}\n"
f"Token\n{message['Token']}\n"
f"TopicArn\n{message['TopicArn']}\n"
f"Type\n{message['Type']}\n"
)
# Verify signature
signature = base64.b64decode(message['Signature'])
try:
public_key.verify(
signature,
string_to_sign.encode('utf-8'),
padding.PKCS1v15(),
hashes.SHA1()
)
except Exception as e:
raise HTTPException(status_code=400, detail=f"Invalid signature: {str(e)}")
return message
except json.JSONDecodeError:
raise HTTPException(status_code=400, detail="Invalid JSON")
except Exception as e:
raise HTTPException(status_code=400, detail=f"Signature verification failed: {str(e)}")
async def process_bounce(bounce_data: dict):
"""Process bounce notification and update database"""
try:
bounce_type = bounce_data.get('bounceType') # Permanent, Transient, Undetermined
bounce_subtype = bounce_data.get('bounceSubType', '')
timestamp = bounce_data.get('timestamp')
feedback_id = bounce_data.get('feedbackId', '')
bounced_recipients = bounce_data.get('bouncedRecipients', [])
with get_db() as conn:
cursor = conn.cursor(dictionary=True)
for recipient in bounced_recipients:
email = recipient.get('emailAddress')
diagnostic_code = recipient.get('diagnosticCode', '')
if not email:
continue
# Find member by email
cursor.execute("SELECT member_id FROM members WHERE email = %s", (email,))
member = cursor.fetchone()
member_id = member['member_id'] if member else None
# Log the bounce
cursor.execute("""
INSERT INTO bounce_logs
(member_id, email, bounce_type, bounce_subtype, diagnostic_code, timestamp, feedback_id)
VALUES (%s, %s, %s, %s, %s, %s, %s)
""", (member_id, email, bounce_type, bounce_subtype, diagnostic_code, timestamp, feedback_id))
# Update member bounce status
if member_id:
# Determine bounce status
if bounce_type == 'Permanent':
new_status = 'hard_bounce'
# Deactivate member with hard bounce
cursor.execute("""
UPDATE members
SET bounce_count = bounce_count + 1,
last_bounce_at = %s,
bounce_status = %s,
active = 0
WHERE member_id = %s
""", (timestamp, new_status, member_id))
elif bounce_type == 'Transient':
# Check current bounce count
cursor.execute("SELECT bounce_count, bounce_status FROM members WHERE member_id = %s", (member_id,))
current = cursor.fetchone()
# If already hard bounced, don't change status
if current and current['bounce_status'] != 'hard_bounce':
new_count = current['bounce_count'] + 1
# After 3 soft bounces, mark as soft_bounce status
new_status = 'soft_bounce' if new_count >= 3 else 'clean'
cursor.execute("""
UPDATE members
SET bounce_count = %s,
last_bounce_at = %s,
bounce_status = %s
WHERE member_id = %s
""", (new_count, timestamp, new_status, member_id))
else: # Undetermined
cursor.execute("""
UPDATE members
SET bounce_count = bounce_count + 1,
last_bounce_at = %s
WHERE member_id = %s
""", (timestamp, member_id))
conn.commit()
cursor.close()
except Exception as e:
print(f"Error processing bounce: {str(e)}")
raise
@app.post("/webhooks/sns", response_class=PlainTextResponse)
async def sns_webhook(request: Request):
"""Handle SNS notifications for bounces and complaints"""
try:
# Verify SNS signature
message = await verify_sns_signature(request)
message_type = message.get('Type')
# Handle subscription confirmation
if message_type == 'SubscriptionConfirmation':
subscribe_url = message.get('SubscribeURL')
if subscribe_url:
# Confirm subscription
async with httpx.AsyncClient() as client:
await client.get(subscribe_url)
return "Subscription confirmed"
# Handle notification
elif message_type == 'Notification':
# Parse the message
notification = json.loads(message.get('Message', '{}'))
notification_type = notification.get('notificationType')
if notification_type == 'Bounce':
bounce = notification.get('bounce', {})
await process_bounce(bounce)
return "Bounce processed"
elif notification_type == 'Complaint':
# We could also track complaints similarly to bounces
return "Complaint received"
return "OK"
except HTTPException:
raise
except Exception as e:
print(f"SNS webhook error: {str(e)}")
raise HTTPException(status_code=500, detail=str(e))
# Bounce management endpoints
class BounceLog(BaseModel):
bounce_id: int
email: str
bounce_type: str
bounce_subtype: Optional[str] = None
diagnostic_code: Optional[str] = None
timestamp: datetime
feedback_id: Optional[str] = None
created_at: datetime
class MemberWithBounces(BaseModel):
member_id: int
name: str
email: str
active: bool
bounce_count: int
last_bounce_at: Optional[datetime] = None
bounce_status: str
@app.get("/members/{member_id}/bounces", response_model=List[BounceLog])
async def get_member_bounces(member_id: int, current_user: CurrentUser = require_read_access()):
"""Get bounce history for a member"""
with get_db() as conn:
cursor = conn.cursor(dictionary=True)
cursor.execute("""
SELECT bounce_id, email, bounce_type, bounce_subtype, diagnostic_code,
timestamp, feedback_id, created_at
FROM bounce_logs
WHERE member_id = %s
ORDER BY timestamp DESC
""", (member_id,))
bounces = cursor.fetchall()
cursor.close()
return bounces
@app.patch("/members/{member_id}/bounce-status")
async def reset_bounce_status(member_id: int, current_user: CurrentUser = require_write_access()):
"""Reset bounce status for a member (e.g., after email address is corrected)"""
with get_db() as conn:
cursor = conn.cursor()
cursor.execute("""
UPDATE members
SET bounce_count = 0,
last_bounce_at = NULL,
bounce_status = 'clean'
WHERE member_id = %s
""", (member_id,))
conn.commit()
if cursor.rowcount == 0:
raise HTTPException(status_code=404, detail="Member not found")
cursor.close()
return {"message": "Bounce status reset successfully"}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)

View File

@@ -8,3 +8,4 @@ email-validator==2.1.0
bcrypt==4.0.1
python-jose[cryptography]==3.3.0
passlib[bcrypt]==1.7.4
httpx==0.25.2

View File

@@ -55,8 +55,12 @@ CREATE TABLE IF NOT EXISTS members (
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
active BOOLEAN DEFAULT TRUE,
bounce_count INT DEFAULT 0,
last_bounce_at TIMESTAMP NULL,
bounce_status ENUM('clean', 'soft_bounce', 'hard_bounce') DEFAULT 'clean',
INDEX idx_email (email),
INDEX idx_active (active)
INDEX idx_active (active),
INDEX idx_bounce_status (bounce_status)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- Table: list_members
@@ -75,6 +79,26 @@ CREATE TABLE IF NOT EXISTS list_members (
INDEX idx_active (active)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- Table: bounce_logs
-- Stores bounce notifications from SES SNS
CREATE TABLE IF NOT EXISTS bounce_logs (
bounce_id INT AUTO_INCREMENT PRIMARY KEY,
member_id INT,
email VARCHAR(255) NOT NULL,
bounce_type ENUM('Permanent', 'Transient', 'Undetermined') NOT NULL,
bounce_subtype VARCHAR(50),
diagnostic_code TEXT,
timestamp TIMESTAMP NOT NULL,
sns_message_id VARCHAR(255),
feedback_id VARCHAR(255),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (member_id) REFERENCES members(member_id) ON DELETE SET NULL,
INDEX idx_member_id (member_id),
INDEX idx_email (email),
INDEX idx_timestamp (timestamp),
INDEX idx_bounce_type (bounce_type)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- Insert sample data
-- Create default admin user (password: 'password')

206
simulate_bounce.sh Executable file
View File

@@ -0,0 +1,206 @@
#!/bin/bash
# Simulate Bounce Events for Testing
# This script inserts bounce data directly into the database to test the UI
# without needing to set up AWS SNS
set -e
echo "=== Bounce Simulation Script ==="
echo
echo "This script will create test bounce events for existing members"
echo
# Get database password from .env
DB_PASSWORD=$(grep MYSQL_ROOT_PASSWORD /home/jamesp/docker/maillist/.env | cut -d'=' -f2)
# Check if containers are running
if ! sudo docker-compose ps | grep -q "maillist-mysql.*Up"; then
echo "❌ MySQL container is not running. Starting containers..."
sudo docker-compose up -d
echo "Waiting for MySQL to be ready..."
sleep 5
fi
echo "1. Fetching existing members..."
echo
# Get list of members
MEMBERS=$(sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -N -e "SELECT member_id, name, email FROM members LIMIT 5;")
if [ -z "$MEMBERS" ]; then
echo "❌ No members found in database. Please add members first."
exit 1
fi
echo "Available members:"
echo "$MEMBERS" | while read -r line; do
MEMBER_ID=$(echo "$line" | awk '{print $1}')
NAME=$(echo "$line" | awk '{print $2, $3}')
EMAIL=$(echo "$line" | awk '{$1=""; $2=""; print $0}' | xargs)
echo " [$MEMBER_ID] $NAME - $EMAIL"
done
echo
# Prompt for member ID
read -p "Enter member ID to simulate bounce for (or press Enter for first member): " MEMBER_ID
if [ -z "$MEMBER_ID" ]; then
MEMBER_ID=$(echo "$MEMBERS" | head -1 | awk '{print $1}')
echo "Using member ID: $MEMBER_ID"
fi
# Get member details
MEMBER_INFO=$(sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -N -e "SELECT member_id, name, email FROM members WHERE member_id = $MEMBER_ID;")
if [ -z "$MEMBER_INFO" ]; then
echo "❌ Member ID $MEMBER_ID not found"
exit 1
fi
MEMBER_EMAIL=$(echo "$MEMBER_INFO" | awk '{$1=""; $2=""; print $0}' | xargs)
MEMBER_NAME=$(echo "$MEMBER_INFO" | awk '{print $2, $3}')
echo
echo "Selected member: $MEMBER_NAME ($MEMBER_EMAIL)"
echo
# Bounce type selection
echo "Select bounce type to simulate:"
echo " 1) Soft Bounce (Transient - e.g., mailbox full)"
echo " 2) Hard Bounce (Permanent - e.g., invalid address)"
echo " 3) Multiple Soft Bounces (3 bounces to trigger soft_bounce status)"
echo " 4) Undetermined Bounce"
echo
read -p "Enter choice (1-4): " BOUNCE_CHOICE
case $BOUNCE_CHOICE in
1)
BOUNCE_TYPE="Transient"
BOUNCE_SUBTYPE="MailboxFull"
DIAGNOSTIC="smtp; 452 4.2.2 Mailbox full"
COUNT=1
;;
2)
BOUNCE_TYPE="Permanent"
BOUNCE_SUBTYPE="General"
DIAGNOSTIC="smtp; 550 5.1.1 User unknown"
COUNT=1
;;
3)
BOUNCE_TYPE="Transient"
BOUNCE_SUBTYPE="General"
DIAGNOSTIC="smtp; 451 4.4.1 Temporary failure"
COUNT=3
;;
4)
BOUNCE_TYPE="Undetermined"
BOUNCE_SUBTYPE=""
DIAGNOSTIC="Unknown error occurred"
COUNT=1
;;
*)
echo "Invalid choice"
exit 1
;;
esac
echo
echo "Simulating $COUNT bounce event(s)..."
echo
# Insert bounce events
for i in $(seq 1 $COUNT); do
TIMESTAMP=$(date -u -d "-$((i * 24)) hours" '+%Y-%m-%d %H:%M:%S')
FEEDBACK_ID="test-feedback-$(date +%s)-$i"
echo "Creating bounce event $i/$COUNT (timestamp: $TIMESTAMP)..."
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
-- Insert bounce log
INSERT INTO bounce_logs (member_id, email, bounce_type, bounce_subtype, diagnostic_code, timestamp, feedback_id)
VALUES ($MEMBER_ID, '$MEMBER_EMAIL', '$BOUNCE_TYPE', '$BOUNCE_SUBTYPE', '$DIAGNOSTIC', '$TIMESTAMP', '$FEEDBACK_ID');
EOF
# Update member bounce status
if [ "$BOUNCE_TYPE" = "Permanent" ]; then
echo " → Hard bounce: Deactivating member..."
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
UPDATE members
SET bounce_count = bounce_count + 1,
last_bounce_at = '$TIMESTAMP',
bounce_status = 'hard_bounce',
active = 0
WHERE member_id = $MEMBER_ID;
EOF
elif [ "$BOUNCE_TYPE" = "Transient" ]; then
# Check current bounce count
CURRENT_COUNT=$(sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -N -e "SELECT bounce_count FROM members WHERE member_id = $MEMBER_ID;")
NEW_COUNT=$((CURRENT_COUNT + 1))
if [ $NEW_COUNT -ge 3 ]; then
echo " → Soft bounce threshold reached: Marking as soft_bounce..."
BOUNCE_STATUS="soft_bounce"
else
echo " → Soft bounce: Incrementing counter ($NEW_COUNT)..."
BOUNCE_STATUS="clean"
fi
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
UPDATE members
SET bounce_count = $NEW_COUNT,
last_bounce_at = '$TIMESTAMP',
bounce_status = '$BOUNCE_STATUS'
WHERE member_id = $MEMBER_ID;
EOF
else
echo " → Undetermined bounce: Incrementing counter..."
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
UPDATE members
SET bounce_count = bounce_count + 1,
last_bounce_at = '$TIMESTAMP'
WHERE member_id = $MEMBER_ID;
EOF
fi
done
echo
echo "✅ Bounce simulation complete!"
echo
# Show updated member status
echo "Updated member status:"
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -e "
SELECT
member_id,
name,
email,
active,
bounce_count,
last_bounce_at,
bounce_status
FROM members
WHERE member_id = $MEMBER_ID\G
"
echo
echo "Bounce history for this member:"
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -e "
SELECT
bounce_id,
bounce_type,
bounce_subtype,
diagnostic_code,
timestamp
FROM bounce_logs
WHERE member_id = $MEMBER_ID
ORDER BY timestamp DESC;
"
echo
echo "🎉 You can now view this bounce in the web UI:"
echo " 1. Open http://localhost:3000"
echo " 2. Go to the Members tab"
echo " 3. Look for $MEMBER_NAME"
echo " 4. Click the 'Bounces' button to see the history"
echo
echo "To simulate more bounces, run this script again!"

76
test_bounce_webhook.sh Executable file
View File

@@ -0,0 +1,76 @@
#!/bin/bash
# Test SNS Bounce Webhook
# This script simulates an SNS bounce notification (without signature verification for testing)
API_URL="http://localhost:8000"
echo "Testing SNS Bounce Webhook..."
echo
# Test 1: Health check
echo "1. Testing API health..."
curl -s "$API_URL/health" | jq .
echo
# Test 2: Create a test subscription confirmation (the webhook will auto-confirm)
echo "2. Testing subscription confirmation handling..."
curl -s -X POST "$API_URL/webhooks/sns" \
-H "Content-Type: application/json" \
-d '{
"Type": "SubscriptionConfirmation",
"MessageId": "test-message-id",
"Token": "test-token",
"TopicArn": "arn:aws:sns:eu-west-2:123456789:test-topic",
"Message": "You have chosen to subscribe to the topic",
"SubscribeURL": "https://example.com/subscribe",
"Timestamp": "2025-01-01T12:00:00.000Z"
}' || echo "Note: Signature verification will fail for test messages (expected)"
echo
echo
# Test 3: Simulated bounce notification (will fail signature verification)
echo "3. Testing bounce notification structure..."
echo "Note: In production, AWS SNS will send properly signed messages."
echo "This test demonstrates the expected structure."
echo
cat << 'EOF' > /tmp/test_bounce.json
{
"Type": "Notification",
"MessageId": "test-notification-id",
"TopicArn": "arn:aws:sns:eu-west-2:123456789:test-topic",
"Subject": "Amazon SES Email Event Notification",
"Message": "{\"notificationType\":\"Bounce\",\"bounce\":{\"bounceType\":\"Permanent\",\"bounceSubType\":\"General\",\"bouncedRecipients\":[{\"emailAddress\":\"bounce@simulator.amazonses.com\",\"diagnosticCode\":\"smtp; 550 5.1.1 user unknown\"}],\"timestamp\":\"2025-01-01T12:00:00.000Z\",\"feedbackId\":\"test-feedback-id\"}}",
"Timestamp": "2025-01-01T12:00:00.000Z",
"SignatureVersion": "1",
"Signature": "test-signature",
"SigningCertURL": "https://sns.eu-west-2.amazonaws.com/test.pem"
}
EOF
cat /tmp/test_bounce.json | jq .
echo
echo "Expected behavior: Signature verification will fail without real AWS credentials"
echo "In production, AWS SNS will send properly signed messages that will be verified"
echo
# Test 4: Check if database schema has bounce tables
echo "4. Checking database schema for bounce tables..."
sudo docker-compose exec -T mysql mysql -u maillist -pmaillist maillist -e "SHOW TABLES LIKE '%bounce%';" 2>/dev/null
echo
# Test 5: Check members table for bounce columns
echo "5. Checking members table for bounce columns..."
sudo docker-compose exec -T mysql mysql -u maillist -pmaillist maillist -e "DESCRIBE members;" 2>/dev/null | grep -i bounce
echo
echo "✓ Setup complete!"
echo
echo "To test with real AWS SNS:"
echo "1. Set up SNS topic in AWS Console"
echo "2. Subscribe webhook: https://your-domain.com:8000/webhooks/sns"
echo "3. Configure SES to send bounce notifications to the SNS topic"
echo "4. Send test email to bounce@simulator.amazonses.com"
echo
echo "See BOUNCE_HANDLING_SETUP.md for detailed setup instructions"

View File

@@ -581,6 +581,67 @@
</div>
</div>
<!-- Bounce History Modal -->
<div class="modal" id="bounceHistoryModal">
<div class="modal-content">
<div class="modal-header">
<h3 id="bounceHistoryTitle">Bounce History</h3>
<button class="modal-close" id="bounceHistoryModalClose">
<i class="fas fa-times"></i>
</button>
</div>
<div class="modal-body">
<div class="member-info-banner" id="bounceHistoryMemberInfo">
<div class="member-avatar">
<i class="fas fa-user"></i>
</div>
<div class="member-details">
<h4 id="bounceHistoryMemberName">Member Name</h4>
<p id="bounceHistoryMemberEmail">member@example.com</p>
</div>
</div>
<div class="bounce-summary">
<div class="bounce-stat">
<i class="fas fa-exclamation-triangle"></i>
<div>
<div class="bounce-stat-label">Total Bounces</div>
<div class="bounce-stat-value" id="bounceTotalCount">0</div>
</div>
</div>
<div class="bounce-stat">
<i class="fas fa-clock"></i>
<div>
<div class="bounce-stat-label">Last Bounce</div>
<div class="bounce-stat-value" id="bounceLastDate">Never</div>
</div>
</div>
<div class="bounce-stat">
<i class="fas fa-info-circle"></i>
<div>
<div class="bounce-stat-label">Status</div>
<div class="bounce-stat-value" id="bounceStatusText">Clean</div>
</div>
</div>
</div>
<div class="bounce-history-section">
<h5>Bounce Events</h5>
<div class="bounce-history-list" id="bounceHistoryList">
<!-- Dynamic content will be inserted here -->
</div>
</div>
</div>
<div class="modal-actions">
<button type="button" class="btn btn-secondary" id="bounceHistoryCloseBtn">Close</button>
<button type="button" class="btn btn-warning" id="bounceHistoryResetBtn" data-requires-write>
<i class="fas fa-redo"></i>
Reset Bounce Status
</button>
</div>
</div>
</div>
<!-- Confirmation Modal -->
<div class="modal" id="confirmModal">
<div class="modal-content">
@@ -601,7 +662,7 @@
</div>
<script src="static/js/api.js"></script>
<script src="static/js/ui.js"></script>
<script src="static/js/app.js"></script>
<script src="static/js/ui.js"></script>
</body>
</html>

View File

@@ -624,6 +624,32 @@ body {
color: var(--gray-600);
}
/* Bounce badges */
.bounce-badge {
display: inline-flex;
align-items: center;
gap: var(--space-1);
padding: var(--space-1) var(--space-2);
border-radius: var(--radius-sm);
font-size: var(--font-size-xs);
font-weight: 500;
}
.bounce-badge.bounce-hard {
background: #fee2e2;
color: #991b1b;
}
.bounce-badge.bounce-soft {
background: #fef3c7;
color: #92400e;
}
.bounce-badge.bounce-warning {
background: #fef9c3;
color: #854d0e;
}
/* Role badges */
.role-badge {
display: inline-flex;
@@ -1479,3 +1505,147 @@ body {
.fade-in {
animation: fadeIn 0.3s ease-out;
}
/* Bounce History Modal Styles */
.bounce-summary {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
gap: var(--space-4);
margin: var(--space-6) 0;
padding: var(--space-4);
background: var(--gray-50);
border-radius: var(--radius);
}
.bounce-stat {
display: flex;
align-items: center;
gap: var(--space-3);
}
.bounce-stat i {
font-size: 1.5rem;
color: var(--primary-color);
}
.bounce-stat-label {
font-size: var(--font-size-xs);
color: var(--gray-500);
text-transform: uppercase;
letter-spacing: 0.5px;
}
.bounce-stat-value {
font-size: var(--font-size-lg);
font-weight: 600;
color: var(--gray-800);
}
.bounce-stat-value.text-danger {
color: var(--danger-color);
}
.bounce-stat-value.text-warning {
color: var(--warning-color);
}
.bounce-stat-value.text-success {
color: var(--success-color);
}
.bounce-history-section {
margin-top: var(--space-6);
}
.bounce-history-section h5 {
margin-bottom: var(--space-4);
color: var(--gray-700);
}
.bounce-history-list {
max-height: 400px;
overflow-y: auto;
padding: var(--space-2);
}
.bounce-history-item {
background: var(--white);
border: 1px solid var(--gray-200);
border-radius: var(--radius);
padding: var(--space-4);
margin-bottom: var(--space-3);
}
.bounce-history-item:last-child {
margin-bottom: 0;
}
.bounce-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: var(--space-2);
}
.bounce-type {
display: inline-flex;
align-items: center;
gap: var(--space-2);
padding: var(--space-1) var(--space-3);
border-radius: var(--radius-sm);
font-size: var(--font-size-sm);
font-weight: 600;
}
.bounce-type.bounce-type-permanent {
background: #fee2e2;
color: #991b1b;
}
.bounce-type.bounce-type-transient {
background: #fef3c7;
color: #92400e;
}
.bounce-type.bounce-type-undetermined {
background: var(--gray-100);
color: var(--gray-600);
}
.bounce-date {
font-size: var(--font-size-sm);
color: var(--gray-500);
}
.bounce-subtype {
font-size: var(--font-size-sm);
color: var(--gray-600);
margin-top: var(--space-2);
font-weight: 500;
}
.bounce-diagnostic {
font-size: var(--font-size-sm);
color: var(--gray-500);
margin-top: var(--space-2);
padding: var(--space-2);
background: var(--gray-50);
border-radius: var(--radius-sm);
border-left: 3px solid var(--warning-color);
font-family: 'Courier New', monospace;
}
.empty-state {
text-align: center;
padding: var(--space-8);
color: var(--gray-400);
}
.empty-state i {
font-size: 3rem;
margin-bottom: var(--space-4);
color: var(--success-color);
}
.empty-state p {
font-size: var(--font-size-sm);
}

View File

@@ -254,6 +254,17 @@ class APIClient {
})
});
}
// Bounce management API
async getMemberBounces(memberId) {
return this.request(`/members/${memberId}/bounces`);
}
async resetBounceStatus(memberId) {
return this.request(`/members/${memberId}/bounce-status`, {
method: 'PATCH'
});
}
}
/**

View File

@@ -438,6 +438,7 @@ class MailingListApp {
row.innerHTML = `
<td>
<div class="font-medium">${uiManager.escapeHtml(member.name)}</div>
${member.bounce_count > 0 ? `<div class="text-xs text-muted" style="margin-top: 2px;"></div>` : ''}
</td>
<td>
<a href="mailto:${member.email}" style="color: var(--primary-color)">
@@ -458,6 +459,15 @@ class MailingListApp {
</td>
`;
// Add bounce badge if member has bounces
if (member.bounce_count > 0) {
const bounceInfoDiv = row.cells[0].querySelector('.text-xs');
const bounceBadge = uiManager.createBounceStatusBadge(member.bounce_status, member.bounce_count);
if (bounceBadge) {
bounceInfoDiv.appendChild(bounceBadge);
}
}
// Add status badge
const statusCell = row.cells[3];
statusCell.appendChild(uiManager.createStatusBadge(member.active));
@@ -474,6 +484,18 @@ class MailingListApp {
uiManager.showMemberSubscriptionsModal(member);
});
// Create Bounces button (show if member has any bounces or for admins/operators)
if (member.bounce_count > 0 || hasWriteAccess) {
const bouncesBtn = document.createElement('button');
bouncesBtn.className = `btn btn-sm ${member.bounce_count > 0 ? 'btn-warning' : 'btn-secondary'}`;
bouncesBtn.innerHTML = `<i class="fas fa-exclamation-triangle"></i> Bounces${member.bounce_count > 0 ? ` (${member.bounce_count})` : ''}`;
bouncesBtn.title = 'View Bounce History';
bouncesBtn.addEventListener('click', () => {
uiManager.showBounceHistoryModal(member);
});
actionsCell.appendChild(bouncesBtn);
}
// Only show edit/delete buttons for users with write access
if (hasWriteAccess) {
const editBtn = uiManager.createActionButton('Edit', 'edit', 'btn-secondary', () => {

View File

@@ -76,6 +76,19 @@ class UIManager {
this.handleMemberSubscriptionsSave();
});
// Bounce history modal
document.getElementById('bounceHistoryModalClose').addEventListener('click', () => {
this.closeModal(document.getElementById('bounceHistoryModal'));
});
document.getElementById('bounceHistoryCloseBtn').addEventListener('click', () => {
this.closeModal(document.getElementById('bounceHistoryModal'));
});
document.getElementById('bounceHistoryResetBtn').addEventListener('click', () => {
this.handleBounceStatusReset();
});
// Form submissions
document.getElementById('listForm').addEventListener('submit', (e) => {
e.preventDefault();
@@ -110,6 +123,19 @@ class UIManager {
this.confirmCallback = null;
this.closeModal(document.getElementById('confirmModal'));
});
// Bounce history modal
document.getElementById('bounceHistoryModalClose').addEventListener('click', () => {
this.closeModal(document.getElementById('bounceHistoryModal'));
});
document.getElementById('bounceHistoryCloseBtn').addEventListener('click', () => {
this.closeModal(document.getElementById('bounceHistoryModal'));
});
document.getElementById('bounceHistoryResetBtn').addEventListener('click', () => {
this.handleResetBounceStatus();
});
}
/**
@@ -664,6 +690,117 @@ class UIManager {
}
}
/**
* Show bounce history modal for a member
*/
async showBounceHistoryModal(member) {
const modal = document.getElementById('bounceHistoryModal');
// Update member info
document.getElementById('bounceHistoryTitle').textContent = `Bounce History - ${member.name}`;
document.getElementById('bounceHistoryMemberName').textContent = member.name;
document.getElementById('bounceHistoryMemberEmail').textContent = member.email;
// Update summary stats
document.getElementById('bounceTotalCount').textContent = member.bounce_count || 0;
document.getElementById('bounceLastDate').textContent = this.formatDateTime(member.last_bounce_at);
const statusText = document.getElementById('bounceStatusText');
statusText.className = 'bounce-stat-value';
if (member.bounce_status === 'hard_bounce') {
statusText.textContent = 'Hard Bounce';
statusText.classList.add('text-danger');
} else if (member.bounce_status === 'soft_bounce') {
statusText.textContent = 'Soft Bounce';
statusText.classList.add('text-warning');
} else {
statusText.textContent = 'Clean';
statusText.classList.add('text-success');
}
try {
this.currentMemberForBounces = member;
// Load bounce history
const bounces = await apiClient.getMemberBounces(member.member_id);
this.renderBounceHistory(bounces);
this.showModal(modal);
} catch (error) {
this.handleError(error, 'Failed to load bounce history');
}
}
/**
* Render bounce history list
*/
renderBounceHistory(bounces) {
const container = document.getElementById('bounceHistoryList');
container.innerHTML = '';
if (bounces.length === 0) {
container.innerHTML = `
<div class="empty-state">
<i class="fas fa-check-circle"></i>
<p>No bounces recorded for this member</p>
</div>
`;
return;
}
bounces.forEach(bounce => {
const item = document.createElement('div');
item.className = 'bounce-history-item';
let typeClass = 'bounce-type-';
if (bounce.bounce_type === 'Permanent') {
typeClass += 'permanent';
} else if (bounce.bounce_type === 'Transient') {
typeClass += 'transient';
} else {
typeClass += 'undetermined';
}
item.innerHTML = `
<div class="bounce-header">
<span class="bounce-type ${typeClass}">
<i class="fas fa-${bounce.bounce_type === 'Permanent' ? 'times-circle' : bounce.bounce_type === 'Transient' ? 'exclamation-circle' : 'question-circle'}"></i>
${bounce.bounce_type}
</span>
<span class="bounce-date">${this.formatDateTime(bounce.timestamp)}</span>
</div>
${bounce.bounce_subtype ? `<div class="bounce-subtype">Subtype: ${this.escapeHtml(bounce.bounce_subtype)}</div>` : ''}
${bounce.diagnostic_code ? `<div class="bounce-diagnostic">${this.escapeHtml(bounce.diagnostic_code)}</div>` : ''}
`;
container.appendChild(item);
});
}
/**
* Handle bounce status reset
*/
async handleResetBounceStatus() {
if (!this.currentMemberForBounces) return;
this.showConfirmation(
`Are you sure you want to reset the bounce status for "${this.currentMemberForBounces.name}"? This will clear the bounce count and allow emails to be sent to this address again.`,
async () => {
try {
this.setLoading(true);
await apiClient.resetBounceStatus(this.currentMemberForBounces.member_id);
this.showNotification('Bounce status reset successfully', 'success');
this.closeModal(document.getElementById('bounceHistoryModal'));
await window.app.loadData();
} catch (error) {
this.handleError(error, 'Failed to reset bounce status');
} finally {
this.setLoading(false);
}
}
);
}
/**
* Handle API errors
*/
@@ -715,6 +852,49 @@ class UIManager {
return badge;
}
/**
* Create bounce status badge
*/
createBounceStatusBadge(bounceStatus, bounceCount) {
const badge = document.createElement('span');
if (bounceStatus === 'hard_bounce') {
badge.className = 'bounce-badge bounce-hard';
badge.innerHTML = `<i class="fas fa-exclamation-triangle"></i> Hard Bounce`;
badge.title = `${bounceCount} bounce(s) - Email permanently failed`;
} else if (bounceStatus === 'soft_bounce') {
badge.className = 'bounce-badge bounce-soft';
badge.innerHTML = `<i class="fas fa-exclamation-circle"></i> Soft Bounce`;
badge.title = `${bounceCount} bounce(s) - Temporary delivery issues`;
} else if (bounceCount > 0) {
badge.className = 'bounce-badge bounce-warning';
badge.innerHTML = `<i class="fas fa-info-circle"></i> ${bounceCount} bounce(s)`;
badge.title = `${bounceCount} bounce(s) recorded`;
} else {
return null; // No badge for clean status
}
return badge;
}
/**
* Format date and time
*/
formatDateTime(dateString) {
if (!dateString) return 'Never';
const date = new Date(dateString);
return date.toLocaleString();
}
/**
* Format date only
*/
formatDate(dateString) {
if (!dateString) return 'Never';
const date = new Date(dateString);
return date.toLocaleDateString();
}
/**
* Format email as mailto link
*/