Ability to disable SNS bounce handling

This commit is contained in:
James Pattinson
2025-10-14 15:39:33 +00:00
parent b34ea2ed84
commit 12a82c8d03
6 changed files with 407 additions and 301 deletions

View File

@@ -20,3 +20,15 @@ MYSQL_ROOT_PASSWORD=change_this_root_password
# API Configuration # API Configuration
API_TOKEN=change_this_to_a_secure_random_token API_TOKEN=change_this_to_a_secure_random_token
# Bounce Handling Configuration (Optional)
# Set to 'true' to enable SNS webhook bounce handling
# Set to 'false' to disable and rely on email-based bounce handling
ENABLE_SNS_WEBHOOKS=false
ENABLE_BOUNCE_HANDLING=false
# If ENABLE_SNS_WEBHOOKS=true, you need:
# 1. AWS SNS topic configured
# 2. SES configured to send notifications to SNS topic
# 3. Valid HTTPS domain for webhook endpoint
# 4. SNS subscription confirmed to your webhook endpoint

View File

@@ -1,10 +1,17 @@
# SES SNS Bounce Handling Setup # SES SNS Bounce Handling Setup (Optional)
**⚠️ NOTICE: Bounce handling is optional and disabled by default.**
This document describes how to configure AWS SES and SNS to handle email bounces automatically in the Mail List Manager. This document describes how to configure AWS SES and SNS to handle email bounces automatically in the Mail List Manager.
**Prerequisites:**
- SES production access (not available in sandbox mode)
- Valid HTTPS domain for webhook endpoint
- Bounce handling must be enabled in configuration
## Overview ## Overview
The system uses AWS Simple Notification Service (SNS) to receive real-time bounce notifications from AWS Simple Email Service (SES). When an email bounces: The system can optionally use AWS Simple Notification Service (SNS) to receive real-time bounce notifications from AWS Simple Email Service (SES). When bounce handling is enabled and an email bounces:
1. SES sends a notification to an SNS topic 1. SES sends a notification to an SNS topic
2. SNS forwards the notification to your webhook endpoint 2. SNS forwards the notification to your webhook endpoint
@@ -22,13 +29,28 @@ The system uses AWS Simple Notification Service (SNS) to receive real-time bounc
## Setup Instructions ## Setup Instructions
### 1. Prerequisites ### 1. Enable Bounce Handling
First, enable bounce handling in your `.env` file:
```bash
# Enable SNS webhook bounce handling
ENABLE_SNS_WEBHOOKS=true
ENABLE_BOUNCE_HANDLING=true
```
Restart the API container after making this change:
```bash
sudo docker-compose restart api
```
### 2. Prerequisites
- AWS account with SES configured and verified - AWS account with SES configured and verified
- Your Mail List Manager deployed and accessible via HTTPS (required for SNS webhook) - Your Mail List Manager deployed and accessible via HTTPS (required for SNS webhook)
- Domain or subdomain for webhook (e.g., `https://lists.yourdomain.com`) - Domain or subdomain for webhook (e.g., `https://lists.yourdomain.com`)
### 2. Create SNS Topic ### 3. Create SNS Topic
1. Log in to AWS Console and navigate to SNS 1. Log in to AWS Console and navigate to SNS
2. Click "Create topic" 2. Click "Create topic"
@@ -38,7 +60,7 @@ The system uses AWS Simple Notification Service (SNS) to receive real-time bounc
6. Click "Create topic" 6. Click "Create topic"
7. **Save the Topic ARN** (you'll need it in step 4) arn:aws:sns:eu-west-2:827164363113:ses-bounces 7. **Save the Topic ARN** (you'll need it in step 4) arn:aws:sns:eu-west-2:827164363113:ses-bounces
### 3. Subscribe Your Webhook to SNS Topic ### 4. Subscribe Your Webhook to SNS Topic
1. In the SNS topic details, click "Create subscription" 1. In the SNS topic details, click "Create subscription"
2. Protocol: `HTTPS` 2. Protocol: `HTTPS`
@@ -49,7 +71,7 @@ The system uses AWS Simple Notification Service (SNS) to receive real-time bounc
5. Click "Create subscription" 5. Click "Create subscription"
6. The subscription will be in "PendingConfirmation" status 6. The subscription will be in "PendingConfirmation" status
### 4. Confirm SNS Subscription ### 5. Confirm SNS Subscription
When you create the subscription, SNS will send a `SubscriptionConfirmation` request to your webhook endpoint. The Mail List Manager API automatically confirms this subscription. When you create the subscription, SNS will send a `SubscriptionConfirmation` request to your webhook endpoint. The Mail List Manager API automatically confirms this subscription.
@@ -61,7 +83,7 @@ When you create the subscription, SNS will send a `SubscriptionConfirmation` req
3. In the AWS SNS console, refresh the subscriptions list 3. In the AWS SNS console, refresh the subscriptions list
4. The status should change from "PendingConfirmation" to "Confirmed" 4. The status should change from "PendingConfirmation" to "Confirmed"
### 5. Configure SES to Send Bounce Notifications ### 6. Configure SES to Send Bounce Notifications
1. Navigate to AWS SES console 1. Navigate to AWS SES console
2. Go to "Configuration Sets" (or "Verified identities" > select your domain > "Notifications") 2. Go to "Configuration Sets" (or "Verified identities" > select your domain > "Notifications")
@@ -78,7 +100,7 @@ When you create the subscription, SNS will send a `SubscriptionConfirmation` req
- Include original headers: Enabled (optional) - Include original headers: Enabled (optional)
- Click "Save changes" - Click "Save changes"
### 6. Verify Setup ### 7. Verify Setup
#### Test with a Bounce Simulator #### Test with a Bounce Simulator
@@ -108,7 +130,7 @@ Or send to your mailing list with a test recipient:
5. Find the test member and click the "Bounces" button 5. Find the test member and click the "Bounces" button
6. You should see the bounce event recorded 6. You should see the bounce event recorded
### 7. Security Considerations ### 8. Security Considerations
#### SNS Signature Verification #### SNS Signature Verification
@@ -149,7 +171,7 @@ server {
} }
``` ```
### 8. Managing Bounces in the UI ### 9. Managing Bounces in the UI
#### View Bounce Status #### View Bounce Status
@@ -178,7 +200,7 @@ If a member's email has been corrected or verified:
**Note**: Only users with write access (administrators and operators) can reset bounce status. **Note**: Only users with write access (administrators and operators) can reset bounce status.
### 9. Monitoring and Maintenance ### 10. Monitoring and Maintenance
#### Check Bounce Logs #### Check Bounce Logs

View File

@@ -172,6 +172,41 @@ docker-compose exec postfix postmap -q "community@lists.sasalliance.org" \
mysql:/etc/postfix/mysql_virtual_alias_maps.cf mysql:/etc/postfix/mysql_virtual_alias_maps.cf
``` ```
### Bounce Handling (Optional)
**Email bounce handling is optional and disabled by default.**
**Two Configuration Options:**
1. **SNS Webhooks** (Requires SES Production Access):
```bash
# In .env file
ENABLE_SNS_WEBHOOKS=true
ENABLE_BOUNCE_HANDLING=true
```
- Real-time bounce notifications via AWS SNS
- Automatic member deactivation for hard bounces
- Bounce history tracking and management
- Requires valid HTTPS domain and SES production access
- See `BOUNCE_HANDLING_SETUP.md` for complete setup
2. **Email-Based Handling** (Default for SES Sandbox):
```bash
# In .env file (or leave these commented out)
ENABLE_SNS_WEBHOOKS=false
ENABLE_BOUNCE_HANDLING=false
```
- Manual bounce management via email notifications
- No automatic processing - requires manual member cleanup
- Works with SES sandbox accounts
- Bounce-related UI elements are hidden
**When bounce handling is disabled:**
- `/webhooks/sns` endpoint is not registered
- Bounce history endpoints return empty results
- Web UI hides bounce badges and bounce management buttons
- No automatic member deactivation occurs
## Security ## Security
- **Environment Variables**: All credentials stored in `.env` (git-ignored) - **Environment Variables**: All credentials stored in `.env` (git-ignored)

View File

@@ -43,6 +43,10 @@ MYSQL_DATABASE = os.getenv('MYSQL_DATABASE', 'maillist')
MYSQL_USER = os.getenv('MYSQL_USER', 'maillist') MYSQL_USER = os.getenv('MYSQL_USER', 'maillist')
MYSQL_PASSWORD = os.getenv('MYSQL_PASSWORD', '') MYSQL_PASSWORD = os.getenv('MYSQL_PASSWORD', '')
# Bounce handling configuration
ENABLE_SNS_WEBHOOKS = os.getenv('ENABLE_SNS_WEBHOOKS', 'false').lower() == 'true'
ENABLE_BOUNCE_HANDLING = os.getenv('ENABLE_BOUNCE_HANDLING', 'false').lower() == 'true'
# Password hashing # Password hashing
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto") pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
@@ -463,6 +467,14 @@ async def health():
except Exception as e: except Exception as e:
raise HTTPException(status_code=503, detail=f"Unhealthy: {str(e)}") raise HTTPException(status_code=503, detail=f"Unhealthy: {str(e)}")
@app.get("/config")
async def get_config():
"""Get public configuration settings"""
return {
"bounce_handling_enabled": ENABLE_BOUNCE_HANDLING,
"sns_webhooks_enabled": ENABLE_SNS_WEBHOOKS
}
# Mailing Lists endpoints # Mailing Lists endpoints
@app.get("/lists", response_model=List[MailingList]) @app.get("/lists", response_model=List[MailingList])
async def get_lists(current_user: CurrentUser = require_read_access()): async def get_lists(current_user: CurrentUser = require_read_access()):
@@ -824,8 +836,9 @@ async def bulk_import_members(bulk_request: BulkImportRequest, current_user: Cur
cursor.close() cursor.close()
raise HTTPException(status_code=500, detail=f"Bulk import failed: {str(e)}") raise HTTPException(status_code=500, detail=f"Bulk import failed: {str(e)}")
# SNS Webhook for Bounce Handling # SNS Webhook for Bounce Handling (conditionally enabled)
async def verify_sns_signature(request: Request) -> dict: if ENABLE_SNS_WEBHOOKS:
async def verify_sns_signature(request: Request) -> dict:
"""Verify SNS message signature""" """Verify SNS message signature"""
try: try:
body = await request.body() body = await request.body()
@@ -908,7 +921,7 @@ async def verify_sns_signature(request: Request) -> dict:
except Exception as e: except Exception as e:
raise HTTPException(status_code=400, detail=f"Signature verification failed: {str(e)}") raise HTTPException(status_code=400, detail=f"Signature verification failed: {str(e)}")
async def process_bounce(bounce_data: dict): async def process_bounce(bounce_data: dict):
"""Process bounce notification and update database""" """Process bounce notification and update database"""
try: try:
bounce_type = bounce_data.get('bounceType') # Permanent, Transient, Undetermined bounce_type = bounce_data.get('bounceType') # Permanent, Transient, Undetermined
@@ -997,8 +1010,8 @@ async def process_bounce(bounce_data: dict):
traceback.print_exc() traceback.print_exc()
raise raise
@app.post("/webhooks/sns", response_class=PlainTextResponse) @app.post("/webhooks/sns", response_class=PlainTextResponse)
async def sns_webhook(request: Request): async def sns_webhook(request: Request):
"""Handle SNS notifications for bounces and complaints""" """Handle SNS notifications for bounces and complaints"""
try: try:
print(f"=== SNS Webhook Request ===") print(f"=== SNS Webhook Request ===")
@@ -1061,8 +1074,13 @@ async def sns_webhook(request: Request):
traceback.print_exc() traceback.print_exc()
raise HTTPException(status_code=500, detail=str(e)) raise HTTPException(status_code=500, detail=str(e))
# Bounce management endpoints else:
class BounceLog(BaseModel): # Provide stub functions when SNS webhooks are disabled
print("SNS webhooks disabled - bounce handling via email only")
# Bounce management endpoints (conditionally enabled)
if ENABLE_BOUNCE_HANDLING:
class BounceLog(BaseModel):
bounce_id: int bounce_id: int
email: str email: str
bounce_type: str bounce_type: str
@@ -1072,7 +1090,7 @@ class BounceLog(BaseModel):
feedback_id: Optional[str] = None feedback_id: Optional[str] = None
created_at: datetime created_at: datetime
class MemberWithBounces(BaseModel): class MemberWithBounces(BaseModel):
member_id: int member_id: int
name: str name: str
email: str email: str
@@ -1081,8 +1099,8 @@ class MemberWithBounces(BaseModel):
last_bounce_at: Optional[datetime] = None last_bounce_at: Optional[datetime] = None
bounce_status: str bounce_status: str
@app.get("/members/{member_id}/bounces", response_model=List[BounceLog]) @app.get("/members/{member_id}/bounces", response_model=List[BounceLog])
async def get_member_bounces(member_id: int, current_user: CurrentUser = require_read_access()): async def get_member_bounces(member_id: int, current_user: CurrentUser = require_read_access()):
"""Get bounce history for a member""" """Get bounce history for a member"""
with get_db() as conn: with get_db() as conn:
cursor = conn.cursor(dictionary=True) cursor = conn.cursor(dictionary=True)
@@ -1097,8 +1115,8 @@ async def get_member_bounces(member_id: int, current_user: CurrentUser = require
cursor.close() cursor.close()
return bounces return bounces
@app.patch("/members/{member_id}/bounce-status") @app.patch("/members/{member_id}/bounce-status")
async def reset_bounce_status(member_id: int, current_user: CurrentUser = require_write_access()): async def reset_bounce_status(member_id: int, current_user: CurrentUser = require_write_access()):
"""Reset bounce status for a member (e.g., after email address is corrected)""" """Reset bounce status for a member (e.g., after email address is corrected)"""
with get_db() as conn: with get_db() as conn:
cursor = conn.cursor() cursor = conn.cursor()
@@ -1117,6 +1135,18 @@ async def reset_bounce_status(member_id: int, current_user: CurrentUser = requir
cursor.close() cursor.close()
return {"message": "Bounce status reset successfully"} return {"message": "Bounce status reset successfully"}
else:
# When bounce handling is disabled, provide stub endpoints that return appropriate responses
@app.get("/members/{member_id}/bounces")
async def get_member_bounces_disabled(member_id: int, current_user: CurrentUser = require_read_access()):
"""Bounce history disabled - returns empty list"""
return []
@app.patch("/members/{member_id}/bounce-status")
async def reset_bounce_status_disabled(member_id: int, current_user: CurrentUser = require_write_access()):
"""Bounce status reset disabled"""
raise HTTPException(status_code=501, detail="Bounce handling is disabled")
if __name__ == "__main__": if __name__ == "__main__":
import uvicorn import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000) uvicorn.run(app, host="0.0.0.0", port=8000)

View File

@@ -105,6 +105,10 @@ class APIClient {
return this.request('/health'); return this.request('/health');
} }
async getConfig() {
return this.request('/config');
}
async testAuth() { async testAuth() {
return this.request('/'); return this.request('/');
} }

View File

@@ -352,12 +352,15 @@ class MailingListApp {
try { try {
uiManager.setLoading(true); uiManager.setLoading(true);
// Load lists and members in parallel // Load configuration, lists and members in parallel
const [lists, members] = await Promise.all([ const [config, lists, members] = await Promise.all([
apiClient.getConfig(),
apiClient.getLists(), apiClient.getLists(),
apiClient.getMembers() apiClient.getMembers()
]); ]);
this.config = config;
this.lists = lists; this.lists = lists;
this.members = members; this.members = members;
@@ -531,7 +534,7 @@ class MailingListApp {
row.innerHTML = ` row.innerHTML = `
<td> <td>
<div class="font-medium">${uiManager.escapeHtml(member.name)}</div> <div class="font-medium">${uiManager.escapeHtml(member.name)}</div>
${member.bounce_count > 0 ? `<div class="text-xs text-muted" style="margin-top: 2px;"></div>` : ''} ${this.config?.bounce_handling_enabled && member.bounce_count > 0 ? `<div class="text-xs text-muted" style="margin-top: 2px;"></div>` : ''}
</td> </td>
<td> <td>
<a href="mailto:${member.email}" style="color: var(--primary-color)"> <a href="mailto:${member.email}" style="color: var(--primary-color)">
@@ -552,8 +555,8 @@ class MailingListApp {
</td> </td>
`; `;
// Add bounce badge if member has bounces // Add bounce badge if member has bounces (only if bounce handling is enabled)
if (member.bounce_count > 0) { if (this.config?.bounce_handling_enabled && member.bounce_count > 0) {
const bounceInfoDiv = row.cells[0].querySelector('.text-xs'); const bounceInfoDiv = row.cells[0].querySelector('.text-xs');
const bounceBadge = uiManager.createBounceStatusBadge(member.bounce_status, member.bounce_count); const bounceBadge = uiManager.createBounceStatusBadge(member.bounce_status, member.bounce_count);
if (bounceBadge) { if (bounceBadge) {
@@ -577,8 +580,8 @@ class MailingListApp {
uiManager.showMemberSubscriptionsModal(member); uiManager.showMemberSubscriptionsModal(member);
}); });
// Create Bounces button (show if member has any bounces or for admins/operators) // Create Bounces button (show if bounce handling is enabled and member has bounces or for admins/operators)
if (member.bounce_count > 0 || hasWriteAccess) { if (this.config?.bounce_handling_enabled && (member.bounce_count > 0 || hasWriteAccess)) {
const bouncesBtn = document.createElement('button'); const bouncesBtn = document.createElement('button');
bouncesBtn.className = `btn btn-sm ${member.bounce_count > 0 ? 'btn-warning' : 'btn-secondary'}`; bouncesBtn.className = `btn btn-sm ${member.bounce_count > 0 ? 'btn-warning' : 'btn-secondary'}`;
bouncesBtn.innerHTML = `<i class="fas fa-exclamation-triangle"></i> Bounces${member.bounce_count > 0 ? ` (${member.bounce_count})` : ''}`; bouncesBtn.innerHTML = `<i class="fas fa-exclamation-triangle"></i> Bounces${member.bounce_count > 0 ? ` (${member.bounce_count})` : ''}`;
@@ -852,8 +855,8 @@ class MailingListApp {
statusBadge.className = `status-badge ${member.active ? 'active' : 'inactive'}`; statusBadge.className = `status-badge ${member.active ? 'active' : 'inactive'}`;
statusBadge.innerHTML = `<i class="fas fa-${member.active ? 'check' : 'times'}"></i> ${member.active ? 'Active' : 'Inactive'}`; statusBadge.innerHTML = `<i class="fas fa-${member.active ? 'check' : 'times'}"></i> ${member.active ? 'Active' : 'Inactive'}`;
// Add bounce status if exists // Add bounce status if exists and bounce handling is enabled
if (member.bounce_status && member.bounce_status !== 'clean') { if (this.config?.bounce_handling_enabled && member.bounce_status && member.bounce_status !== 'clean') {
const bounceIndicator = document.createElement('span'); const bounceIndicator = document.createElement('span');
bounceIndicator.className = `bounce-badge bounce-${member.bounce_status === 'hard_bounce' ? 'hard' : 'soft'}`; bounceIndicator.className = `bounce-badge bounce-${member.bounce_status === 'hard_bounce' ? 'hard' : 'soft'}`;
bounceIndicator.innerHTML = `<i class="fas fa-exclamation-triangle"></i> Bounces`; bounceIndicator.innerHTML = `<i class="fas fa-exclamation-triangle"></i> Bounces`;
@@ -868,8 +871,8 @@ class MailingListApp {
const actionsCell = row.insertCell(); const actionsCell = row.insertCell();
actionsCell.className = 'action-buttons'; actionsCell.className = 'action-buttons';
// Bounces button (if member has bounce data) // Bounces button (if bounce handling is enabled and member has bounce data)
if (member.bounce_count > 0) { if (this.config?.bounce_handling_enabled && member.bounce_count > 0) {
const bouncesBtn = uiManager.createActionButton('Bounces', 'exclamation-triangle', 'btn-warning', () => { const bouncesBtn = uiManager.createActionButton('Bounces', 'exclamation-triangle', 'btn-warning', () => {
uiManager.showBounceHistory(member); uiManager.showBounceHistory(member);
}); });