Compare commits

..

7 Commits

Author SHA1 Message Date
James Pattinson
f3d7592e7d Email bounce handling with postfix 2025-10-14 16:16:44 +00:00
James Pattinson
12a82c8d03 Ability to disable SNS bounce handling 2025-10-14 15:39:33 +00:00
James Pattinson
b34ea2ed84 Dynamic search 2025-10-13 20:05:08 +00:00
James Pattinson
8fd951fd1f FIx menu bar 2025-10-13 19:58:46 +00:00
James Pattinson
ecbc38cf8e Cleanuop and CORS fixing 2025-10-13 19:30:19 +00:00
James Pattinson
d37027ee5a Fixed bounce handling 2025-10-13 16:53:22 +00:00
James Pattinson
72f3297a80 SES SNS Bounce Handling 2025-10-13 15:05:42 +00:00
28 changed files with 3939 additions and 104 deletions

View File

@@ -20,3 +20,26 @@ MYSQL_ROOT_PASSWORD=change_this_root_password
# API Configuration
API_TOKEN=change_this_to_a_secure_random_token
# Bounce Handling Configuration (Optional)
# Choose one of the bounce handling methods below:
# Method 1: SNS Webhook Bounce Handling (Requires SES Production Access)
# Set to 'true' to enable real-time SNS webhook bounce handling
ENABLE_SNS_WEBHOOKS=false
ENABLE_BOUNCE_HANDLING=false
# If ENABLE_SNS_WEBHOOKS=true, you need:
# 1. AWS SNS topic configured
# 2. SES configured to send notifications to SNS topic
# 3. Valid HTTPS domain for webhook endpoint
# 4. SNS subscription confirmed to your webhook endpoint
# Method 2: Email-Based Bounce Processing (Works with SES Sandbox)
# Set to 'true' to enable email-based bounce processing
# This processes bounce emails that are sent to bounces@lists.sasalliance.org
ENABLE_EMAIL_BOUNCE_PROCESSING=false
# Note: You can enable both methods, but typically only one is needed
# Email-based processing works with SES sandbox accounts
# SNS webhooks provide real-time processing but require SES production access

277
BOUNCE_HANDLING_SETUP.md Normal file
View File

@@ -0,0 +1,277 @@
# SES SNS Bounce Handling Setup (Optional)
**⚠️ NOTICE: Bounce handling is optional and disabled by default.**
This document describes how to configure AWS SES and SNS to handle email bounces automatically in the Mail List Manager.
**Prerequisites:**
- SES production access (not available in sandbox mode)
- Valid HTTPS domain for webhook endpoint
- Bounce handling must be enabled in configuration
## Overview
The system can optionally use AWS Simple Notification Service (SNS) to receive real-time bounce notifications from AWS Simple Email Service (SES). When bounce handling is enabled and an email bounces:
1. SES sends a notification to an SNS topic
2. SNS forwards the notification to your webhook endpoint
3. The API processes the notification and updates the database
4. Members with hard bounces are automatically deactivated
5. Bounce history is tracked and displayed in the UI
## Bounce Status Types
- **Clean**: No bounces recorded
- **Soft Bounce**: Temporary delivery issues (e.g., mailbox full, temporary server issues)
- After 3 soft bounces, the member is marked with soft bounce status
- **Hard Bounce**: Permanent delivery failure (e.g., invalid email address, domain doesn't exist)
- Member is automatically deactivated and cannot receive emails
## Setup Instructions
### 1. Enable Bounce Handling
First, enable bounce handling in your `.env` file:
```bash
# Enable SNS webhook bounce handling
ENABLE_SNS_WEBHOOKS=true
ENABLE_BOUNCE_HANDLING=true
```
Restart the API container after making this change:
```bash
sudo docker-compose restart api
```
### 2. Prerequisites
- AWS account with SES configured and verified
- Your Mail List Manager deployed and accessible via HTTPS (required for SNS webhook)
- Domain or subdomain for webhook (e.g., `https://lists.yourdomain.com`)
### 3. Create SNS Topic
1. Log in to AWS Console and navigate to SNS
2. Click "Create topic"
3. Choose "Standard" topic type
4. Name: `ses-bounce-notifications` (or your preferred name)
5. Display name: `SES Bounce Notifications`
6. Click "Create topic"
7. **Save the Topic ARN** (you'll need it in step 4) arn:aws:sns:eu-west-2:827164363113:ses-bounces
### 4. Subscribe Your Webhook to SNS Topic
1. In the SNS topic details, click "Create subscription"
2. Protocol: `HTTPS`
3. Endpoint: `https://yourdomain.com:8000/webhooks/sns`
- Replace `yourdomain.com` with your actual domain
- The API must be accessible via HTTPS (SNS doesn't support HTTP)
4. Enable raw message delivery: **Unchecked**
5. Click "Create subscription"
6. The subscription will be in "PendingConfirmation" status
### 5. Confirm SNS Subscription
When you create the subscription, SNS will send a `SubscriptionConfirmation` request to your webhook endpoint. The Mail List Manager API automatically confirms this subscription.
1. Check your API logs:
```bash
sudo docker-compose logs -f api
```
2. You should see a log entry indicating the subscription was confirmed
3. In the AWS SNS console, refresh the subscriptions list
4. The status should change from "PendingConfirmation" to "Confirmed"
### 6. Configure SES to Send Bounce Notifications
1. Navigate to AWS SES console
2. Go to "Configuration Sets" (or "Verified identities" > select your domain > "Notifications")
3. For configuration sets:
- Create a new configuration set or select existing
- Add "Event destination"
- Event types: Select **"Bounce"** (and optionally "Complaint")
- Destination: SNS topic
- Select your SNS topic created in step 2
4. For verified identities:
- Select your sending domain/email
- Click "Edit" in the "Notifications" section
- Bounce feedback: Select your SNS topic
- Include original headers: Enabled (optional)
- Click "Save changes"
### 7. Verify Setup
#### Test with a Bounce Simulator
AWS SES provides bounce simulator addresses:
```bash
# From inside Postfix container
docker-compose exec postfix bash
echo "Test bounce" | mail -s "Test" bounce@simulator.amazonses.com
```
Or send to your mailing list with a test recipient:
1. Add `bounce@simulator.amazonses.com` as a member
2. Subscribe to a test list
3. Send an email to the list
#### Check the Results
1. Wait a few minutes for SES to process and send the notification
2. Check API logs:
```bash
sudo docker-compose logs api | grep -i bounce
```
3. Log in to the web UI
4. Go to Members tab
5. Find the test member and click the "Bounces" button
6. You should see the bounce event recorded
### 8. Security Considerations
#### SNS Signature Verification
The webhook endpoint automatically verifies SNS message signatures to ensure notifications are genuine AWS messages. This prevents unauthorized parties from sending fake bounce notifications.
#### HTTPS Requirement
SNS requires HTTPS for webhooks. You'll need:
- Valid SSL/TLS certificate for your domain
- Reverse proxy (e.g., Nginx, Apache) in front of the API container
- Or use AWS API Gateway as a proxy
#### Example Nginx Configuration
```nginx
server {
listen 443 ssl http2;
server_name lists.yourdomain.com;
ssl_certificate /path/to/cert.pem;
ssl_certificate_key /path/to/key.pem;
# Webhook endpoint
location /webhooks/sns {
proxy_pass http://localhost:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Optional: proxy API for web UI
location /api {
proxy_pass http://localhost:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
```
### 9. Managing Bounces in the UI
#### View Bounce Status
In the Members tab, bounced emails are indicated with:
- Warning badge showing bounce count
- Color-coded status (yellow for soft bounce, red for hard bounce)
- Last bounce timestamp
#### View Bounce History
1. Click the "Bounces" button next to a member
2. View detailed bounce history including:
- Bounce type (Permanent, Transient, Undetermined)
- Bounce subtype
- Diagnostic code from the receiving mail server
- Timestamp of each bounce
#### Reset Bounce Status
If a member's email has been corrected or verified:
1. Open the bounce history modal
2. Click "Reset Bounce Status"
3. Confirm the action
4. The member's bounce count is cleared and they can receive emails again
**Note**: Only users with write access (administrators and operators) can reset bounce status.
### 10. Monitoring and Maintenance
#### Check Bounce Logs
```bash
# View all bounces in database
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT * FROM bounce_logs ORDER BY timestamp DESC LIMIT 20;"
# Count bounces by type
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT bounce_type, COUNT(*) as count FROM bounce_logs GROUP BY bounce_type;"
# Find members with bounces
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT name, email, bounce_count, bounce_status FROM members WHERE bounce_count > 0;"
```
#### API Health Check
```bash
# Check if webhook is accessible
curl -X POST https://yourdomain.com:8000/webhooks/sns \
-H "Content-Type: application/json" \
-d '{"Type":"test"}'
```
#### Clean Up Old Bounce Records
Periodically review and clean up old bounce records:
```sql
-- Delete bounce logs older than 90 days
DELETE FROM bounce_logs WHERE created_at < DATE_SUB(NOW(), INTERVAL 90 DAY);
```
## Troubleshooting
### SNS Subscription Not Confirming
- Ensure the API container is running and accessible via HTTPS
- Check API logs for errors
- Verify firewall rules allow HTTPS traffic to port 8000
- Test the endpoint manually: `curl https://yourdomain.com:8000/health`
### Bounces Not Being Recorded
1. Verify SNS topic is receiving messages:
- Check SNS topic metrics in AWS Console
2. Verify subscription is active:
- Check subscription status in SNS console
3. Check API logs for webhook errors:
```bash
sudo docker-compose logs api | grep -i "sns\|bounce"
```
4. Test signature verification:
- Temporarily add debug logging to the webhook endpoint
### Members Not Being Deactivated
- Check if bounce type is "Permanent"
- Review member's bounce_status in database:
```bash
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT * FROM members WHERE email='problem@example.com';"
```
- Verify bounce processing logic in API logs
### SSL Certificate Issues
If using self-signed certificates, SNS will reject the webhook. You must use:
- Valid certificate from a trusted CA (Let's Encrypt, etc.)
- Or use AWS Certificate Manager with API Gateway
## Additional Resources
- [AWS SES Bounce Handling](https://docs.aws.amazon.com/ses/latest/dg/event-publishing-retrieving-sns.html)
- [AWS SNS HTTPS Subscriptions](https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html)
- [SES Bounce Types](https://docs.aws.amazon.com/ses/latest/dg/notification-contents.html#bounce-types)

240
CODE_REVIEW_FINDINGS.md Normal file
View File

@@ -0,0 +1,240 @@
# Code Review Findings & Fixes
## Date: 13 October 2025
## Critical Issues Found and Fixed
### 1. ✅ FIXED: Header Merging Bug in API Client
**Severity:** HIGH
**Location:** `web/static/js/api.js` - `request()` method
**Issue:**
The header merging was incorrect. When passing custom headers in options, they were being overridden by the default headers due to incorrect spread operator order:
```javascript
// BEFORE (BUGGY):
const config = {
headers: { ...this.headers },
...options // This overwrites headers!
};
```
**Fix:**
```javascript
// AFTER (FIXED):
const config = {
...options,
headers: {
...this.headers,
...(options.headers || {}) // Custom headers take precedence
}
};
```
**Impact:**
- The `login()` method was trying to remove the Authorization header for unauthenticated requests, but it was being overridden
- Could have caused authentication issues or unexpected behavior
---
### 2. ✅ FIXED: Missing API Client Methods
**Severity:** HIGH
**Location:** `web/static/js/api.js`
**Issue:** Multiple methods were being called by the UI but didn't exist in the API client:
- `login(username, password)` - ❌ Missing
- `logout()` - ❌ Missing
- `getCurrentUser()` - ❌ Missing
- `getUsers()` - ❌ Missing
- `createUser(userData)` - ❌ Missing
- `updateUser(userId, userData)` - ❌ Missing
- `deleteUser(userId)` - ❌ Missing
- `getMemberBounces(memberId)` - ❌ Missing
- `resetBounceStatus(memberId)` - ❌ Missing
**Fix:** Added all missing methods with proper implementation
---
### 3. ✅ FIXED: Incorrect API Base URL for Production
**Severity:** HIGH
**Location:** `web/static/js/api.js` - `getBaseURL()` method
**Issue:**
When running behind a reverse proxy (Caddy), the API client was trying to connect to `https://lists.sasalliance.org:8000` which:
- Port 8000 is not exposed through Caddy
- Would cause CORS issues
- Would fail all API requests
**Fix:**
```javascript
// Production detection
if (hostname === 'localhost' || hostname === '127.0.0.1') {
return `${protocol}//${hostname}:8000`; // Development
}
// Production - use reverse proxy path
return `${protocol}//${hostname}/api`;
```
---
## Potential Issues (Not Critical, But Worth Noting)
### 4. ⚠️ CORS Configuration
**Severity:** MEDIUM
**Location:** `api/main.py`
**Current Configuration:**
```python
app.add_middleware(
CORSMiddleware,
allow_origins=["*"], # Allows all origins
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
```
**Issue:**
- `allow_origins=["*"]` with `allow_credentials=True` is not allowed by browsers
- Currently working because Caddy reverse proxy makes same-origin requests
- If you ever need direct API access from different origins, this will fail
**Recommendation:**
```python
# For production
allow_origins=[
"https://lists.sasalliance.org",
"http://localhost:3000", # For development
],
```
---
### 5. ✅ GOOD: Authentication Token Storage
**Severity:** INFO
**Location:** `web/static/js/app.js`
**Current Implementation:**
```javascript
localStorage.setItem('authToken', response.access_token);
```
**Assessment:**
- ✅ Proper use of localStorage for JWT tokens
- ✅ Token is cleared on logout
- ✅ Token is validated on page load
- ✅ Expired tokens trigger automatic logout
**Note:** localStorage is appropriate for JWT tokens. HttpOnly cookies would be more secure but require different architecture.
---
### 6. ✅ GOOD: Error Handling
**Severity:** INFO
**Location:** `web/static/js/ui.js` - `handleError()` method
**Assessment:**
- ✅ Proper handling of 401 (auth) errors with automatic logout
- ✅ Proper handling of 403 (forbidden) errors
- ✅ Proper handling of 404 (not found) errors
- ✅ Proper handling of 500 (server) errors
- ✅ User-friendly error messages
---
### 7. ✅ GOOD: Role-Based Access Control
**Severity:** INFO
**Location:** Multiple files
**Assessment:**
- ✅ Backend enforces RBAC with decorators (`require_admin()`, `require_write_access()`, etc.)
- ✅ Frontend checks user roles before showing UI elements
- ✅ Three roles properly implemented: administrator, operator, read-only
- ✅ UI elements are hidden/shown based on permissions
---
### 8. ⚠️ Token Expiration Handling
**Severity:** LOW
**Location:** `api/main.py` and `web/static/js/app.js`
**Current Configuration:**
- JWT tokens expire in 30 minutes
- Session records expire in 24 hours (but not enforced in JWT)
**Potential Issue:**
- Users will be logged out after 30 minutes without warning
- No token refresh mechanism
**Recommendation (Future Enhancement):**
Consider implementing:
- Token refresh endpoint
- Warning before token expiration
- Silent token refresh in background
---
### 9. ✅ GOOD: SQL Injection Prevention
**Severity:** INFO
**Location:** `api/main.py`
**Assessment:**
- ✅ All SQL queries use parameterized statements with `%s` placeholders
- ✅ No string concatenation in SQL queries
- ✅ MySQL connector properly escapes parameters
---
### 10. ✅ GOOD: Password Security
**Severity:** INFO
**Location:** `api/main.py`
**Assessment:**
- ✅ Uses bcrypt for password hashing (`passlib.context.CryptContext`)
- ✅ Passwords are never stored in plain text
- ✅ Passwords are never logged or exposed in API responses
---
## Testing Recommendations
1. **Hard refresh browser cache** after updates (Ctrl+Shift+R)
2. **Test authentication flow:**
- Login with valid credentials
- Login with invalid credentials
- Token expiration after 30 minutes
- Logout functionality
3. **Test role-based access:**
- Administrator can see Users tab
- Operator can modify lists/members but not users
- Read-only can view but not modify
4. **Test bounce handling:**
- View bounce history
- Reset bounce status
5. **Test bulk import:**
- CSV upload
- Subscription assignment
---
## Summary
**3 Critical Bugs Fixed:**
1. Header merging bug in request method
2. Missing API client methods
3. Incorrect production API URL
**All Core Functionality Working:**
- Authentication and authorization
- CRUD operations for lists, members, and users
- Subscription management
- Bounce tracking
- Bulk import
⚠️ **Minor Improvements Suggested:**
1. Tighten CORS policy for production
2. Consider token refresh mechanism
3. Add user session timeout warnings
**Overall Assessment:** The codebase is now production-ready with proper security, error handling, and functionality. The critical bugs have been fixed and deployed.

View File

@@ -0,0 +1,243 @@
# Email-Based Bounce Handling Setup
This document explains the email-based bounce handling system implemented as an alternative to SNS webhooks for environments without SES production access.
## Overview
The system processes email bounces directly within the Postfix container by:
1. Rewriting return paths to direct bounces to a processing address
2. Processing bounce emails via Python script
3. Updating member bounce statistics in MySQL database
4. Automatically disabling members with excessive bounces
2. Setting up an alias that pipes bounce emails to a Python processing script
3. The script parses bounce emails, extracts bounced addresses, and updates the database
4. Members with hard bounces are automatically deactivated
5. Bounce history is tracked and displayed in the UI (same as SNS method)
## Advantages
- **Works with SES Sandbox**: No production SES access required
- **No External Dependencies**: Doesn't require SNS, webhooks, or HTTPS domains
- **Self-Contained**: All processing happens within the existing containers
- **Real-time Processing**: Bounces are processed as soon as emails arrive
- **Compatible**: Uses the same database schema and UI as SNS bounce handling
## Configuration
### 1. Enable Email Bounce Processing
In your `.env` file:
```bash
# Enable email-based bounce processing
ENABLE_EMAIL_BOUNCE_PROCESSING=true
# This will automatically enable bounce handling features
ENABLE_BOUNCE_HANDLING=true # Automatically set to true when email processing is enabled
```
### 2. Restart the System
```bash
sudo docker-compose down
sudo docker-compose up --build -d
```
### 3. Verify Configuration
Check that email bounce processing is enabled:
```bash
curl -s http://localhost:8000/config | jq .
```
Expected output:
```json
{
"bounce_handling_enabled": true,
"sns_webhooks_enabled": false,
"email_bounce_processing_enabled": true
}
```
## How It Works
### Postfix Configuration
The system configures Postfix with:
- `bounce_notice_recipient = bounces@lists.sasalliance.org`
- `2bounce_notice_recipient = bounces@lists.sasalliance.org`
- `error_notice_recipient = bounces@lists.sasalliance.org`
### Aliases Configuration
The `bounces` address is configured to pipe emails to the processing script:
```
bounces: "|/usr/local/bin/process-bounce.py"
```
### Bounce Processing Script
The Python script (`/usr/local/bin/process-bounce.py`):
1. **Reads bounce emails** from stdin (via pipe)
2. **Parses email content** using multiple regex patterns to extract bounced addresses
3. **Analyzes bounce type** based on SMTP error codes:
- 5xx codes = Permanent bounces
- 4xx codes = Transient bounces
- Unknown = Undetermined
4. **Updates database** using the same schema as SNS bounce handling:
- Logs bounce in `bounce_logs` table
- Updates member `bounce_count`, `bounce_status`, and `last_bounce_at`
- Deactivates members with permanent bounces
- Marks members with soft bounce status after 3 transient bounces
## Testing
### Test the Processing Script
You can test the bounce processing script in test mode:
```bash
# Test with sample bounce email
sudo docker-compose exec postfix /usr/local/bin/process-bounce.py --test
```
Expected output:
```
2025-10-14 15:49:16,041 - bounce-processor - INFO - Starting bounce processing
2025-10-14 15:49:16,041 - bounce-processor - INFO - Running in test mode with sample bounce email
2025-10-14 15:49:16,050 - bounce-processor - INFO - Extracted addresses: ['testuser@example.com']
2025-10-14 15:49:16,050 - bounce-processor - INFO - Test mode - would process 1 bounce(s):
2025-10-14 15:49:16,050 - bounce-processor - INFO - {'email': 'testuser@example.com', 'bounce_type': 'Permanent', 'bounce_subtype': 'General', 'diagnostic_code': '', 'timestamp': '2025-10-14 15:49:16'}
```
### Test with Real Bounce Email
To test with a real bounce:
1. Send an email to a non-existent address via your mailing list
2. Wait for the bounce to be processed
3. Check the database for bounce logs:
```bash
# Check bounce logs
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;"
# Check member bounce status
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT member_id, email, bounce_count, bounce_status, last_bounce_at FROM members WHERE bounce_count > 0;"
```
### View in Web Interface
1. Open http://localhost:3000
2. Navigate to the Members tab
3. Look for bounce badges and bounce counts next to member names
4. Click the "Bounces" button next to a member to view bounce history
## Supported Bounce Email Formats
The processing script recognizes these bounce patterns:
### SMTP Error Codes
- 550, 551, 553, 552, 554 (permanent failures)
- 450, 451, 452 (temporary failures)
### Delivery Status Notification (DSN)
- `Final-Recipient:` headers
- `Original-Recipient:` headers
### Common Bounce Messages
- "user unknown"
- "does not exist"
- "not found"
- "mailbox unavailable"
- "recipient rejected"
## Monitoring
### View Processing Logs
```bash
# View bounce processing logs
sudo docker-compose logs postfix | grep bounce-processor
# Follow logs in real-time
sudo docker-compose logs -f postfix | grep bounce-processor
```
### Check Aliases Configuration
```bash
# Verify aliases are configured correctly
sudo docker-compose exec postfix cat /etc/aliases
# Check alias database
sudo docker-compose exec postfix postmap -q bounces /etc/aliases
```
## Troubleshooting
### Bounces Not Being Processed
1. **Check aliases configuration:**
```bash
sudo docker-compose exec postfix cat /etc/aliases
```
2. **Verify script permissions:**
```bash
sudo docker-compose exec postfix ls -la /usr/local/bin/process-bounce.py
```
3. **Test script manually:**
```bash
sudo docker-compose exec postfix /usr/local/bin/process-bounce.py --test
```
4. **Check Postfix logs:**
```bash
sudo docker-compose logs postfix | grep -i bounce
```
### Database Connection Issues
1. **Check environment variables:**
```bash
sudo docker-compose exec postfix env | grep MYSQL
```
2. **Test database connection:**
```bash
sudo docker-compose exec postfix python3 -c "import pymysql; print('PyMySQL available')"
```
### Script Errors
View detailed error logs:
```bash
sudo docker-compose logs postfix | grep -A 10 -B 10 "bounce-processor.*ERROR"
```
## Comparison with SNS Webhooks
| Feature | Email-Based | SNS Webhooks |
|---------|-------------|--------------|
| **SES Requirement** | Sandbox OK | Production access required |
| **External Dependencies** | None | SNS, HTTPS domain |
| **Processing Speed** | Real-time | Real-time |
| **Setup Complexity** | Low | High |
| **Reliability** | High | High |
| **Bounce Detection** | Regex-based | AWS-provided |
| **Cost** | Free | SNS charges apply |
## Next Steps
1. **Monitor bounce processing** to ensure it's working correctly
2. **Review bounce patterns** in the logs to improve detection if needed
3. **Set up bounce notification alerts** (optional)
4. **Consider upgrading to SNS webhooks** when SES production access is available
Email-based bounce handling provides a robust alternative that works immediately with any SES configuration while providing the same bounce management features as the SNS webhook method.

View File

@@ -0,0 +1,212 @@
# Email-Based Bounce Handling Setup (Comprehensive)
This document explains the complete email-based bounce handling system implemented as an alternative to SNS webhooks for environments without SES production access.
## Overview
The system processes email bounces directly within the Postfix container by:
1. Rewriting return paths to direct bounces to a processing address
2. Processing bounce emails via Python script
3. Updating member bounce statistics in MySQL database
4. Automatically disabling members with excessive bounces
## Configuration
### Environment Variables (in .env)
```bash
# Bounce handling feature flags
ENABLE_BOUNCE_HANDLING=true # Master switch for bounce functionality
ENABLE_EMAIL_BOUNCE_PROCESSING=true # Enable email-based processing
ENABLE_SNS_WEBHOOKS=false # Disable SNS webhooks (optional)
# Database settings (required for bounce processing)
MYSQL_ROOT_PASSWORD=your_root_password
MYSQL_DATABASE=maillist
MYSQL_USER=maillist
MYSQL_PASSWORD=your_maillist_password
```
### Key Components
#### 1. Return Path Rewriting (`postfix/smtp_generic`)
Routes all bounces to the processing address:
```
@lists.sasalliance.org bounces@lists.sasalliance.org
@sasalliance.org bounces@lists.sasalliance.org
```
#### 2. Bounce Processing Script (`postfix/process-bounce.py`)
Python script that:
- Parses bounce emails for recipient addresses and bounce types
- Updates bounce counts in MySQL database
- Automatically disables members after 5 hard bounces
- Logs all bounce events
#### 3. Postfix Integration (`postfix/main.cf.template`)
```
# Return path rewriting for outbound mail
smtp_generic_maps = hash:/etc/postfix/smtp_generic
# Bounce processing
bounce_notice_recipient = bounces@lists.sasalliance.org
```
#### 4. Email Aliases (`postfix/entrypoint.sh`)
```
# Route bounces to processing script
bounces: "|/usr/local/bin/python3 /etc/postfix/process-bounce.py"
```
## How It Works
### Outbound Email Flow
1. User sends email to `community@lists.sasalliance.org`
2. Postfix expands to member list via MySQL
3. Email sent via SES with return path rewritten to `bounces@lists.sasalliance.org`
4. If delivery fails, bounce goes to bounce processing address
### Bounce Processing Flow
1. Bounce email arrives at `bounces@lists.sasalliance.org`
2. Postfix pipes email to `process-bounce.py` script
3. Script parses bounce for recipient and bounce type
4. Database updated with bounce information
5. Member automatically disabled if hard bounce threshold reached
### Bounce Types Detected
- **Hard Bounces**: Permanent failures (5.x.x SMTP codes)
- Invalid email addresses
- Domain doesn't exist
- Mailbox doesn't exist
- **Soft Bounces**: Temporary failures (4.x.x SMTP codes)
- Mailbox full
- Temporary server issues
- Rate limiting
## Database Schema
### Bounce Logs Table
```sql
CREATE TABLE bounce_logs (
id INT PRIMARY KEY AUTO_INCREMENT,
email VARCHAR(255) NOT NULL,
bounce_type ENUM('hard', 'soft', 'complaint') NOT NULL,
bounce_reason TEXT,
bounced_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
raw_message TEXT,
INDEX idx_email (email),
INDEX idx_bounced_at (bounced_at)
);
```
### Members Table (bounce tracking fields)
```sql
ALTER TABLE members ADD COLUMN bounce_count INT DEFAULT 0;
ALTER TABLE members ADD COLUMN last_bounce_at TIMESTAMP NULL;
ALTER TABLE members ADD COLUMN active BOOLEAN DEFAULT true;
```
## Testing Bounce Handling
### 1. Test Return Path Configuration
```bash
# Check that return path rewriting is working
sudo docker-compose exec postfix postconf smtp_generic_maps
sudo docker-compose exec postfix postmap -q "test@lists.sasalliance.org" hash:/etc/postfix/smtp_generic
```
### 2. Simulate Bounce Email
```bash
# Send test bounce to processing script
echo "Subject: Delivery Status Notification (Failure)
From: MAILER-DAEMON@ses.amazonaws.com
To: bounces@lists.sasalliance.org
The following message could not be delivered:
Recipient: test@example.com
Reason: 550 5.1.1 User unknown" | docker-compose exec -T postfix mail -s "Test Bounce" bounces@lists.sasalliance.org
```
### 3. Check Bounce Processing
```bash
# View bounce logs
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY bounced_at DESC LIMIT 5;"
# Check member bounce counts
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT email, bounce_count, last_bounce_at, active FROM members WHERE bounce_count > 0;"
```
## Monitoring and Maintenance
### View Processing Logs
```bash
# Monitor bounce processing
sudo docker-compose logs -f postfix | grep -E "(bounce|process-bounce)"
# Check API bounce handling status
curl -H "Authorization: Bearer $API_TOKEN" http://localhost:8000/config
```
### Reset Member Bounce Count
```bash
# Via API
curl -X POST http://localhost:8000/members/{member_id}/reset-bounces \
-H "Authorization: Bearer $API_TOKEN"
# Via Database
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "UPDATE members SET bounce_count=0, active=true WHERE email='user@example.com';"
```
## Troubleshooting
### Common Issues
#### Bounces Not Being Processed
1. Check that `bounces` alias exists: `sudo docker-compose exec postfix cat /etc/aliases | grep bounces`
2. Verify Python script permissions: `sudo docker-compose exec postfix ls -la /etc/postfix/process-bounce.py`
3. Test script manually: `sudo docker-compose exec postfix python3 /etc/postfix/process-bounce.py --test`
#### Return Path Not Rewritten
1. Check smtp_generic configuration: `sudo docker-compose exec postfix postconf smtp_generic_maps`
2. Verify map file exists: `sudo docker-compose exec postfix ls -la /etc/postfix/smtp_generic*`
3. Test mapping: `sudo docker-compose exec postfix postmap -q "test@lists.sasalliance.org" hash:/etc/postfix/smtp_generic`
#### Database Connection Issues
1. Check PyMySQL installation: `sudo docker-compose exec postfix python3 -c "import pymysql; print('OK')"`
2. Test database connection: `sudo docker-compose exec postfix python3 -c "import pymysql; pymysql.connect(host='mysql', user='maillist', password='your_password', database='maillist')"`
3. Verify network connectivity: `sudo docker-compose exec postfix ping mysql`
### Log Analysis
```bash
# Postfix logs
sudo docker-compose logs postfix | grep -E "(bounce|MAILER-DAEMON|process-bounce)"
# MySQL connection logs
sudo docker-compose logs postfix | grep -E "(pymysql|mysql)"
# SES relay logs
sudo docker-compose logs postfix | grep -E "(relay|sent|deferred)"
```
## Security Considerations
- Bounce processing script runs with limited privileges
- Database credentials secured in environment variables
- Bounce emails contain sensitive delivery information - logs are rotated
- Return path rewriting prevents bounce loops
- Processing script validates email format before database updates
## Performance Notes
- Bounce processing is asynchronous (doesn't block email delivery)
- Database queries are indexed for bounce lookups
- Bounce logs should be periodically archived for large volumes
- SMTP generic maps are cached by Postfix for performance
## Advantages over SNS Webhooks
- **Works with SES Sandbox**: No production SES access required
- **No External Dependencies**: Doesn't require SNS, webhooks, or HTTPS domains
- **Self-Contained**: All processing happens within existing containers
- **Real-time Processing**: Bounces processed as emails arrive
- **Compatible**: Uses same database schema and UI as SNS method

View File

@@ -172,6 +172,53 @@ docker-compose exec postfix postmap -q "community@lists.sasalliance.org" \
mysql:/etc/postfix/mysql_virtual_alias_maps.cf
```
### Bounce Handling (Optional)
**Email bounce handling is optional and disabled by default.**
**Three Configuration Options:**
1. **SNS Webhooks** (Requires SES Production Access):
```bash
# In .env file
ENABLE_SNS_WEBHOOKS=true
ENABLE_BOUNCE_HANDLING=true
```
- Real-time bounce notifications via AWS SNS webhooks
- Automatic member deactivation for hard bounces
- Bounce history tracking and management in web UI
- Requires valid HTTPS domain and SES production access
- See `BOUNCE_HANDLING_SETUP.md` for complete setup
2. **Email-Based Processing** (Works with SES Sandbox):
```bash
# In .env file
ENABLE_EMAIL_BOUNCE_PROCESSING=true
ENABLE_BOUNCE_HANDLING=true # Automatically enabled
```
- Automatic bounce processing via email parsing
- Same bounce management features as SNS webhooks
- Works with SES sandbox accounts (no production access needed)
- Self-contained processing within existing containers
- See `EMAIL_BOUNCE_HANDLING_SETUP.md` for complete setup
3. **Disabled** (Manual Processing Only):
```bash
# In .env file (or leave these commented out)
ENABLE_SNS_WEBHOOKS=false
ENABLE_BOUNCE_HANDLING=false
ENABLE_EMAIL_BOUNCE_PROCESSING=false
```
- No automatic bounce processing
- Manual bounce management via email notifications
- Bounce-related UI elements are hidden
**When bounce handling is disabled:**
- `/webhooks/sns` endpoint is not registered
- Bounce history endpoints return empty results
- Web UI hides bounce badges and bounce management buttons
- No automatic member deactivation occurs
## Security
- **Environment Variables**: All credentials stored in `.env` (git-ignored)
@@ -317,6 +364,7 @@ docker-compose up -d # Reinitializes from schema.sql
- ✅ **REST API** - Complete programmatic access with token auth
- ✅ **Sender Whitelist** - Only authorized domains can send to lists
- ✅ **SES Integration** - Reliable email delivery through Amazon SES
- ✅ **Bounce Handling** - Automatic tracking and management of email bounces via SNS
- ✅ **Secure** - Private Docker network, token auth, environment-based credentials
- ✅ **Flexible** - Manage via web, API, or direct database access
- ✅ **Scalable** - Database-driven architecture supports many lists and members
@@ -326,6 +374,7 @@ docker-compose up -d # Reinitializes from schema.sql
- **Web Interface**: See `web/README.md` for frontend features and usage
- **REST API**: See `api/README.md` for complete API reference
- **Database**: See `database/README.md` for schema and SQL examples
- **Bounce Handling**: See `BOUNCE_HANDLING_SETUP.md` for SNS configuration
- **AI Agents**: See `.github/copilot-instructions.md` for development guidance
## Roadmap
@@ -336,6 +385,7 @@ docker-compose up -d # Reinitializes from schema.sql
- [x] Multi-service Docker Compose architecture
- [x] REST API with authentication
- [x] Sender whitelist for authorized domains
- [x] Bounce handling with SES SNS integration
- [ ] Email verification workflow for new members
- [ ] Subscription confirmation (double opt-in)
- [ ] List archive functionality

398
SES_BOUNCE_TESTING_GUIDE.md Normal file
View File

@@ -0,0 +1,398 @@
# Testing Bounces from SES Sandbox
AWS SES provides **built-in bounce simulator addresses** that work even in sandbox mode. This guide shows you how to use them to test your bounce handling.
## Quick Answer
Send email to these special AWS addresses to simulate different bounce types:
### Hard Bounce (Permanent - Invalid Address)
```bash
bounce@simulator.amazonses.com
```
### Soft Bounce (Temporary - Mailbox Full)
```bash
ooto@simulator.amazonses.com
```
### Complaint (Spam Report)
```bash
complaint@simulator.amazonses.com
```
### Successful Delivery (No Bounce)
```bash
success@simulator.amazonses.com
```
## Step-by-Step Testing Guide
### Option 1: Using Your Mailing Lists (Recommended)
This tests the complete flow: Postfix → SES → SNS → API
1. **Add the simulator address as a member:**
```bash
# Using the API
curl -X POST http://localhost:8000/members \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "Bounce Test",
"email": "bounce@simulator.amazonses.com",
"active": true
}'
```
Or use the Web UI:
- Go to http://localhost:3000
- Members tab → Add Member
- Name: `Bounce Test`
- Email: `bounce@simulator.amazonses.com`
2. **Subscribe to a test list:**
- Click "Subscriptions" button for the test member
- Toggle on one of your mailing lists
3. **Send email to the list:**
```bash
# From inside Postfix container
sudo docker compose exec postfix bash
echo "This will bounce" | mail -s "Test Bounce" community@lists.sasalliance.org
exit
```
Replace `community@lists.sasalliance.org` with your actual list email.
4. **Wait 30-60 seconds** for:
- Email to be sent via SES
- SES to process the bounce
- SNS to send notification to your webhook
5. **Check the results:**
**Watch API logs in real-time:**
```bash
sudo docker compose logs api -f
```
You should see:
```
============================================================
SNS Webhook Request Received
============================================================
Content-Type: text/plain; charset=UTF-8
User-Agent: Amazon Simple Notification Service Agent
✓ Notification Received
Notification Type: Bounce
✓ Processing Bounce
Bounce Type: Permanent
Recipients: ['bounce@simulator.amazonses.com']
✓ Bounce processed successfully
============================================================
```
**Check the database:**
```bash
# View bounce logs
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist \
-e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;"
# Check member status
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist \
-e "SELECT email, active, bounce_count, bounce_status FROM members WHERE email='bounce@simulator.amazonses.com';"
```
**View in Web UI:**
- Open http://localhost:3000
- Go to Members tab
- Find "Bounce Test" member
- Should show: ❌ Inactive (red), bounce badge, last bounce timestamp
- Click "Bounces" button to see detailed history
### Option 2: Direct Email via Postfix (Simpler)
Send directly to the simulator without going through your list:
```bash
# Enter Postfix container
sudo docker compose exec postfix bash
# Send test email
echo "Testing hard bounce" | mail -s "Hard Bounce Test" bounce@simulator.amazonses.com
# Or soft bounce
echo "Testing soft bounce" | mail -s "Soft Bounce Test" ooto@simulator.amazonses.com
# Exit container
exit
```
### Option 3: Using AWS SES Console (For Non-Sandbox Testing)
If you have SES production access:
1. Go to AWS SES Console
2. Click "Send test email"
3. To: `bounce@simulator.amazonses.com`
4. From: Your verified email/domain
5. Subject: "Bounce test"
6. Body: "Testing bounce handling"
7. Click "Send test email"
## Testing Different Bounce Types
### 1. Hard Bounce (Permanent Failure)
```bash
echo "Test" | mail -s "Test" bounce@simulator.amazonses.com
```
**Expected Result:**
- Member marked as `hard_bounce`
- Member deactivated (`active = 0`)
- `bounce_count` incremented
- Entry in `bounce_logs` table
### 2. Soft Bounce (Transient Failure)
```bash
echo "Test" | mail -s "Test" ooto@simulator.amazonses.com
```
**Expected Result:**
- Member marked as `clean` (first time)
- After 3 soft bounces → `soft_bounce` status
- `bounce_count` incremented
- Member stays active
### 3. Complaint (Spam Report)
```bash
echo "Test" | mail -s "Test" complaint@simulator.amazonses.com
```
**Expected Result:**
- API receives complaint notification
- Currently logged but not processed (you can extend the handler)
## Monitoring the Test
### Real-Time Monitoring (Recommended)
Open 3 terminal windows:
**Terminal 1 - API Logs:**
```bash
sudo docker compose logs api -f
```
**Terminal 2 - Postfix Logs:**
```bash
sudo docker compose logs postfix -f
```
**Terminal 3 - Send Test Email:**
```bash
sudo docker compose exec postfix bash
echo "Test" | mail -s "Bounce Test" bounce@simulator.amazonses.com
```
### Timeline
Here's what happens and when:
- **T+0s**: Email sent to Postfix
- **T+1-3s**: Postfix relays to SES
- **T+5-10s**: SES processes and generates bounce
- **T+10-30s**: SNS sends notification to your webhook
- **T+30-60s**: API processes bounce and updates database
## Verifying the Complete Flow
### 1. Check Postfix Logs
```bash
sudo docker compose logs postfix | grep bounce@simulator
```
Should show:
```
postfix/smtp[xxx]: ... to=<bounce@simulator.amazonses.com>, relay=email-smtp.eu-west-2.amazonaws.com[...], ... status=sent
```
### 2. Check SNS Subscription Status
- Go to AWS SNS Console
- Find your topic
- Check "Subscriptions" tab
- Status should be "Confirmed"
- Messages delivered should be > 0
### 3. Check API Logs
```bash
sudo docker compose logs api | grep -A 20 "SNS Webhook"
```
Should show successful processing.
### 4. Check Database
```bash
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist <<EOF
-- Show recent bounces
SELECT
b.bounce_id,
m.email,
b.bounce_type,
b.diagnostic_code,
b.timestamp
FROM bounce_logs b
JOIN members m ON b.member_id = m.member_id
ORDER BY b.timestamp DESC
LIMIT 5;
-- Show members with bounces
SELECT
email,
active,
bounce_count,
bounce_status,
last_bounce_at
FROM members
WHERE bounce_count > 0;
EOF
```
## Troubleshooting
### "Email not sent" or "Relay access denied"
**Problem**: Postfix not configured to relay via SES
**Check**:
```bash
sudo docker compose exec postfix postconf relayhost
sudo docker compose exec postfix postconf smtp_sasl_auth_enable
```
Should show:
```
relayhost = [email-smtp.eu-west-2.amazonaws.com]:587
smtp_sasl_auth_enable = yes
```
### "No bounce received after 5 minutes"
**Possible causes**:
1. **SNS subscription not confirmed**
- Check AWS SNS console
- Status should be "Confirmed", not "Pending"
2. **SNS topic not configured in SES**
- Check SES → Configuration Sets or Verified Identities → Notifications
- Bounce notifications should point to your SNS topic
3. **Webhook endpoint not accessible**
- SNS requires HTTPS
- Test: `curl https://your-domain.com:8000/health`
4. **API container not running**
```bash
sudo docker compose ps api
```
### "Bounce received but not in database"
**Check API logs for errors**:
```bash
sudo docker compose logs api | grep -i error
```
**Check database tables exist**:
```bash
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e "SHOW TABLES;"
```
Should include: `bounce_logs`, `members`
## Testing Multiple Bounces
To test the "3 soft bounces = soft_bounce status" logic:
```bash
sudo docker compose exec postfix bash
# Send 3 emails to soft bounce simulator
for i in {1..3}; do
echo "Soft bounce test $i" | mail -s "Test $i" ooto@simulator.amazonses.com
sleep 70 # Wait between sends for SNS processing
done
```
After the 3rd bounce:
- Member's `bounce_status` should change from `clean` to `soft_bounce`
- `bounce_count` should be 3
## Cleanup After Testing
Remove test bounce data:
```bash
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist <<EOF
-- Delete test bounce logs
DELETE FROM bounce_logs WHERE email IN ('bounce@simulator.amazonses.com', 'ooto@simulator.amazonses.com');
-- Reset test member
UPDATE members
SET bounce_count = 0,
bounce_status = 'clean',
last_bounce_at = NULL,
active = 1
WHERE email IN ('bounce@simulator.amazonses.com', 'ooto@simulator.amazonses.com');
EOF
```
Or use the Web UI:
- Go to Members tab
- Find the test member
- Click "Bounces" button
- Click "Reset Bounce Status"
## Alternative: Simulate Bounces Without SES
If SNS isn't set up yet, use the included script to simulate bounces directly in the database:
```bash
./simulate_bounce.sh
```
This is useful for:
- Testing the UI without AWS
- Development environments
- Demonstrating bounce handling to stakeholders
## Next Steps
Once bounce handling is working:
1. **Remove simulator addresses** from your member list
2. **Monitor real bounces** in production
3. **Set up alerts** for high bounce rates
4. **Review bounced members** regularly and update/remove invalid addresses
5. **Consider complaint handling** (similar to bounces, for spam reports)
## Summary Commands
**Quick test sequence**:
```bash
# 1. Watch logs
sudo docker compose logs api -f &
# 2. Send test bounce
echo "Test" | sudo docker compose exec -T postfix mail -s "Bounce Test" bounce@simulator.amazonses.com
# 3. Wait 60 seconds, then check database
sleep 60
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 1;"
```
That's it! The bounce simulator is the easiest way to test your bounce handling without needing real bounced emails.

132
SNS_DEBUG_GUIDE.md Normal file
View File

@@ -0,0 +1,132 @@
# SNS Webhook Debugging Guide
## Current Status
**Detailed logging is now active!** The API will log all incoming SNS requests with:
- Full headers (including SNS-specific headers)
- Request body content
- Parsed message structure
- Processing steps and results
## How to View Logs in Real-Time
```bash
# Watch API logs as SNS sends notifications
sudo docker compose logs api -f
```
## Testing with Real AWS SNS
### Step 1: Trigger a Test from SNS Console
1. Go to AWS SNS Console → Your Topic
2. Click "Publish message"
3. Send a test notification
### Step 2: Check the Logs
The logs will show you exactly what SNS sent:
```
============================================================
SNS Webhook Request Received
============================================================
Content-Type: text/plain; charset=UTF-8
User-Agent: Amazon Simple Notification Service Agent
X-Amz-SNS-Message-Type: Notification
X-Amz-SNS-Topic-Arn: arn:aws:sns:...
Message Type: Notification
Message Keys: ['Type', 'MessageId', 'TopicArn', 'Message', ...]
✓ Notification Received
Message (first 200 chars): {"notificationType":"Bounce",...
Notification Type: Bounce
✓ Processing Bounce
Bounce Type: Permanent
Recipients: ['bounce@example.com']
✓ Bounce processed successfully
============================================================
```
## What the Logs Tell You
### If you see this:
- **"SNS webhook received body: b''"** → SNS sent empty body (check SNS configuration)
- **"Missing SigningCertURL"** → Message missing required SNS fields
- **"Invalid certificate URL"** → Certificate URL not from amazonaws.com
- **"Invalid signature"** → Signature verification failed (message tampered or wrong cert)
### Successful Flow:
1. Headers logged → Shows SNS user agent and message type
2. Body logged → Shows the raw JSON
3. Message parsed → Shows the Type field
4. Notification processed → Shows bounce/complaint details
5. Database updated → Confirmation of processing
## Understanding Previous Errors
The error you saw earlier:
```
SNS webhook error: Expecting value: line 1 column 1 (char 0)
```
This means SNS was sending an **empty body** or the body was already consumed. Possible causes:
1. SNS subscription confirmation URL was opened in a browser (GET request, not POST)
2. Network issue causing body to be lost
3. SNS configuration issue
## Testing Bounce Handling
### Option 1: AWS SES Bounce Simulator
Send email to: `bounce@simulator.amazonses.com`
### Option 2: Verify Database Updates
After a bounce is processed:
```bash
# Check bounce logs
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;"
# Check member bounce status
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT member_id, email, bounce_count, bounce_status, last_bounce_at FROM members WHERE bounce_count > 0;"
```
## Common SNS Message Types
### 1. SubscriptionConfirmation
First message when you create the subscription. API auto-confirms by calling SubscribeURL.
### 2. Notification (Bounce)
```json
{
"Type": "Notification",
"Message": "{\"notificationType\":\"Bounce\",\"bounce\":{...}}"
}
```
### 3. Notification (Complaint)
```json
{
"Type": "Notification",
"Message": "{\"notificationType\":\"Complaint\",\"complaint\":{...}}"
}
```
## Next Steps
1. **Monitor logs while SNS sends**: `sudo docker compose logs api -f`
2. **Trigger a real bounce**: Send to `bounce@simulator.amazonses.com`
3. **Check the detailed logs** to see exactly what SNS is sending
4. **Verify database updates** to confirm bounces are being recorded
## If You Still See Errors
The enhanced logging will now show you:
- What headers SNS is sending
- What body content is arriving (or if it's empty)
- Where in the processing pipeline the error occurs
- Full stack traces for any exceptions
Copy the relevant log section and we can diagnose the exact issue!

View File

@@ -5,6 +5,7 @@ FastAPI-based REST API for managing mailing lists and members
from fastapi import FastAPI, HTTPException, Depends, Header, status, Request
from fastapi.middleware.cors import CORSMiddleware
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from fastapi.responses import PlainTextResponse
from pydantic import BaseModel, EmailStr
from typing import List, Optional, Annotated
import mysql.connector
@@ -19,6 +20,15 @@ import bcrypt
from jose import JWTError, jwt
from passlib.context import CryptContext
from enum import Enum
import json
import base64
from urllib.parse import urlparse
import httpx
from cryptography import x509
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives.serialization import load_pem_public_key
# Configuration
API_TOKEN = os.getenv('API_TOKEN', 'change-this-token') # Keep for backward compatibility during transition
@@ -33,6 +43,14 @@ MYSQL_DATABASE = os.getenv('MYSQL_DATABASE', 'maillist')
MYSQL_USER = os.getenv('MYSQL_USER', 'maillist')
MYSQL_PASSWORD = os.getenv('MYSQL_PASSWORD', '')
# Bounce handling configuration
ENABLE_SNS_WEBHOOKS = os.getenv('ENABLE_SNS_WEBHOOKS', 'false').lower() == 'true'
ENABLE_BOUNCE_HANDLING = os.getenv('ENABLE_BOUNCE_HANDLING', 'false').lower() == 'true'
ENABLE_EMAIL_BOUNCE_PROCESSING = os.getenv('ENABLE_EMAIL_BOUNCE_PROCESSING', 'false').lower() == 'true'
# Enable bounce handling if either method is enabled
ENABLE_BOUNCE_HANDLING = ENABLE_BOUNCE_HANDLING or ENABLE_SNS_WEBHOOKS or ENABLE_EMAIL_BOUNCE_PROCESSING
# Password hashing
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
@@ -44,12 +62,15 @@ app = FastAPI(
)
# Add CORS middleware
# Get allowed origins from environment or use secure defaults
ALLOWED_ORIGINS = os.getenv('ALLOWED_ORIGINS', 'http://localhost:3000,http://127.0.0.1:3000').split(',')
app.add_middleware(
CORSMiddleware,
allow_origins=["*"], # In production, specify your frontend domain
allow_origins=ALLOWED_ORIGINS, # Specific origins only
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
allow_methods=["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"], # Specific methods
allow_headers=["Authorization", "Content-Type", "Accept", "Origin", "X-Requested-With"], # Specific headers
)
security = HTTPBearer()
@@ -229,6 +250,9 @@ class Member(BaseModel):
name: str
email: EmailStr
active: bool = True
bounce_count: Optional[int] = 0
last_bounce_at: Optional[datetime] = None
bounce_status: Optional[str] = 'clean'
class MemberUpdate(BaseModel):
name: Optional[str] = None
@@ -447,6 +471,15 @@ async def health():
except Exception as e:
raise HTTPException(status_code=503, detail=f"Unhealthy: {str(e)}")
@app.get("/config")
async def get_config():
"""Get public configuration settings"""
return {
"bounce_handling_enabled": ENABLE_BOUNCE_HANDLING,
"sns_webhooks_enabled": ENABLE_SNS_WEBHOOKS,
"email_bounce_processing_enabled": ENABLE_EMAIL_BOUNCE_PROCESSING
}
# Mailing Lists endpoints
@app.get("/lists", response_model=List[MailingList])
async def get_lists(current_user: CurrentUser = require_read_access()):
@@ -808,6 +841,317 @@ async def bulk_import_members(bulk_request: BulkImportRequest, current_user: Cur
cursor.close()
raise HTTPException(status_code=500, detail=f"Bulk import failed: {str(e)}")
# SNS Webhook for Bounce Handling (conditionally enabled)
if ENABLE_SNS_WEBHOOKS:
async def verify_sns_signature(request: Request) -> dict:
"""Verify SNS message signature"""
try:
body = await request.body()
print(f"SNS webhook received body: {body}")
print(f"SNS webhook body length: {len(body)}")
print(f"SNS webhook headers: {dict(request.headers)}")
if not body:
print("ERROR: Empty body received")
raise HTTPException(status_code=400, detail="Empty request body")
message = json.loads(body.decode('utf-8'))
print(f"SNS webhook parsed message type: {message.get('Type')}")
# For SubscriptionConfirmation and UnsubscribeConfirmation, we don't validate signature
# AWS will send a URL to confirm
if message.get('Type') in ['SubscriptionConfirmation', 'UnsubscribeConfirmation']:
return message
# Get certificate URL and download certificate
cert_url = message.get('SigningCertURL')
if not cert_url:
raise HTTPException(status_code=400, detail="Missing SigningCertURL")
# Verify certificate URL is from AWS
parsed_url = urlparse(cert_url)
if not parsed_url.hostname.endswith('.amazonaws.com'):
raise HTTPException(status_code=400, detail="Invalid certificate URL")
# Download certificate
async with httpx.AsyncClient() as client:
cert_response = await client.get(cert_url)
cert_response.raise_for_status()
cert_pem = cert_response.content
# Load certificate and extract public key
cert = x509.load_pem_x509_certificate(cert_pem, default_backend())
public_key = cert.public_key()
# Build string to sign
if message.get('Type') == 'Notification':
string_to_sign = (
f"Message\n{message['Message']}\n"
f"MessageId\n{message['MessageId']}\n"
)
if 'Subject' in message:
string_to_sign += f"Subject\n{message['Subject']}\n"
string_to_sign += (
f"Timestamp\n{message['Timestamp']}\n"
f"TopicArn\n{message['TopicArn']}\n"
f"Type\n{message['Type']}\n"
)
else:
string_to_sign = (
f"Message\n{message['Message']}\n"
f"MessageId\n{message['MessageId']}\n"
f"SubscribeURL\n{message['SubscribeURL']}\n"
f"Timestamp\n{message['Timestamp']}\n"
f"Token\n{message['Token']}\n"
f"TopicArn\n{message['TopicArn']}\n"
f"Type\n{message['Type']}\n"
)
# Verify signature
signature = base64.b64decode(message['Signature'])
try:
public_key.verify(
signature,
string_to_sign.encode('utf-8'),
padding.PKCS1v15(),
hashes.SHA1()
)
except Exception as e:
raise HTTPException(status_code=400, detail=f"Invalid signature: {str(e)}")
return message
except json.JSONDecodeError:
raise HTTPException(status_code=400, detail="Invalid JSON")
except Exception as e:
raise HTTPException(status_code=400, detail=f"Signature verification failed: {str(e)}")
async def process_bounce(bounce_data: dict):
"""Process bounce notification and update database"""
try:
bounce_type = bounce_data.get('bounceType') # Permanent, Transient, Undetermined
bounce_subtype = bounce_data.get('bounceSubType', '')
timestamp_str = bounce_data.get('timestamp')
feedback_id = bounce_data.get('feedbackId', '')
# Convert ISO 8601 timestamp to MySQL datetime format
# SES sends: '2025-10-13T16:22:40.359Z'
# MySQL needs: '2025-10-13 16:22:40'
from datetime import datetime as dt
timestamp = dt.fromisoformat(timestamp_str.replace('Z', '+00:00')).strftime('%Y-%m-%d %H:%M:%S')
bounced_recipients = bounce_data.get('bouncedRecipients', [])
with get_db() as conn:
cursor = conn.cursor(dictionary=True)
for recipient in bounced_recipients:
email = recipient.get('emailAddress')
diagnostic_code = recipient.get('diagnosticCode', '')
if not email:
continue
# Find member by email
cursor.execute("SELECT member_id FROM members WHERE email = %s", (email,))
member = cursor.fetchone()
member_id = member['member_id'] if member else None
# Log the bounce
cursor.execute("""
INSERT INTO bounce_logs
(member_id, email, bounce_type, bounce_subtype, diagnostic_code, timestamp, feedback_id)
VALUES (%s, %s, %s, %s, %s, %s, %s)
""", (member_id, email, bounce_type, bounce_subtype, diagnostic_code, timestamp, feedback_id))
# Update member bounce status
if member_id:
# Determine bounce status
if bounce_type == 'Permanent':
new_status = 'hard_bounce'
# Deactivate member with hard bounce
cursor.execute("""
UPDATE members
SET bounce_count = bounce_count + 1,
last_bounce_at = %s,
bounce_status = %s,
active = 0
WHERE member_id = %s
""", (timestamp, new_status, member_id))
elif bounce_type == 'Transient':
# Check current bounce count
cursor.execute("SELECT bounce_count, bounce_status FROM members WHERE member_id = %s", (member_id,))
current = cursor.fetchone()
# If already hard bounced, don't change status
if current and current['bounce_status'] != 'hard_bounce':
new_count = current['bounce_count'] + 1
# After 3 soft bounces, mark as soft_bounce status
new_status = 'soft_bounce' if new_count >= 3 else 'clean'
cursor.execute("""
UPDATE members
SET bounce_count = %s,
last_bounce_at = %s,
bounce_status = %s
WHERE member_id = %s
""", (new_count, timestamp, new_status, member_id))
else: # Undetermined
cursor.execute("""
UPDATE members
SET bounce_count = bounce_count + 1,
last_bounce_at = %s
WHERE member_id = %s
""", (timestamp, member_id))
conn.commit()
cursor.close()
except Exception as e:
print(f"✗ Error processing bounce: {str(e)}")
print(f"Error type: {type(e).__name__}")
print(f"Bounce data: {bounce_data}")
import traceback
traceback.print_exc()
raise
@app.post("/webhooks/sns", response_class=PlainTextResponse)
async def sns_webhook(request: Request):
"""Handle SNS notifications for bounces and complaints"""
try:
print(f"=== SNS Webhook Request ===")
print(f"Headers: {dict(request.headers)}")
print(f"Content-Type: {request.headers.get('content-type')}")
print(f"User-Agent: {request.headers.get('user-agent')}")
# Verify SNS signature
message = await verify_sns_signature(request)
print(f"Message Type: {message.get('Type')}")
print(f"Message Keys: {list(message.keys())}")
message_type = message.get('Type')
# Handle subscription confirmation
if message_type == 'SubscriptionConfirmation':
subscribe_url = message.get('SubscribeURL')
print(f"Subscription confirmation received, URL: {subscribe_url}")
if subscribe_url:
# Confirm subscription
async with httpx.AsyncClient() as client:
response = await client.get(subscribe_url)
print(f"Subscription confirmation response: {response.status_code}")
return "Subscription confirmed"
# Handle notification
elif message_type == 'Notification':
# Parse the message
inner_message = message.get('Message', '{}')
print(f"Inner message (first 500 chars): {inner_message[:500]}")
notification = json.loads(inner_message)
# SES can send either 'notificationType' or 'eventType' depending on configuration
notification_type = notification.get('notificationType') or notification.get('eventType')
print(f"Notification type: {notification_type}")
if notification_type == 'Bounce':
bounce = notification.get('bounce', {})
print(f"\n✓ Processing Bounce")
print(f" Bounce Type: {bounce.get('bounceType')}")
print(f" Recipients: {[r.get('emailAddress') for r in bounce.get('bouncedRecipients', [])]}")
await process_bounce(bounce)
print(f" ✓ Bounce processed successfully")
return "Bounce processed"
elif notification_type == 'Complaint':
# We could also track complaints similarly to bounces
print(f"\n✓ Complaint received")
return "Complaint received"
print(f"=== End SNS Webhook Request ===")
return "OK"
except HTTPException:
raise
except Exception as e:
print(f"SNS webhook error: {str(e)}")
import traceback
traceback.print_exc()
raise HTTPException(status_code=500, detail=str(e))
else:
# Provide stub functions when SNS webhooks are disabled
print("SNS webhooks disabled - bounce handling via email only")
# Bounce management endpoints (conditionally enabled)
if ENABLE_BOUNCE_HANDLING:
class BounceLog(BaseModel):
bounce_id: int
email: str
bounce_type: str
bounce_subtype: Optional[str] = None
diagnostic_code: Optional[str] = None
timestamp: datetime
feedback_id: Optional[str] = None
created_at: datetime
class MemberWithBounces(BaseModel):
member_id: int
name: str
email: str
active: bool
bounce_count: int
last_bounce_at: Optional[datetime] = None
bounce_status: str
@app.get("/members/{member_id}/bounces", response_model=List[BounceLog])
async def get_member_bounces(member_id: int, current_user: CurrentUser = require_read_access()):
"""Get bounce history for a member"""
with get_db() as conn:
cursor = conn.cursor(dictionary=True)
cursor.execute("""
SELECT bounce_id, email, bounce_type, bounce_subtype, diagnostic_code,
timestamp, feedback_id, created_at
FROM bounce_logs
WHERE member_id = %s
ORDER BY timestamp DESC
""", (member_id,))
bounces = cursor.fetchall()
cursor.close()
return bounces
@app.patch("/members/{member_id}/bounce-status")
async def reset_bounce_status(member_id: int, current_user: CurrentUser = require_write_access()):
"""Reset bounce status for a member (e.g., after email address is corrected)"""
with get_db() as conn:
cursor = conn.cursor()
cursor.execute("""
UPDATE members
SET bounce_count = 0,
last_bounce_at = NULL,
bounce_status = 'clean'
WHERE member_id = %s
""", (member_id,))
conn.commit()
if cursor.rowcount == 0:
raise HTTPException(status_code=404, detail="Member not found")
cursor.close()
return {"message": "Bounce status reset successfully"}
else:
# When bounce handling is disabled, provide stub endpoints that return appropriate responses
@app.get("/members/{member_id}/bounces")
async def get_member_bounces_disabled(member_id: int, current_user: CurrentUser = require_read_access()):
"""Bounce history disabled - returns empty list"""
return []
@app.patch("/members/{member_id}/bounce-status")
async def reset_bounce_status_disabled(member_id: int, current_user: CurrentUser = require_write_access()):
"""Bounce status reset disabled"""
raise HTTPException(status_code=501, detail="Bounce handling is disabled")
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)

View File

@@ -8,3 +8,4 @@ email-validator==2.1.0
bcrypt==4.0.1
python-jose[cryptography]==3.3.0
passlib[bcrypt]==1.7.4
httpx==0.25.2

View File

@@ -55,8 +55,12 @@ CREATE TABLE IF NOT EXISTS members (
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
active BOOLEAN DEFAULT TRUE,
bounce_count INT DEFAULT 0,
last_bounce_at TIMESTAMP NULL,
bounce_status ENUM('clean', 'soft_bounce', 'hard_bounce') DEFAULT 'clean',
INDEX idx_email (email),
INDEX idx_active (active)
INDEX idx_active (active),
INDEX idx_bounce_status (bounce_status)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- Table: list_members
@@ -75,6 +79,26 @@ CREATE TABLE IF NOT EXISTS list_members (
INDEX idx_active (active)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- Table: bounce_logs
-- Stores bounce notifications from SES SNS
CREATE TABLE IF NOT EXISTS bounce_logs (
bounce_id INT AUTO_INCREMENT PRIMARY KEY,
member_id INT,
email VARCHAR(255) NOT NULL,
bounce_type ENUM('Permanent', 'Transient', 'Undetermined') NOT NULL,
bounce_subtype VARCHAR(50),
diagnostic_code TEXT,
timestamp TIMESTAMP NOT NULL,
sns_message_id VARCHAR(255),
feedback_id VARCHAR(255),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (member_id) REFERENCES members(member_id) ON DELETE SET NULL,
INDEX idx_member_id (member_id),
INDEX idx_email (email),
INDEX idx_timestamp (timestamp),
INDEX idx_bounce_type (bounce_type)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- Insert sample data
-- Create default admin user (password: 'password')

View File

@@ -0,0 +1,59 @@
# Example nginx configuration for production with SNS webhook support
server {
listen 80;
server_name yourdomain.com;
# Redirect HTTP to HTTPS in production
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name yourdomain.com;
# SSL configuration (add your certificates)
# ssl_certificate /path/to/your/cert.pem;
# ssl_certificate_key /path/to/your/key.pem;
# Frontend (static files)
location / {
proxy_pass http://maillist-web:80;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# API endpoints (including SNS webhook)
location /api/ {
# Remove /api prefix when forwarding to backend
rewrite ^/api/(.*) /$1 break;
proxy_pass http://maillist-api:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Important for SNS webhooks - increase timeouts and body size
proxy_connect_timeout 60s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
client_max_body_size 10M;
}
# Direct SNS webhook endpoint (alternative path)
location /webhooks/sns {
proxy_pass http://maillist-api:8000/webhooks/sns;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# SNS specific settings
proxy_connect_timeout 30s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;
client_max_body_size 1M;
}
}

View File

@@ -9,6 +9,8 @@ RUN apt-get update && \
mailutils \
gettext-base \
netcat-openbsd \
python3 \
python3-pymysql \
&& apt-get clean && rm -rf /var/lib/apt/lists/*
# Copy configs
@@ -16,11 +18,16 @@ COPY main.cf.template /etc/postfix/main.cf.template
COPY sasl_passwd.template /etc/postfix/sasl_passwd.template
COPY mysql_virtual_alias_maps.cf /etc/postfix/mysql_virtual_alias_maps.cf.template
COPY sender_access /etc/postfix/sender_access
COPY smtp_generic /etc/postfix/smtp_generic
COPY aliases /etc/aliases
COPY process-bounce.py /usr/local/bin/process-bounce.py
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
RUN chmod +x /entrypoint.sh /usr/local/bin/process-bounce.py
# Generate Postfix maps for sender access
RUN postmap /etc/postfix/sender_access
# Generate Postfix maps for sender access, sender canonical, and aliases
RUN postmap /etc/postfix/sender_access && \
postmap /etc/postfix/smtp_generic && \
newaliases
# Expose SMTP
EXPOSE 25

17
postfix/aliases Normal file
View File

@@ -0,0 +1,17 @@
# Postfix aliases for bounce handling
#
# This file defines how bounces are processed
# The bounces address pipes messages to our bounce processing script
# Bounce processing - pipe to Python script
bounces: "|/usr/bin/python3 /usr/local/bin/process-bounce.py"
# Standard aliases
postmaster: root
mailer-daemon: postmaster
abuse: postmaster
spam: postmaster
root: postmaster
# Default fallback
MAILER-DAEMON: bounces

View File

@@ -32,5 +32,28 @@ chmod 644 /etc/postfix/sender_access /etc/postfix/sender_access.db
# Set permissions on MySQL config
chmod 644 /etc/postfix/mysql_virtual_alias_maps.cf
# Configure bounce processing based on environment variable
if [ "${ENABLE_EMAIL_BOUNCE_PROCESSING:-false}" = "true" ]; then
echo "Email-based bounce processing enabled"
# Regenerate aliases database to enable bounce processing
newaliases
chmod 644 /etc/aliases /etc/aliases.db
# Generate smtp generic maps to ensure bounces come to our bounce address
postmap /etc/postfix/smtp_generic
chmod 644 /etc/postfix/smtp_generic /etc/postfix/smtp_generic.db
echo "Configured return path rewriting to bounces@lists.sasalliance.org"
else
echo "Email-based bounce processing disabled"
# Create minimal aliases without bounce processing
echo "postmaster: root" > /etc/aliases
echo "root: postmaster" >> /etc/aliases
newaliases
# Disable smtp generic maps
echo "# Email bounce processing disabled" > /etc/postfix/smtp_generic
postmap /etc/postfix/smtp_generic
fi
# Start Postfix in foreground
exec postfix start-fg

View File

@@ -33,3 +33,23 @@ smtpd_recipient_restrictions =
# Other recommended settings
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
# Bounce handling configuration for email-based processing
# Configure bounce notification recipients
bounce_notice_recipient = bounces@lists.sasalliance.org
2bounce_notice_recipient = bounces@lists.sasalliance.org
delay_notice_recipient =
error_notice_recipient = bounces@lists.sasalliance.org
# Bounce settings
bounce_size_limit = 50000
bounce_queue_lifetime = 5d
maximal_bounce_delay = 1d
# Return path configuration - CRITICAL for bounce handling
# This ensures that when we relay emails via mailing lists through SES,
# bounces come back to our bounce processing address
#
# Use smtp_generic_maps instead of sender_canonical_maps because we only want
# to rewrite the return path for outbound SMTP (via SES), not for local delivery
smtp_generic_maps = hash:/etc/postfix/smtp_generic

348
postfix/process-bounce.py Normal file
View File

@@ -0,0 +1,348 @@
#!/usr/bin/env python3
"""
Email-based bounce processing script for Postfix
Parses bounce emails and updates the database with bounce information
This script is called by Postfix when bounce emails are received.
It reads the email from stdin, parses it for bounce information,
and updates the database accordingly.
"""
import sys
import os
import re
import email
from email.message import Message
import logging
from datetime import datetime
from typing import List, Tuple, Optional, Dict
import pymysql
# Configure logging
log_file = '/var/log/bounce-processor.log'
log_handlers = [logging.StreamHandler()]
# Add file handler if we can write to the log directory
try:
log_handlers.append(logging.FileHandler(log_file))
except (PermissionError, FileNotFoundError):
pass # Just use stdout if we can't write to log file
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
handlers=log_handlers
)
logger = logging.getLogger('bounce-processor')
# Database configuration from environment
MYSQL_HOST = os.getenv('MYSQL_HOST', 'mysql')
MYSQL_PORT = int(os.getenv('MYSQL_PORT', 3306))
MYSQL_DATABASE = os.getenv('MYSQL_DATABASE', 'maillist')
MYSQL_USER = os.getenv('MYSQL_USER', 'maillist')
MYSQL_PASSWORD = os.getenv('MYSQL_PASSWORD', '')
class BounceProcessor:
"""Processes bounce emails and updates the database"""
def __init__(self):
self.bounce_patterns = self._compile_bounce_patterns()
def _compile_bounce_patterns(self) -> List[re.Pattern]:
"""Compile regex patterns for detecting bounce information"""
patterns = [
# Standard bounce formats
re.compile(r'550.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
re.compile(r'554.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
re.compile(r'553.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
re.compile(r'552.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
re.compile(r'551.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
# Delivery Status Notification (DSN) format
re.compile(r'Final-Recipient:.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
re.compile(r'Original-Recipient:.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
# Common bounce message patterns
re.compile(r'user.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?unknown', re.IGNORECASE),
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?does not exist', re.IGNORECASE),
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?not found', re.IGNORECASE),
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?mailbox.*?unavailable', re.IGNORECASE),
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?recipient.*?rejected', re.IGNORECASE),
# Generic email extraction (fallback)
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
]
return patterns
def parse_bounce_email(self, email_content: str) -> List[Dict]:
"""Parse bounce email and extract bounce information"""
try:
# Parse the email
msg = email.message_from_string(email_content)
bounces = []
# Get the email body
body = self._get_email_body(msg)
if not body:
logger.warning("No email body found")
return bounces
logger.info(f"Processing email body (first 500 chars): {body[:500]}")
# Extract bounced addresses
bounced_addresses = self._extract_bounced_addresses(body)
# Determine bounce type and create bounce records
for address in bounced_addresses:
bounce_info = self._analyze_bounce(body, address)
if bounce_info:
bounces.append(bounce_info)
return bounces
except Exception as e:
logger.error(f"Error parsing bounce email: {str(e)}")
return []
def _get_email_body(self, msg: Message) -> Optional[str]:
"""Extract the email body from the message"""
body = ""
if msg.is_multipart():
for part in msg.walk():
content_type = part.get_content_type()
if content_type in ['text/plain', 'text/html']:
payload = part.get_payload(decode=True)
if payload:
body += payload.decode('utf-8', errors='ignore') + "\n"
else:
payload = msg.get_payload(decode=True)
if payload:
body = payload.decode('utf-8', errors='ignore')
return body.strip() if body else None
def _extract_bounced_addresses(self, body: str) -> List[str]:
"""Extract bounced email addresses from the bounce message"""
addresses = []
for pattern in self.bounce_patterns:
matches = pattern.findall(body)
for match in matches:
email_addr = match.strip().lower()
if self._is_valid_email(email_addr) and email_addr not in addresses:
# Skip our own addresses
if not email_addr.endswith('@lists.sasalliance.org'):
addresses.append(email_addr)
logger.info(f"Extracted addresses: {addresses}")
return addresses
def _is_valid_email(self, email_addr: str) -> bool:
"""Validate email address format"""
email_pattern = re.compile(r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$')
return bool(email_pattern.match(email_addr))
def _analyze_bounce(self, body: str, email_addr: str) -> Optional[Dict]:
"""Analyze bounce message to determine bounce type and details"""
bounce_info = {
'email': email_addr,
'bounce_type': 'Undetermined',
'bounce_subtype': '',
'diagnostic_code': '',
'timestamp': datetime.now().strftime('%Y-%m-%d %H:%M:%S')
}
# Analyze bounce type based on SMTP codes and message content
if re.search(r'5[0-9]{2}', body): # 5xx codes are permanent failures
bounce_info['bounce_type'] = 'Permanent'
if re.search(r'550|551|553', body):
bounce_info['bounce_subtype'] = 'General'
elif re.search(r'552', body):
bounce_info['bounce_subtype'] = 'MailboxFull'
elif re.search(r'554', body):
bounce_info['bounce_subtype'] = 'MessageTooLarge'
elif re.search(r'4[0-9]{2}', body): # 4xx codes are temporary failures
bounce_info['bounce_type'] = 'Transient'
if re.search(r'450|451|452', body):
bounce_info['bounce_subtype'] = 'General'
elif re.search(r'452', body):
bounce_info['bounce_subtype'] = 'MailboxFull'
# Extract diagnostic code
smtp_code_match = re.search(r'([45][0-9]{2}.*?)[\r\n]', body)
if smtp_code_match:
bounce_info['diagnostic_code'] = smtp_code_match.group(1).strip()[:500] # Limit length
return bounce_info
def update_database(self, bounces: List[Dict]) -> None:
"""Update the database with bounce information"""
if not bounces:
logger.info("No bounces to process")
return
try:
connection = pymysql.connect(
host=MYSQL_HOST,
port=MYSQL_PORT,
database=MYSQL_DATABASE,
user=MYSQL_USER,
password=MYSQL_PASSWORD,
cursorclass=pymysql.cursors.DictCursor
)
cursor = connection.cursor()
for bounce in bounces:
try:
email_addr = bounce['email']
# Find member by email
cursor.execute("SELECT member_id FROM members WHERE email = %s", (email_addr,))
member = cursor.fetchone()
member_id = member['member_id'] if member else None
# Log the bounce
cursor.execute("""
INSERT INTO bounce_logs
(member_id, email, bounce_type, bounce_subtype, diagnostic_code, timestamp, created_at)
VALUES (%s, %s, %s, %s, %s, %s, NOW())
""", (member_id, email_addr, bounce['bounce_type'], bounce['bounce_subtype'],
bounce['diagnostic_code'], bounce['timestamp']))
logger.info(f"Logged bounce for {email_addr}: {bounce['bounce_type']}")
# Update member bounce status if member exists
if member_id:
if bounce['bounce_type'] == 'Permanent':
# Hard bounce - deactivate member
cursor.execute("""
UPDATE members
SET bounce_count = bounce_count + 1,
last_bounce_at = %s,
bounce_status = 'hard_bounce',
active = 0
WHERE member_id = %s
""", (bounce['timestamp'], member_id))
logger.info(f"Deactivated member {email_addr} due to hard bounce")
elif bounce['bounce_type'] == 'Transient':
# Soft bounce - increment counter and check threshold
cursor.execute("SELECT bounce_count, bounce_status FROM members WHERE member_id = %s", (member_id,))
current = cursor.fetchone()
if current and current['bounce_status'] != 'hard_bounce':
new_count = current['bounce_count'] + 1
new_status = 'soft_bounce' if new_count >= 3 else 'clean'
cursor.execute("""
UPDATE members
SET bounce_count = %s,
last_bounce_at = %s,
bounce_status = %s
WHERE member_id = %s
""", (new_count, bounce['timestamp'], new_status, member_id))
logger.info(f"Updated member {email_addr} bounce count to {new_count}, status: {new_status}")
else:
# Undetermined - just increment counter
cursor.execute("""
UPDATE members
SET bounce_count = bounce_count + 1,
last_bounce_at = %s
WHERE member_id = %s
""", (bounce['timestamp'], member_id))
logger.info(f"Updated member {email_addr} bounce count (undetermined)")
except Exception as e:
logger.error(f"Database error processing bounce for {bounce['email']}: {str(e)}")
continue
connection.commit()
cursor.close()
connection.close()
logger.info(f"Successfully processed {len(bounces)} bounces")
except pymysql.Error as e:
logger.error(f"Database connection error: {str(e)}")
except Exception as e:
logger.error(f"Unexpected error updating database: {str(e)}")
def main():
"""Main function - reads email from stdin and processes bounces"""
try:
logger.info("Starting bounce processing")
# Check if we're in test mode
test_mode = len(sys.argv) > 1 and sys.argv[1] == '--test'
if test_mode:
# Test mode - use sample bounce email
email_content = """From: Mail Delivery Subsystem <MAILER-DAEMON@example.com>
To: bounces@lists.sasalliance.org
Subject: Delivery Status Notification (Failure)
Content-Type: multipart/report; report-type=delivery-status; boundary="boundary123"
--boundary123
Content-Type: text/plain
This is a test bounce message.
The following address(es) failed:
testuser@example.com
SMTP error from remote mail server after RCPT TO:<testuser@example.com>:
550 5.1.1 User unknown
--boundary123
Content-Type: message/delivery-status
Reporting-MTA: dns; mail.example.com
Received-From-MTA: dns; localhost
Final-Recipient: rfc822; testuser@example.com
Action: failed
Status: 5.1.1
Diagnostic-Code: smtp; 550 5.1.1 User unknown
--boundary123--
"""
logger.info("Running in test mode with sample bounce email")
else:
# Read email from stdin
email_content = sys.stdin.read()
if not email_content.strip():
logger.warning("No email content received")
return
logger.info(f"Received email content ({len(email_content)} bytes)")
# Process the bounce
processor = BounceProcessor()
bounces = processor.parse_bounce_email(email_content)
if test_mode:
logger.info(f"Test mode - would process {len(bounces)} bounce(s):")
for bounce in bounces:
logger.info(f" {bounce}")
else:
if bounces:
processor.update_database(bounces)
logger.info(f"Processed {len(bounces)} bounce(s)")
else:
logger.info("No bounces detected in email")
except Exception as e:
logger.error(f"Error in main: {str(e)}")
import traceback
logger.error(f"Traceback: {traceback.format_exc()}")
sys.exit(1)
if __name__ == "__main__":
main()

23
postfix/smtp_generic Normal file
View File

@@ -0,0 +1,23 @@
# Postfix SMTP Generic Maps
# This file controls the return path (envelope sender) for outbound SMTP emails
# Only applies to emails being relayed through external SMTP (SES in our case)
#
# Format: original_sender rewritten_sender
#
# For mailing list emails, we want bounces to go to our bounce processing address
# This is critical for email-based bounce handling to work properly
# Rewrite envelope sender for all mailing list addresses to bounce address
# When emails are forwarded through mailing lists via SES, bounces come back to us
community@lists.sasalliance.org bounces@lists.sasalliance.org
board@lists.sasalliance.org bounces@lists.sasalliance.org
members@lists.sasalliance.org bounces@lists.sasalliance.org
announcements@lists.sasalliance.org bounces@lists.sasalliance.org
# Generic pattern - any @lists.sasalliance.org sender gets rewritten to bounce address
# This catches any new lists automatically
@lists.sasalliance.org bounces@lists.sasalliance.org
# Also rewrite any envelope sender that's sending through our system
# This ensures ALL outgoing mail via SES has our bounce address as return path
@sasalliance.org bounces@lists.sasalliance.org

View File

@@ -1,11 +0,0 @@
# Community mailing list - general announcements
community@lists.sasalliance.org james@pattinson.org, james.pattinson@sasalliance.org
# Board members mailing list
board@lists.sasalliance.org james.pattinson@sasalliance.org
# All members mailing list
members@lists.sasalliance.org james@pattinson.org, james.pattinson@sasalliance.org
# Announcements mailing list
announcements@lists.sasalliance.org james@pattinson.org, james.pattinson@sasalliance.org

206
simulate_bounce.sh Executable file
View File

@@ -0,0 +1,206 @@
#!/bin/bash
# Simulate Bounce Events for Testing
# This script inserts bounce data directly into the database to test the UI
# without needing to set up AWS SNS
set -e
echo "=== Bounce Simulation Script ==="
echo
echo "This script will create test bounce events for existing members"
echo
# Get database password from .env
DB_PASSWORD=$(grep MYSQL_ROOT_PASSWORD /home/jamesp/docker/maillist/.env | cut -d'=' -f2)
# Check if containers are running
if ! sudo docker-compose ps | grep -q "maillist-mysql.*Up"; then
echo "❌ MySQL container is not running. Starting containers..."
sudo docker-compose up -d
echo "Waiting for MySQL to be ready..."
sleep 5
fi
echo "1. Fetching existing members..."
echo
# Get list of members
MEMBERS=$(sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -N -e "SELECT member_id, name, email FROM members LIMIT 5;")
if [ -z "$MEMBERS" ]; then
echo "❌ No members found in database. Please add members first."
exit 1
fi
echo "Available members:"
echo "$MEMBERS" | while read -r line; do
MEMBER_ID=$(echo "$line" | awk '{print $1}')
NAME=$(echo "$line" | awk '{print $2, $3}')
EMAIL=$(echo "$line" | awk '{$1=""; $2=""; print $0}' | xargs)
echo " [$MEMBER_ID] $NAME - $EMAIL"
done
echo
# Prompt for member ID
read -p "Enter member ID to simulate bounce for (or press Enter for first member): " MEMBER_ID
if [ -z "$MEMBER_ID" ]; then
MEMBER_ID=$(echo "$MEMBERS" | head -1 | awk '{print $1}')
echo "Using member ID: $MEMBER_ID"
fi
# Get member details
MEMBER_INFO=$(sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -N -e "SELECT member_id, name, email FROM members WHERE member_id = $MEMBER_ID;")
if [ -z "$MEMBER_INFO" ]; then
echo "❌ Member ID $MEMBER_ID not found"
exit 1
fi
MEMBER_EMAIL=$(echo "$MEMBER_INFO" | awk '{$1=""; $2=""; print $0}' | xargs)
MEMBER_NAME=$(echo "$MEMBER_INFO" | awk '{print $2, $3}')
echo
echo "Selected member: $MEMBER_NAME ($MEMBER_EMAIL)"
echo
# Bounce type selection
echo "Select bounce type to simulate:"
echo " 1) Soft Bounce (Transient - e.g., mailbox full)"
echo " 2) Hard Bounce (Permanent - e.g., invalid address)"
echo " 3) Multiple Soft Bounces (3 bounces to trigger soft_bounce status)"
echo " 4) Undetermined Bounce"
echo
read -p "Enter choice (1-4): " BOUNCE_CHOICE
case $BOUNCE_CHOICE in
1)
BOUNCE_TYPE="Transient"
BOUNCE_SUBTYPE="MailboxFull"
DIAGNOSTIC="smtp; 452 4.2.2 Mailbox full"
COUNT=1
;;
2)
BOUNCE_TYPE="Permanent"
BOUNCE_SUBTYPE="General"
DIAGNOSTIC="smtp; 550 5.1.1 User unknown"
COUNT=1
;;
3)
BOUNCE_TYPE="Transient"
BOUNCE_SUBTYPE="General"
DIAGNOSTIC="smtp; 451 4.4.1 Temporary failure"
COUNT=3
;;
4)
BOUNCE_TYPE="Undetermined"
BOUNCE_SUBTYPE=""
DIAGNOSTIC="Unknown error occurred"
COUNT=1
;;
*)
echo "Invalid choice"
exit 1
;;
esac
echo
echo "Simulating $COUNT bounce event(s)..."
echo
# Insert bounce events
for i in $(seq 1 $COUNT); do
TIMESTAMP=$(date -u -d "-$((i * 24)) hours" '+%Y-%m-%d %H:%M:%S')
FEEDBACK_ID="test-feedback-$(date +%s)-$i"
echo "Creating bounce event $i/$COUNT (timestamp: $TIMESTAMP)..."
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
-- Insert bounce log
INSERT INTO bounce_logs (member_id, email, bounce_type, bounce_subtype, diagnostic_code, timestamp, feedback_id)
VALUES ($MEMBER_ID, '$MEMBER_EMAIL', '$BOUNCE_TYPE', '$BOUNCE_SUBTYPE', '$DIAGNOSTIC', '$TIMESTAMP', '$FEEDBACK_ID');
EOF
# Update member bounce status
if [ "$BOUNCE_TYPE" = "Permanent" ]; then
echo " → Hard bounce: Deactivating member..."
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
UPDATE members
SET bounce_count = bounce_count + 1,
last_bounce_at = '$TIMESTAMP',
bounce_status = 'hard_bounce',
active = 0
WHERE member_id = $MEMBER_ID;
EOF
elif [ "$BOUNCE_TYPE" = "Transient" ]; then
# Check current bounce count
CURRENT_COUNT=$(sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -N -e "SELECT bounce_count FROM members WHERE member_id = $MEMBER_ID;")
NEW_COUNT=$((CURRENT_COUNT + 1))
if [ $NEW_COUNT -ge 3 ]; then
echo " → Soft bounce threshold reached: Marking as soft_bounce..."
BOUNCE_STATUS="soft_bounce"
else
echo " → Soft bounce: Incrementing counter ($NEW_COUNT)..."
BOUNCE_STATUS="clean"
fi
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
UPDATE members
SET bounce_count = $NEW_COUNT,
last_bounce_at = '$TIMESTAMP',
bounce_status = '$BOUNCE_STATUS'
WHERE member_id = $MEMBER_ID;
EOF
else
echo " → Undetermined bounce: Incrementing counter..."
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
UPDATE members
SET bounce_count = bounce_count + 1,
last_bounce_at = '$TIMESTAMP'
WHERE member_id = $MEMBER_ID;
EOF
fi
done
echo
echo "✅ Bounce simulation complete!"
echo
# Show updated member status
echo "Updated member status:"
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -e "
SELECT
member_id,
name,
email,
active,
bounce_count,
last_bounce_at,
bounce_status
FROM members
WHERE member_id = $MEMBER_ID\G
"
echo
echo "Bounce history for this member:"
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -e "
SELECT
bounce_id,
bounce_type,
bounce_subtype,
diagnostic_code,
timestamp
FROM bounce_logs
WHERE member_id = $MEMBER_ID
ORDER BY timestamp DESC;
"
echo
echo "🎉 You can now view this bounce in the web UI:"
echo " 1. Open http://localhost:3000"
echo " 2. Go to the Members tab"
echo " 3. Look for $MEMBER_NAME"
echo " 4. Click the 'Bounces' button to see the history"
echo
echo "To simulate more bounces, run this script again!"

View File

@@ -0,0 +1,46 @@
<!DOCTYPE html>
<html>
<head>
<title>CORS Understanding Test</title>
</head>
<body>
<h1>CORS Test</h1>
<div id="results"></div>
<script>
const results = document.getElementById('results');
// Test 1: Same-origin request (should work without CORS headers)
console.log('Current origin:', window.location.origin);
// Test what your API client actually does
const protocol = window.location.protocol;
const hostname = window.location.hostname;
let baseURL;
if (hostname === 'localhost' || hostname === '127.0.0.1') {
baseURL = `${protocol}//${hostname}:8000`;
results.innerHTML += `<p><strong>Development Mode Detected</strong></p>`;
results.innerHTML += `<p>Frontend: ${window.location.origin}</p>`;
results.innerHTML += `<p>API: ${baseURL}</p>`;
results.innerHTML += `<p>This is a <strong>CROSS-ORIGIN</strong> request - CORS applies!</p>`;
} else {
baseURL = `${protocol}//${hostname}/api`;
results.innerHTML += `<p><strong>Production Mode Detected</strong></p>`;
results.innerHTML += `<p>Frontend: ${window.location.origin}</p>`;
results.innerHTML += `<p>API: ${baseURL}</p>`;
results.innerHTML += `<p>This is a <strong>SAME-ORIGIN</strong> request - NO CORS needed!</p>`;
}
// Test the actual request
fetch(`${baseURL}/health`)
.then(response => response.json())
.then(data => {
results.innerHTML += `<p style="color: green;">✅ API Request Successful: ${JSON.stringify(data)}</p>`;
})
.catch(error => {
results.innerHTML += `<p style="color: red;">❌ API Request Failed: ${error.message}</p>`;
});
</script>
</body>
</html>

45
test_bounce_quick.sh Normal file
View File

@@ -0,0 +1,45 @@
#!/bin/bash
# Quick SES Bounce Test - One-liner to test bounce handling
echo "🚀 Testing SES Bounce Handling"
echo "================================"
echo ""
# Check if API and Postfix are running
if ! sudo docker compose ps | grep -q "maillist-api.*Up"; then
echo "❌ API container not running. Start with: sudo docker compose up -d"
exit 1
fi
if ! sudo docker compose ps | grep -q "maillist-postfix.*Up"; then
echo "❌ Postfix container not running. Start with: sudo docker compose up -d"
exit 1
fi
echo "✅ Containers are running"
echo ""
# Start watching logs in background
echo "📋 Opening log viewer (press Ctrl+C to stop)..."
echo ""
sleep 2
# Show the command they can run
echo "Run this command to send a test bounce:"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "echo 'Test bounce' | sudo docker compose exec -T postfix mail -s 'Bounce Test' bounce@simulator.amazonses.com"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo ""
echo "Then wait 30-60 seconds and check:"
echo ""
echo "1. API logs (in this window)"
echo "2. Database: sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e 'SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;'"
echo "3. Web UI: http://localhost:3000 → Members tab"
echo ""
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "Watching API logs now..."
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo ""
# Follow logs
sudo docker compose logs api -f --tail 20

76
test_bounce_webhook.sh Executable file
View File

@@ -0,0 +1,76 @@
#!/bin/bash
# Test SNS Bounce Webhook
# This script simulates an SNS bounce notification (without signature verification for testing)
API_URL="http://localhost:8000"
echo "Testing SNS Bounce Webhook..."
echo
# Test 1: Health check
echo "1. Testing API health..."
curl -s "$API_URL/health" | jq .
echo
# Test 2: Create a test subscription confirmation (the webhook will auto-confirm)
echo "2. Testing subscription confirmation handling..."
curl -s -X POST "$API_URL/webhooks/sns" \
-H "Content-Type: application/json" \
-d '{
"Type": "SubscriptionConfirmation",
"MessageId": "test-message-id",
"Token": "test-token",
"TopicArn": "arn:aws:sns:eu-west-2:123456789:test-topic",
"Message": "You have chosen to subscribe to the topic",
"SubscribeURL": "https://example.com/subscribe",
"Timestamp": "2025-01-01T12:00:00.000Z"
}' || echo "Note: Signature verification will fail for test messages (expected)"
echo
echo
# Test 3: Simulated bounce notification (will fail signature verification)
echo "3. Testing bounce notification structure..."
echo "Note: In production, AWS SNS will send properly signed messages."
echo "This test demonstrates the expected structure."
echo
cat << 'EOF' > /tmp/test_bounce.json
{
"Type": "Notification",
"MessageId": "test-notification-id",
"TopicArn": "arn:aws:sns:eu-west-2:123456789:test-topic",
"Subject": "Amazon SES Email Event Notification",
"Message": "{\"notificationType\":\"Bounce\",\"bounce\":{\"bounceType\":\"Permanent\",\"bounceSubType\":\"General\",\"bouncedRecipients\":[{\"emailAddress\":\"bounce@simulator.amazonses.com\",\"diagnosticCode\":\"smtp; 550 5.1.1 user unknown\"}],\"timestamp\":\"2025-01-01T12:00:00.000Z\",\"feedbackId\":\"test-feedback-id\"}}",
"Timestamp": "2025-01-01T12:00:00.000Z",
"SignatureVersion": "1",
"Signature": "test-signature",
"SigningCertURL": "https://sns.eu-west-2.amazonaws.com/test.pem"
}
EOF
cat /tmp/test_bounce.json | jq .
echo
echo "Expected behavior: Signature verification will fail without real AWS credentials"
echo "In production, AWS SNS will send properly signed messages that will be verified"
echo
# Test 4: Check if database schema has bounce tables
echo "4. Checking database schema for bounce tables..."
sudo docker-compose exec -T mysql mysql -u maillist -pmaillist maillist -e "SHOW TABLES LIKE '%bounce%';" 2>/dev/null
echo
# Test 5: Check members table for bounce columns
echo "5. Checking members table for bounce columns..."
sudo docker-compose exec -T mysql mysql -u maillist -pmaillist maillist -e "DESCRIBE members;" 2>/dev/null | grep -i bounce
echo
echo "✓ Setup complete!"
echo
echo "To test with real AWS SNS:"
echo "1. Set up SNS topic in AWS Console"
echo "2. Subscribe webhook: https://your-domain.com:8000/webhooks/sns"
echo "3. Configure SES to send bounce notifications to the SNS topic"
echo "4. Send test email to bounce@simulator.amazonses.com"
echo
echo "See BOUNCE_HANDLING_SETUP.md for detailed setup instructions"

View File

@@ -54,22 +54,47 @@
</div>
</div>
<!-- Header (shown after login) -->
<header class="header" id="mainHeader" style="display: none;">
<div class="container">
<div class="header-content">
<h1 class="logo">
<!-- Menu Bar Header (shown after login) -->
<header class="menu-bar" id="mainHeader" style="display: none;">
<div class="menu-bar-content">
<div class="app-title">
<i class="fas fa-envelope"></i>
Mailing List Manager
</h1>
<div class="auth-section">
<div class="user-info" id="userInfo">
<span>Mailing List Manager</span>
</div>
<div class="menu-spacer"></div>
<div class="user-dropdown" id="userDropdown">
<button class="user-dropdown-trigger" id="userDropdownTrigger">
<div class="user-avatar">
<i class="fas fa-user"></i>
</div>
<div class="user-details">
<span class="user-name" id="currentUsername">User</span>
<span class="user-role" id="currentUserRole">role</span>
</div>
<button class="btn btn-secondary" id="logoutBtn">Logout</button>
<i class="fas fa-chevron-down dropdown-arrow"></i>
</button>
<div class="user-dropdown-menu" id="userDropdownMenu">
<div class="dropdown-header">
<div class="dropdown-user-info">
<div class="dropdown-avatar">
<i class="fas fa-user"></i>
</div>
<div class="dropdown-details">
<div class="dropdown-name" id="dropdownUsername">User</div>
<div class="dropdown-role" id="dropdownUserRole">role</div>
</div>
</div>
</div>
<div class="dropdown-divider"></div>
<button class="dropdown-item" id="userManagementBtn" style="display: none;">
<i class="fas fa-user-shield"></i>
<span>User Management</span>
</button>
<div class="dropdown-divider" id="userManagementDivider" style="display: none;"></div>
<button class="dropdown-item" id="logoutBtn">
<i class="fas fa-sign-out-alt"></i>
<span>Sign Out</span>
</button>
</div>
</div>
</div>
@@ -88,10 +113,6 @@
<i class="fas fa-users"></i>
Members
</button>
<button class="tab-btn" data-tab="users" id="usersTab" style="display: none;">
<i class="fas fa-user-shield"></i>
Users
</button>
</nav>
<!-- Notification Area -->
@@ -134,13 +155,7 @@
<!-- Members Tab -->
<div class="tab-content" id="members-tab">
<div class="section-header">
<div class="header-content">
<h2>Members</h2>
<div class="header-help">
<i class="fas fa-info-circle"></i>
<span>Click the "Lists" button next to any member to manage their subscriptions</span>
</div>
</div>
<div class="button-group">
<button class="btn btn-primary" id="addMemberBtn">
<i class="fas fa-plus"></i>
@@ -153,6 +168,26 @@
</div>
</div>
<!-- Member Search -->
<div class="search-section">
<div class="search-container">
<div class="search-input-wrapper">
<i class="fas fa-search search-icon"></i>
<input type="text"
id="memberSearchInput"
class="search-input"
placeholder="Search members by name or email..."
autocomplete="off">
<button class="search-clear" id="memberSearchClear" style="display: none;">
<i class="fas fa-times"></i>
</button>
</div>
<div class="search-results-info" id="memberSearchInfo" style="display: none;">
<span id="memberSearchCount">0</span> members found
</div>
</div>
</div>
<div class="data-table">
<table class="table" id="membersTable">
<thead>
@@ -581,6 +616,67 @@
</div>
</div>
<!-- Bounce History Modal -->
<div class="modal" id="bounceHistoryModal">
<div class="modal-content">
<div class="modal-header">
<h3 id="bounceHistoryTitle">Bounce History</h3>
<button class="modal-close" id="bounceHistoryModalClose">
<i class="fas fa-times"></i>
</button>
</div>
<div class="modal-body">
<div class="member-info-banner" id="bounceHistoryMemberInfo">
<div class="member-avatar">
<i class="fas fa-user"></i>
</div>
<div class="member-details">
<h4 id="bounceHistoryMemberName">Member Name</h4>
<p id="bounceHistoryMemberEmail">member@example.com</p>
</div>
</div>
<div class="bounce-summary">
<div class="bounce-stat">
<i class="fas fa-exclamation-triangle"></i>
<div>
<div class="bounce-stat-label">Total Bounces</div>
<div class="bounce-stat-value" id="bounceTotalCount">0</div>
</div>
</div>
<div class="bounce-stat">
<i class="fas fa-clock"></i>
<div>
<div class="bounce-stat-label">Last Bounce</div>
<div class="bounce-stat-value" id="bounceLastDate">Never</div>
</div>
</div>
<div class="bounce-stat">
<i class="fas fa-info-circle"></i>
<div>
<div class="bounce-stat-label">Status</div>
<div class="bounce-stat-value" id="bounceStatusText">Clean</div>
</div>
</div>
</div>
<div class="bounce-history-section">
<h5>Bounce Events</h5>
<div class="bounce-history-list" id="bounceHistoryList">
<!-- Dynamic content will be inserted here -->
</div>
</div>
</div>
<div class="modal-actions">
<button type="button" class="btn btn-secondary" id="bounceHistoryCloseBtn">Close</button>
<button type="button" class="btn btn-warning" id="bounceHistoryResetBtn" data-requires-write>
<i class="fas fa-redo"></i>
Reset Bounce Status
</button>
</div>
</div>
</div>
<!-- Confirmation Modal -->
<div class="modal" id="confirmModal">
<div class="modal-content">
@@ -601,7 +697,7 @@
</div>
<script src="static/js/api.js"></script>
<script src="static/js/ui.js"></script>
<script src="static/js/app.js"></script>
<script src="static/js/ui.js"></script>
</body>
</html>

View File

@@ -285,7 +285,8 @@ body {
}
/* Header */
.header {
/* Menu Bar Header */
.menu-bar {
background: var(--white);
border-bottom: 1px solid var(--gray-200);
box-shadow: var(--shadow-sm);
@@ -294,24 +295,29 @@ body {
z-index: 100;
}
.header-content {
.menu-bar-content {
display: flex;
justify-content: space-between;
align-items: center;
padding: var(--space-4) 0;
justify-content: space-between;
padding: var(--space-2) var(--space-4);
min-height: 56px;
}
.logo {
font-size: var(--font-size-xl);
font-weight: 700;
color: var(--primary-color);
.app-title {
display: flex;
align-items: center;
gap: var(--space-2);
font-size: var(--font-size-lg);
font-weight: 600;
color: var(--primary-color);
}
.logo i {
font-size: var(--font-size-2xl);
.app-title i {
font-size: var(--font-size-xl);
}
.menu-spacer {
flex: 1;
}
/* Authentication */
@@ -342,31 +348,185 @@ body {
box-shadow: 0 0 0 3px rgb(37 99 235 / 0.1);
}
.user-info {
/* User Dropdown - Menu Bar Style */
.user-dropdown {
position: relative;
}
.user-dropdown-trigger {
display: flex;
align-items: center;
gap: var(--space-3);
gap: var(--space-2);
padding: var(--space-2) var(--space-3);
background: transparent;
border: 1px solid var(--gray-200);
border-radius: var(--radius);
cursor: pointer;
transition: var(--transition);
font-size: var(--font-size-sm);
height: 40px;
position: relative;
z-index: 10;
pointer-events: auto;
}
.user-dropdown-trigger:hover {
background: var(--gray-50);
border-color: var(--gray-300);
}
.user-dropdown-trigger.active {
background: var(--gray-50);
border-color: var(--primary-color);
box-shadow: 0 0 0 3px rgb(37 99 235 / 0.1);
}
.user-avatar {
width: 24px;
height: 24px;
background: var(--primary-color);
color: var(--white);
border-radius: 50%;
display: flex;
align-items: center;
justify-content: center;
font-size: var(--font-size-xs);
flex-shrink: 0;
}
.user-details {
display: flex;
flex-direction: column;
align-items: flex-end;
gap: var(--space-1);
align-items: flex-start;
gap: 1px;
min-width: 0;
}
.user-name {
font-weight: 500;
font-size: var(--font-size-xs);
color: var(--gray-900);
line-height: 1.2;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
max-width: 100px;
}
.user-role {
font-size: 10px;
color: var(--gray-500);
text-transform: capitalize;
line-height: 1;
white-space: nowrap;
}
.dropdown-arrow {
color: var(--gray-400);
font-size: 10px;
transition: var(--transition);
flex-shrink: 0;
}
.user-dropdown-trigger.active .dropdown-arrow {
transform: rotate(180deg);
}
.user-dropdown-menu {
position: absolute;
top: 100%;
right: 0;
min-width: 220px;
background: var(--white);
border: 1px solid var(--gray-200);
border-radius: var(--radius);
box-shadow: var(--shadow-lg);
z-index: 1000;
opacity: 0;
visibility: hidden;
transform: translateY(-8px);
transition: all 0.2s ease;
margin-top: var(--space-1);
}
.user-dropdown.active .user-dropdown-menu {
opacity: 1;
visibility: visible;
transform: translateY(0);
}
.dropdown-header {
padding: var(--space-3);
border-bottom: 1px solid var(--gray-100);
}
.dropdown-user-info {
display: flex;
align-items: center;
gap: var(--space-3);
}
.dropdown-avatar {
width: 40px;
height: 40px;
background: var(--primary-color);
color: var(--white);
border-radius: 50%;
display: flex;
align-items: center;
justify-content: center;
font-size: var(--font-size-base);
}
.dropdown-details {
display: flex;
flex-direction: column;
gap: var(--space-1);
}
.dropdown-name {
font-weight: 600;
font-size: var(--font-size-sm);
color: var(--gray-900);
}
.user-role {
.dropdown-role {
font-size: var(--font-size-xs);
color: var(--gray-500);
text-transform: capitalize;
}
.dropdown-divider {
height: 1px;
background: var(--gray-100);
margin: 0;
}
.dropdown-item {
display: flex;
align-items: center;
gap: var(--space-3);
width: 100%;
padding: var(--space-3);
background: transparent;
border: none;
text-align: left;
font-size: var(--font-size-sm);
color: var(--gray-700);
cursor: pointer;
transition: var(--transition);
}
.dropdown-item:hover {
background: var(--gray-50);
color: var(--gray-900);
}
.dropdown-item i {
width: 16px;
color: var(--gray-400);
}
.status-indicator {
display: flex;
align-items: center;
@@ -523,6 +683,113 @@ body {
color: var(--info-color);
}
/* Search Section */
.search-section {
margin-bottom: var(--space-6);
}
.search-container {
display: flex;
flex-direction: column;
gap: var(--space-3);
}
.search-input-wrapper {
position: relative;
display: flex;
align-items: center;
max-width: 400px;
}
.search-input {
width: 100%;
padding: var(--space-3) var(--space-4) var(--space-3) var(--space-10);
border: 1px solid var(--gray-300);
border-radius: var(--radius);
font-size: var(--font-size-sm);
transition: var(--transition);
background: var(--white);
}
.search-input:focus {
outline: none;
border-color: var(--primary-color);
box-shadow: 0 0 0 3px rgb(37 99 235 / 0.1);
}
.search-input:not(:placeholder-shown) {
padding-right: var(--space-10);
}
.search-icon {
position: absolute;
left: var(--space-3);
color: var(--gray-400);
font-size: var(--font-size-sm);
pointer-events: none;
z-index: 1;
}
.search-clear {
position: absolute;
right: var(--space-3);
background: transparent;
border: none;
color: var(--gray-400);
cursor: pointer;
padding: var(--space-1);
border-radius: var(--radius-sm);
display: flex;
align-items: center;
justify-content: center;
transition: var(--transition);
}
.search-clear:hover {
color: var(--gray-600);
background: var(--gray-100);
}
.search-results-info {
font-size: var(--font-size-sm);
color: var(--gray-600);
font-weight: 500;
}
.search-results-info #memberSearchCount {
color: var(--primary-color);
font-weight: 600;
}
/* No results message */
.no-results {
text-align: center;
padding: var(--space-8) var(--space-4);
color: var(--gray-500);
}
.no-results i {
font-size: var(--font-size-3xl);
color: var(--gray-300);
margin-bottom: var(--space-4);
}
.no-results h3 {
font-size: var(--font-size-lg);
font-weight: 600;
margin-bottom: var(--space-2);
color: var(--gray-700);
}
.no-results p {
font-size: var(--font-size-sm);
margin-bottom: var(--space-4);
}
.no-results .btn {
margin-top: var(--space-2);
}
/* Notifications */
.notification {
display: flex;
@@ -624,6 +891,32 @@ body {
color: var(--gray-600);
}
/* Bounce badges */
.bounce-badge {
display: inline-flex;
align-items: center;
gap: var(--space-1);
padding: var(--space-1) var(--space-2);
border-radius: var(--radius-sm);
font-size: var(--font-size-xs);
font-weight: 500;
}
.bounce-badge.bounce-hard {
background: #fee2e2;
color: #991b1b;
}
.bounce-badge.bounce-soft {
background: #fef3c7;
color: #92400e;
}
.bounce-badge.bounce-warning {
background: #fef9c3;
color: #854d0e;
}
/* Role badges */
.role-badge {
display: inline-flex;
@@ -919,12 +1212,36 @@ body {
padding: 0 var(--space-3);
}
.header-content {
.menu-bar-content {
flex-direction: column;
gap: var(--space-4);
gap: var(--space-2);
padding: var(--space-2) var(--space-3);
align-items: stretch;
}
.app-title {
order: 1;
justify-content: center;
font-size: var(--font-size-base);
}
.user-dropdown {
order: 2;
align-self: flex-end;
width: auto;
}
.menu-spacer {
display: none;
}
.user-dropdown-menu {
right: 0;
left: auto;
width: auto;
min-width: 200px;
}
.auth-controls {
flex-direction: column;
gap: var(--space-2);
@@ -944,6 +1261,18 @@ body {
align-items: stretch;
}
.search-input-wrapper {
max-width: none;
}
.search-input {
font-size: 16px; /* Prevent zoom on iOS */
}
.search-container {
margin-bottom: var(--space-4);
}
.table-responsive {
overflow-x: auto;
}
@@ -1479,3 +1808,147 @@ body {
.fade-in {
animation: fadeIn 0.3s ease-out;
}
/* Bounce History Modal Styles */
.bounce-summary {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
gap: var(--space-4);
margin: var(--space-6) 0;
padding: var(--space-4);
background: var(--gray-50);
border-radius: var(--radius);
}
.bounce-stat {
display: flex;
align-items: center;
gap: var(--space-3);
}
.bounce-stat i {
font-size: 1.5rem;
color: var(--primary-color);
}
.bounce-stat-label {
font-size: var(--font-size-xs);
color: var(--gray-500);
text-transform: uppercase;
letter-spacing: 0.5px;
}
.bounce-stat-value {
font-size: var(--font-size-lg);
font-weight: 600;
color: var(--gray-800);
}
.bounce-stat-value.text-danger {
color: var(--danger-color);
}
.bounce-stat-value.text-warning {
color: var(--warning-color);
}
.bounce-stat-value.text-success {
color: var(--success-color);
}
.bounce-history-section {
margin-top: var(--space-6);
}
.bounce-history-section h5 {
margin-bottom: var(--space-4);
color: var(--gray-700);
}
.bounce-history-list {
max-height: 400px;
overflow-y: auto;
padding: var(--space-2);
}
.bounce-history-item {
background: var(--white);
border: 1px solid var(--gray-200);
border-radius: var(--radius);
padding: var(--space-4);
margin-bottom: var(--space-3);
}
.bounce-history-item:last-child {
margin-bottom: 0;
}
.bounce-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: var(--space-2);
}
.bounce-type {
display: inline-flex;
align-items: center;
gap: var(--space-2);
padding: var(--space-1) var(--space-3);
border-radius: var(--radius-sm);
font-size: var(--font-size-sm);
font-weight: 600;
}
.bounce-type.bounce-type-permanent {
background: #fee2e2;
color: #991b1b;
}
.bounce-type.bounce-type-transient {
background: #fef3c7;
color: #92400e;
}
.bounce-type.bounce-type-undetermined {
background: var(--gray-100);
color: var(--gray-600);
}
.bounce-date {
font-size: var(--font-size-sm);
color: var(--gray-500);
}
.bounce-subtype {
font-size: var(--font-size-sm);
color: var(--gray-600);
margin-top: var(--space-2);
font-weight: 500;
}
.bounce-diagnostic {
font-size: var(--font-size-sm);
color: var(--gray-500);
margin-top: var(--space-2);
padding: var(--space-2);
background: var(--gray-50);
border-radius: var(--radius-sm);
border-left: 3px solid var(--warning-color);
font-family: 'Courier New', monospace;
}
.empty-state {
text-align: center;
padding: var(--space-8);
color: var(--gray-400);
}
.empty-state i {
font-size: 3rem;
margin-bottom: var(--space-4);
color: var(--success-color);
}
.empty-state p {
font-size: var(--font-size-sm);
}

View File

@@ -25,8 +25,8 @@ class APIClient {
return `${protocol}//${hostname}:8000`;
}
// If running in production, assume API is on port 8000
return `${protocol}//${hostname}:8000`;
// If running in production behind a reverse proxy, use /api path
return `${protocol}//${hostname}/api`;
}
/**
@@ -50,9 +50,14 @@ class APIClient {
*/
async request(endpoint, options = {}) {
const url = `${this.baseURL}${endpoint}`;
// Merge options, with custom headers taking precedence
const config = {
headers: { ...this.headers },
...options
...options,
headers: {
...this.headers,
...(options.headers || {})
}
};
try {
@@ -100,20 +105,26 @@ class APIClient {
return this.request('/health');
}
async getConfig() {
return this.request('/config');
}
async testAuth() {
return this.request('/');
}
// Authentication API
async login(username, password) {
// Don't include Authorization header for login
const tempHeaders = { ...this.headers };
delete tempHeaders['Authorization'];
const response = await this.request('/auth/login', {
method: 'POST',
body: JSON.stringify({
username: username,
password: password
})
headers: tempHeaders,
body: JSON.stringify({ username, password })
});
// Set the token from the response
if (response.access_token) {
this.setToken(response.access_token);
}
@@ -126,41 +137,16 @@ class APIClient {
await this.request('/auth/logout', {
method: 'POST'
});
} catch (error) {
// Ignore logout errors, we'll clear the token anyway
}
} finally {
// Clear token even if logout fails
this.clearToken();
}
}
async getCurrentUser() {
return this.request('/auth/me');
}
// User management API
async getUsers() {
return this.request('/users');
}
async createUser(userData) {
return this.request('/users', {
method: 'POST',
body: JSON.stringify(userData)
});
}
async updateUser(userId, userData) {
return this.request(`/users/${userId}`, {
method: 'PATCH',
body: JSON.stringify(userData)
});
}
async deleteUser(userId) {
return this.request(`/users/${userId}`, {
method: 'DELETE'
});
}
// Mailing Lists API
async getLists() {
return this.request('/lists');
@@ -254,6 +240,42 @@ class APIClient {
})
});
}
// User Management API
async getUsers() {
return this.request('/users');
}
async createUser(userData) {
return this.request('/users', {
method: 'POST',
body: JSON.stringify(userData)
});
}
async updateUser(userId, userData) {
return this.request(`/users/${userId}`, {
method: 'PATCH',
body: JSON.stringify(userData)
});
}
async deleteUser(userId) {
return this.request(`/users/${userId}`, {
method: 'DELETE'
});
}
// Bounce Management API
async getMemberBounces(memberId) {
return this.request(`/members/${memberId}/bounces`);
}
async resetBounceStatus(memberId) {
return this.request(`/members/${memberId}/bounce-status`, {
method: 'PATCH'
});
}
}
/**

View File

@@ -52,6 +52,33 @@ class MailingListApp {
this.logout();
});
// User dropdown functionality
const userDropdownTrigger = document.getElementById('userDropdownTrigger');
const userDropdown = document.getElementById('userDropdown');
userDropdownTrigger.addEventListener('click', (e) => {
e.preventDefault();
e.stopPropagation();
userDropdown.classList.toggle('active');
userDropdownTrigger.classList.toggle('active');
});
// Close dropdown when clicking outside
document.addEventListener('click', (e) => {
if (!userDropdown.contains(e.target)) {
userDropdown.classList.remove('active');
userDropdownTrigger.classList.remove('active');
}
});
// Close dropdown on escape key
document.addEventListener('keydown', (e) => {
if (e.key === 'Escape') {
userDropdown.classList.remove('active');
userDropdownTrigger.classList.remove('active');
}
});
// Bulk import button
document.getElementById('showBulkImportBtn').addEventListener('click', () => {
uiManager.showBulkImportModal();
@@ -61,6 +88,47 @@ class MailingListApp {
document.getElementById('addUserBtn').addEventListener('click', () => {
uiManager.showUserModal();
});
// User management dropdown item
document.getElementById('userManagementBtn').addEventListener('click', () => {
// Close the dropdown
userDropdown.classList.remove('active');
userDropdownTrigger.classList.remove('active');
// Switch to users tab
this.switchToUsersTab();
});
// Member search functionality
const memberSearchInput = document.getElementById('memberSearchInput');
const memberSearchClear = document.getElementById('memberSearchClear');
memberSearchInput.addEventListener('input', (e) => {
this.filterMembers(e.target.value);
// Show/hide clear button
if (e.target.value.trim()) {
memberSearchClear.style.display = 'flex';
} else {
memberSearchClear.style.display = 'none';
}
});
memberSearchClear.addEventListener('click', () => {
memberSearchInput.value = '';
memberSearchClear.style.display = 'none';
this.filterMembers('');
memberSearchInput.focus();
});
// Clear search when switching tabs
document.querySelectorAll('.tab-btn').forEach(btn => {
btn.addEventListener('click', () => {
if (btn.dataset.tab !== 'members') {
memberSearchInput.value = '';
memberSearchClear.style.display = 'none';
}
});
});
}
/**
@@ -214,10 +282,13 @@ class MailingListApp {
if (this.currentUser) {
document.getElementById('currentUsername').textContent = this.currentUser.username;
document.getElementById('currentUserRole').textContent = this.currentUser.role;
document.getElementById('dropdownUsername').textContent = this.currentUser.username;
document.getElementById('dropdownUserRole').textContent = this.currentUser.role;
// Show/hide admin-only features
const isAdmin = this.currentUser.role === 'administrator';
document.getElementById('usersTab').style.display = isAdmin ? 'block' : 'none';
document.getElementById('userManagementBtn').style.display = isAdmin ? 'block' : 'none';
document.getElementById('userManagementDivider').style.display = isAdmin ? 'block' : 'none';
// Show/hide write access features
const hasWriteAccess = this.currentUser.role === 'administrator' || this.currentUser.role === 'operator';
@@ -250,6 +321,28 @@ class MailingListApp {
document.getElementById('showBulkImportBtn').setAttribute('data-requires-write', '');
}
/**
* Switch to users tab (triggered from user dropdown)
*/
switchToUsersTab() {
// Switch to users tab programmatically
const tabButtons = document.querySelectorAll('.tab-btn');
const tabContents = document.querySelectorAll('.tab-content');
// Remove active class from all tabs and contents
tabButtons.forEach(btn => btn.classList.remove('active'));
tabContents.forEach(content => content.classList.remove('active'));
// Show users tab content
const usersTab = document.getElementById('users-tab');
if (usersTab) {
usersTab.classList.add('active');
}
// Load users data if needed
this.loadUsers();
}
/**
* Load all data from API
*/
@@ -259,12 +352,15 @@ class MailingListApp {
try {
uiManager.setLoading(true);
// Load lists and members in parallel
const [lists, members] = await Promise.all([
// Load configuration, lists and members in parallel
const [config, lists, members] = await Promise.all([
apiClient.getConfig(),
apiClient.getLists(),
apiClient.getMembers()
]);
this.config = config;
this.lists = lists;
this.members = members;
@@ -438,6 +534,7 @@ class MailingListApp {
row.innerHTML = `
<td>
<div class="font-medium">${uiManager.escapeHtml(member.name)}</div>
${this.config?.bounce_handling_enabled && member.bounce_count > 0 ? `<div class="text-xs text-muted" style="margin-top: 2px;"></div>` : ''}
</td>
<td>
<a href="mailto:${member.email}" style="color: var(--primary-color)">
@@ -458,6 +555,15 @@ class MailingListApp {
</td>
`;
// Add bounce badge if member has bounces (only if bounce handling is enabled)
if (this.config?.bounce_handling_enabled && member.bounce_count > 0) {
const bounceInfoDiv = row.cells[0].querySelector('.text-xs');
const bounceBadge = uiManager.createBounceStatusBadge(member.bounce_status, member.bounce_count);
if (bounceBadge) {
bounceInfoDiv.appendChild(bounceBadge);
}
}
// Add status badge
const statusCell = row.cells[3];
statusCell.appendChild(uiManager.createStatusBadge(member.active));
@@ -474,6 +580,18 @@ class MailingListApp {
uiManager.showMemberSubscriptionsModal(member);
});
// Create Bounces button (show if bounce handling is enabled and member has bounces or for admins/operators)
if (this.config?.bounce_handling_enabled && (member.bounce_count > 0 || hasWriteAccess)) {
const bouncesBtn = document.createElement('button');
bouncesBtn.className = `btn btn-sm ${member.bounce_count > 0 ? 'btn-warning' : 'btn-secondary'}`;
bouncesBtn.innerHTML = `<i class="fas fa-exclamation-triangle"></i> Bounces${member.bounce_count > 0 ? ` (${member.bounce_count})` : ''}`;
bouncesBtn.title = 'View Bounce History';
bouncesBtn.addEventListener('click', () => {
uiManager.showBounceHistoryModal(member);
});
actionsCell.appendChild(bouncesBtn);
}
// Only show edit/delete buttons for users with write access
if (hasWriteAccess) {
const editBtn = uiManager.createActionButton('Edit', 'edit', 'btn-secondary', () => {
@@ -645,6 +763,149 @@ class MailingListApp {
uiManager.setLoading(false);
}
}
/**
* Filter members by search term (name or email)
*/
filterMembers(searchTerm) {
const searchInfo = document.getElementById('memberSearchInfo');
const searchCount = document.getElementById('memberSearchCount');
const tbody = document.getElementById('membersTableBody');
if (!this.members || !Array.isArray(this.members)) {
return;
}
// If no search term, show all members
if (!searchTerm || searchTerm.trim() === '') {
this.renderMembers();
searchInfo.style.display = 'none';
return;
}
const normalizedSearch = searchTerm.trim().toLowerCase();
// Filter members by name or email
const filteredMembers = this.members.filter(member => {
const name = (member.name || '').toLowerCase();
const email = (member.email || '').toLowerCase();
return name.includes(normalizedSearch) || email.includes(normalizedSearch);
});
// Update search results info
searchCount.textContent = filteredMembers.length;
searchInfo.style.display = 'block';
// Render filtered results
this.renderFilteredMembers(filteredMembers);
}
/**
* Render filtered members (similar to renderMembers but with filtered data)
*/
renderFilteredMembers(filteredMembers) {
const tbody = document.getElementById('membersTableBody');
tbody.innerHTML = '';
if (filteredMembers.length === 0) {
// Show no results message
const row = tbody.insertRow();
const cell = row.insertCell();
cell.colSpan = 5;
cell.className = 'no-results';
cell.innerHTML = `
<i class="fas fa-search"></i>
<h3>No members found</h3>
<p>Try adjusting your search terms or check the spelling.</p>
`;
return;
}
filteredMembers.forEach(member => {
const row = tbody.insertRow();
// Name
const nameCell = row.insertCell();
nameCell.textContent = member.name;
// Email
const emailCell = row.insertCell();
const emailLink = document.createElement('a');
emailLink.href = `mailto:${member.email}`;
emailLink.textContent = member.email;
emailLink.style.color = 'var(--primary-color)';
emailCell.appendChild(emailLink);
// Lists (show member's subscriptions)
const listsCell = row.insertCell();
const memberLists = [];
this.subscriptions.forEach((members, listId) => {
if (members.some(m => m.member_id === member.member_id)) {
const list = this.lists.find(l => l.list_id === listId);
if (list) {
memberLists.push(list.list_name);
}
}
});
listsCell.textContent = memberLists.length > 0 ? memberLists.join(', ') : 'None';
// Status
const statusCell = row.insertCell();
const statusBadge = document.createElement('span');
statusBadge.className = `status-badge ${member.active ? 'active' : 'inactive'}`;
statusBadge.innerHTML = `<i class="fas fa-${member.active ? 'check' : 'times'}"></i> ${member.active ? 'Active' : 'Inactive'}`;
// Add bounce status if exists and bounce handling is enabled
if (this.config?.bounce_handling_enabled && member.bounce_status && member.bounce_status !== 'clean') {
const bounceIndicator = document.createElement('span');
bounceIndicator.className = `bounce-badge bounce-${member.bounce_status === 'hard_bounce' ? 'hard' : 'soft'}`;
bounceIndicator.innerHTML = `<i class="fas fa-exclamation-triangle"></i> Bounces`;
bounceIndicator.title = `${member.bounce_count || 0} bounces - ${member.bounce_status}`;
statusCell.appendChild(document.createElement('br'));
statusCell.appendChild(bounceIndicator);
}
statusCell.appendChild(statusBadge);
// Actions
const actionsCell = row.insertCell();
actionsCell.className = 'action-buttons';
// Bounces button (if bounce handling is enabled and member has bounce data)
if (this.config?.bounce_handling_enabled && member.bounce_count > 0) {
const bouncesBtn = uiManager.createActionButton('Bounces', 'exclamation-triangle', 'btn-warning', () => {
uiManager.showBounceHistory(member);
});
actionsCell.appendChild(bouncesBtn);
}
const subscriptionsBtn = uiManager.createActionButton('Lists', 'list', 'btn-secondary', () => {
uiManager.showMemberSubscriptions(member);
});
actionsCell.appendChild(subscriptionsBtn);
const editBtn = uiManager.createActionButton('Edit', 'edit', 'btn-primary', () => {
uiManager.showMemberModal(member);
});
editBtn.setAttribute('data-requires-write', '');
actionsCell.appendChild(editBtn);
const deleteBtn = uiManager.createActionButton('Delete', 'trash', 'btn-danger', () => {
uiManager.showConfirmation(
`Are you sure you want to delete member "${member.name}"? This will also remove them from all mailing lists.`,
async () => {
await this.deleteMember(member.member_id);
}
);
});
deleteBtn.setAttribute('data-requires-write', '');
actionsCell.appendChild(deleteBtn);
});
// Update UI permissions for the filtered results
const hasWriteAccess = this.currentUser && (this.currentUser.role === 'administrator' || this.currentUser.role === 'operator');
this.updateUIForPermissions(hasWriteAccess);
}
}
// Initialize the application when DOM is loaded

View File

@@ -76,6 +76,19 @@ class UIManager {
this.handleMemberSubscriptionsSave();
});
// Bounce history modal
document.getElementById('bounceHistoryModalClose').addEventListener('click', () => {
this.closeModal(document.getElementById('bounceHistoryModal'));
});
document.getElementById('bounceHistoryCloseBtn').addEventListener('click', () => {
this.closeModal(document.getElementById('bounceHistoryModal'));
});
document.getElementById('bounceHistoryResetBtn').addEventListener('click', () => {
this.handleBounceStatusReset();
});
// Form submissions
document.getElementById('listForm').addEventListener('submit', (e) => {
e.preventDefault();
@@ -110,6 +123,19 @@ class UIManager {
this.confirmCallback = null;
this.closeModal(document.getElementById('confirmModal'));
});
// Bounce history modal
document.getElementById('bounceHistoryModalClose').addEventListener('click', () => {
this.closeModal(document.getElementById('bounceHistoryModal'));
});
document.getElementById('bounceHistoryCloseBtn').addEventListener('click', () => {
this.closeModal(document.getElementById('bounceHistoryModal'));
});
document.getElementById('bounceHistoryResetBtn').addEventListener('click', () => {
this.handleResetBounceStatus();
});
}
/**
@@ -664,6 +690,117 @@ class UIManager {
}
}
/**
* Show bounce history modal for a member
*/
async showBounceHistoryModal(member) {
const modal = document.getElementById('bounceHistoryModal');
// Update member info
document.getElementById('bounceHistoryTitle').textContent = `Bounce History - ${member.name}`;
document.getElementById('bounceHistoryMemberName').textContent = member.name;
document.getElementById('bounceHistoryMemberEmail').textContent = member.email;
// Update summary stats
document.getElementById('bounceTotalCount').textContent = member.bounce_count || 0;
document.getElementById('bounceLastDate').textContent = this.formatDateTime(member.last_bounce_at);
const statusText = document.getElementById('bounceStatusText');
statusText.className = 'bounce-stat-value';
if (member.bounce_status === 'hard_bounce') {
statusText.textContent = 'Hard Bounce';
statusText.classList.add('text-danger');
} else if (member.bounce_status === 'soft_bounce') {
statusText.textContent = 'Soft Bounce';
statusText.classList.add('text-warning');
} else {
statusText.textContent = 'Clean';
statusText.classList.add('text-success');
}
try {
this.currentMemberForBounces = member;
// Load bounce history
const bounces = await apiClient.getMemberBounces(member.member_id);
this.renderBounceHistory(bounces);
this.showModal(modal);
} catch (error) {
this.handleError(error, 'Failed to load bounce history');
}
}
/**
* Render bounce history list
*/
renderBounceHistory(bounces) {
const container = document.getElementById('bounceHistoryList');
container.innerHTML = '';
if (bounces.length === 0) {
container.innerHTML = `
<div class="empty-state">
<i class="fas fa-check-circle"></i>
<p>No bounces recorded for this member</p>
</div>
`;
return;
}
bounces.forEach(bounce => {
const item = document.createElement('div');
item.className = 'bounce-history-item';
let typeClass = 'bounce-type-';
if (bounce.bounce_type === 'Permanent') {
typeClass += 'permanent';
} else if (bounce.bounce_type === 'Transient') {
typeClass += 'transient';
} else {
typeClass += 'undetermined';
}
item.innerHTML = `
<div class="bounce-header">
<span class="bounce-type ${typeClass}">
<i class="fas fa-${bounce.bounce_type === 'Permanent' ? 'times-circle' : bounce.bounce_type === 'Transient' ? 'exclamation-circle' : 'question-circle'}"></i>
${bounce.bounce_type}
</span>
<span class="bounce-date">${this.formatDateTime(bounce.timestamp)}</span>
</div>
${bounce.bounce_subtype ? `<div class="bounce-subtype">Subtype: ${this.escapeHtml(bounce.bounce_subtype)}</div>` : ''}
${bounce.diagnostic_code ? `<div class="bounce-diagnostic">${this.escapeHtml(bounce.diagnostic_code)}</div>` : ''}
`;
container.appendChild(item);
});
}
/**
* Handle bounce status reset
*/
async handleResetBounceStatus() {
if (!this.currentMemberForBounces) return;
this.showConfirmation(
`Are you sure you want to reset the bounce status for "${this.currentMemberForBounces.name}"? This will clear the bounce count and allow emails to be sent to this address again.`,
async () => {
try {
this.setLoading(true);
await apiClient.resetBounceStatus(this.currentMemberForBounces.member_id);
this.showNotification('Bounce status reset successfully', 'success');
this.closeModal(document.getElementById('bounceHistoryModal'));
await window.app.loadData();
} catch (error) {
this.handleError(error, 'Failed to reset bounce status');
} finally {
this.setLoading(false);
}
}
);
}
/**
* Handle API errors
*/
@@ -715,6 +852,49 @@ class UIManager {
return badge;
}
/**
* Create bounce status badge
*/
createBounceStatusBadge(bounceStatus, bounceCount) {
const badge = document.createElement('span');
if (bounceStatus === 'hard_bounce') {
badge.className = 'bounce-badge bounce-hard';
badge.innerHTML = `<i class="fas fa-exclamation-triangle"></i> Hard Bounce`;
badge.title = `${bounceCount} bounce(s) - Email permanently failed`;
} else if (bounceStatus === 'soft_bounce') {
badge.className = 'bounce-badge bounce-soft';
badge.innerHTML = `<i class="fas fa-exclamation-circle"></i> Soft Bounce`;
badge.title = `${bounceCount} bounce(s) - Temporary delivery issues`;
} else if (bounceCount > 0) {
badge.className = 'bounce-badge bounce-warning';
badge.innerHTML = `<i class="fas fa-info-circle"></i> ${bounceCount} bounce(s)`;
badge.title = `${bounceCount} bounce(s) recorded`;
} else {
return null; // No badge for clean status
}
return badge;
}
/**
* Format date and time
*/
formatDateTime(dateString) {
if (!dateString) return 'Never';
const date = new Date(dateString);
return date.toLocaleString();
}
/**
* Format date only
*/
formatDate(dateString) {
if (!dateString) return 'Never';
const date = new Date(dateString);
return date.toLocaleDateString();
}
/**
* Format email as mailto link
*/