Compare commits

...

16 Commits

Author SHA1 Message Date
James Pattinson
f3d7592e7d Email bounce handling with postfix 2025-10-14 16:16:44 +00:00
James Pattinson
12a82c8d03 Ability to disable SNS bounce handling 2025-10-14 15:39:33 +00:00
James Pattinson
b34ea2ed84 Dynamic search 2025-10-13 20:05:08 +00:00
James Pattinson
8fd951fd1f FIx menu bar 2025-10-13 19:58:46 +00:00
James Pattinson
ecbc38cf8e Cleanuop and CORS fixing 2025-10-13 19:30:19 +00:00
James Pattinson
d37027ee5a Fixed bounce handling 2025-10-13 16:53:22 +00:00
James Pattinson
72f3297a80 SES SNS Bounce Handling 2025-10-13 15:05:42 +00:00
James Pattinson
ac23638125 Failed login feedback 2025-10-13 14:23:17 +00:00
James Pattinson
4fc1ed96dd Failed login feedback 2025-10-13 14:23:02 +00:00
James Pattinson
459e16b26f Schema correction 2025-10-13 14:12:33 +00:00
James Pattinson
d4b88e0952 RBAC and Doc updates 2025-10-13 14:05:01 +00:00
James Pattinson
f721be7280 CSV Import 2025-10-13 13:28:12 +00:00
James Pattinson
9b6a6dab06 Sub mamangement 2025-10-12 21:20:53 +00:00
James Pattinson
ba1bf32393 Added web front end 2025-10-12 20:55:13 +00:00
James Pattinson
b8a91103e9 Now we have an API 2025-10-12 19:33:45 +00:00
James Pattinson
35f710049a MySQL support 2025-10-12 19:24:14 +00:00
38 changed files with 10291 additions and 110 deletions

View File

@@ -8,4 +8,38 @@ SES_PASS=your_ses_secret_access_key
# Optional: SMTP server configuration
# Default is EU West 2 - change if using different region
SMTP_HOST=email-smtp.eu-west-2.amazonaws.com
SMTP_PORT=587
SMTP_PORT=587
# MySQL Database Configuration
MYSQL_HOST=mysql
MYSQL_PORT=3306
MYSQL_DATABASE=maillist
MYSQL_USER=maillist
MYSQL_PASSWORD=change_this_password
MYSQL_ROOT_PASSWORD=change_this_root_password
# API Configuration
API_TOKEN=change_this_to_a_secure_random_token
# Bounce Handling Configuration (Optional)
# Choose one of the bounce handling methods below:
# Method 1: SNS Webhook Bounce Handling (Requires SES Production Access)
# Set to 'true' to enable real-time SNS webhook bounce handling
ENABLE_SNS_WEBHOOKS=false
ENABLE_BOUNCE_HANDLING=false
# If ENABLE_SNS_WEBHOOKS=true, you need:
# 1. AWS SNS topic configured
# 2. SES configured to send notifications to SNS topic
# 3. Valid HTTPS domain for webhook endpoint
# 4. SNS subscription confirmed to your webhook endpoint
# Method 2: Email-Based Bounce Processing (Works with SES Sandbox)
# Set to 'true' to enable email-based bounce processing
# This processes bounce emails that are sent to bounces@lists.sasalliance.org
ENABLE_EMAIL_BOUNCE_PROCESSING=false
# Note: You can enable both methods, but typically only one is needed
# Email-based processing works with SES sandbox accounts
# SNS webhooks provide real-time processing but require SES production access

View File

@@ -1,97 +1,169 @@
# Copilot Instructions: Mail List Manager (Postfix + SES + Web Interface)
# Copilot Instructions: Mail List Manager (Postfix + SES + MySQL + API + Web)
## Architecture Overview
This is a containerized mailing list management system built around Postfix as an SMTP relay through Amazon SES. **Currently in Phase 1** with static configuration; **planned expansion** includes web frontend and SQL database for dynamic member management.
This is a complete containerized mailing list management system with four services:
**Current Components (Phase 1):**
- `docker-compose.yaml`: Single-service orchestration with SES credentials
- `postfix/`: Complete Postfix container configuration
- Static virtual aliases system for mailing list distribution
1. **Web Frontend** (Port 3000) - Nginx serving modern HTML/CSS/JS interface
2. **REST API** (Port 8000) - FastAPI backend with token authentication
3. **MySQL Database** (Internal) - Stores lists, members, and subscriptions
4. **Postfix Mail Server** (Port 25) - SMTP server with native MySQL integration
**Planned Architecture (Phase 2+):**
- Web frontend for list management (view/add/remove members)
- SQL database backend for member storage
- Dynamic Postfix configuration generation
- Multi-service Docker Compose setup
**Key Architecture Principles:**
- Real-time operation: Changes take effect immediately without Postfix reload
- Native integration: Postfix queries MySQL directly (no Python scripts)
- Security-first: Private Docker network, token auth, sender whitelist
- User-friendly: Web interface for easy management by non-technical users
## Configuration Pattern
**Current (Static):** Environment variable substitution for secure credential management:
**Dynamic Database-Driven Architecture:**
1. **Credentials Flow**: SES credentials passed via environment → `sasl_passwd.template` → runtime generation in `entrypoint.sh`
2. **Config Files**: Static configs (`main.cf`, `virtual_aliases.cf`) + dynamic SASL auth file
3. **Postfix Maps**: Hash databases generated at build time (virtual aliases) and runtime (SASL)
1. **Data Flow**: Web UI → REST API → MySQL ← Postfix (real-time queries)
2. **Credentials**: All sensitive data in `.env` file (SES, MySQL, API_TOKEN)
3. **Mail Processing**:
- Email arrives for `list@lists.sasalliance.org`
- Postfix queries MySQL via `mysql_virtual_alias_maps.cf`
- MySQL returns comma-separated member emails
- Postfix expands alias and delivers via SES
4. **Security**: Private Docker network (maillist-internal), sender whitelist (@sasalliance.org)
**Future (Dynamic):** Database-driven configuration:
- Member lists stored in SQL database
- Web interface for CRUD operations on members
- `virtual_aliases.cf` generated from database at runtime
- Postfix reload triggered by configuration changes
**No Static Configuration:**
- Virtual aliases are stored in MySQL database, not config files
- Changes take effect immediately without container restarts
- Postfix caches query results for performance
## Key Files and Their Roles
- `main.cf`: Core Postfix config - relay through SES, domain settings, security
- `sasl_passwd.template`: Template for SES authentication (uses `${SES_USER}:${SES_PASS}`)
- `virtual_aliases.cf`: Static email forwarding rules (one mailing list currently)
- `entrypoint.sh`: Runtime credential processing and Postfix startup
### Docker & Environment
- `docker-compose.yaml`: Multi-service orchestration (mysql, postfix, api, web)
- `.env`: All credentials and configuration (SES, MySQL, API_TOKEN)
### Web Frontend (`web/`)
- `index.html`: Single-page application interface
- `static/js/api.js`: API client with authentication
- `static/js/ui.js`: UI helpers and modal management
- `static/js/app.js`: Main application controller
- `static/css/style.css`: Complete styling system
- `nginx.conf`: Nginx configuration for static file serving
### REST API (`api/`)
- `main.py`: FastAPI application with all CRUD endpoints
- `requirements.txt`: Python dependencies (fastapi, mysql-connector-python, etc.)
- `Dockerfile`: Python 3.11 container build
### Database (`database/`)
- `schema.sql`: Database initialization with tables and sample data
- Tables: `lists`, `members`, `list_members` (many-to-many junction)
### Postfix (`postfix/`)
- `main.cf.template`: Core Postfix config with MySQL virtual alias maps
- `mysql_virtual_alias_maps.cf`: Native Postfix MySQL query configuration
- `sender_access`: Sender whitelist (sasalliance.org domain allowed)
- `sasl_passwd.template`: SES authentication template
- `entrypoint.sh`: Container startup with MySQL health check and config generation
## Development Workflows
**Building and Running:**
Always use sudo for Docker commands. Don't bother opening the simple browser UI for Docker as it is not very useful.
```bash
docker-compose up --build # Build and start mail server
docker-compose logs -f # Monitor mail delivery logs
sudo docker-compose up --build # Build and start all services
sudo docker-compose logs -f # Monitor all service logs
sudo docker-compose logs -f api # Monitor specific service
```
**Adding Mailing Lists (Current):**
1. Edit `virtual_aliases.cf` with new list → recipients mapping
2. Rebuild container (postmap runs at build time)
3. No restart needed for existing virtual alias changes
**Managing Lists and Members:**
**Future Web Interface Workflow:**
1. Access web frontend at configured port
2. Use CRUD interface to manage mailing lists and members
3. Changes automatically update database and regenerate Postfix configs
**Via Web Interface (Recommended):**
1. Open http://localhost:3000
2. Enter API_TOKEN from .env file
3. Use intuitive interface to manage lists, members, and subscriptions
4. Click "Subscriptions" button on any member for toggle-based subscription management
**Via REST API:**
```bash
# Get all lists
curl -H "Authorization: Bearer $API_TOKEN" http://localhost:8000/lists
# Create new list
curl -X POST http://localhost:8000/lists \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"list_name":"Test","list_email":"test@lists.sasalliance.org","active":true}'
# Subscribe member to list
curl -X POST http://localhost:8000/subscriptions \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"list_email":"test@lists.sasalliance.org","member_email":"user@example.com"}'
```
**Via Direct Database Access:**
```bash
docker-compose exec mysql mysql -u maillist -p maillist
# See database/README.md for SQL examples
```
**Testing Mail Delivery:**
```bash
# From inside container
docker-compose exec postfix bash
echo "Test message" | mail -s "Subject" community@lists.sasalliance.org
# Check logs for SES relay status
docker-compose logs postfix | grep -E "(sent|bounced|deferred)"
# Verify MySQL query works
docker-compose exec postfix postmap -q "community@lists.sasalliance.org" \
mysql:/etc/postfix/mysql_virtual_alias_maps.cf
```
## Security Considerations
- SES credentials exposed in docker-compose.yaml (consider using Docker secrets)
- All credentials stored in `.env` file (git-ignored)
- MySQL on private Docker network (no host port mapping)
- API requires Bearer token authentication
- Sender whitelist restricts who can send to lists (@sasalliance.org only)
- SASL password file permissions set to 600 in entrypoint
- TLS encryption enforced for SES relay (`smtp_tls_security_level = encrypt`)
- Only localhost and configured hostname accepted for local delivery
- Web interface requires token for all operations
## Configuration Conventions
- **Hostname Pattern**: `lists.sasalliance.org` for mailing lists, origin domain `sasalliance.org`
- **Virtual Aliases**: Simple format `listname@lists.domain → recipient1, recipient2`
- **Virtual Aliases**: Dynamically queried from MySQL database in real-time
- **SES Region**: EU West 2 (`email-smtp.eu-west-2.amazonaws.com`)
- **Port Mapping**: Standard SMTP port 25 exposed to host
- **Database Tables**: `lists`, `members`, `list_members` (many-to-many design)
- **API Port**: 8000 for REST API
- **Web Port**: 3000 for web interface
## Common Modifications
**Adding Recipients to Existing List (Current):**
Edit `virtual_aliases.cf`, add comma-separated emails, rebuild container.
**Adding Recipients to a List:**
Use web interface or API - changes are immediate!
**New Mailing List (Current):**
Add line to `virtual_aliases.cf`: `newlist@lists.sasalliance.org recipients...`
**Creating New Mailing List:**
1. Via Web: Click "Add List" button, fill in form
2. Via API: POST to /lists endpoint with list details
3. Via MySQL: INSERT into lists table
**Adding New Member:**
1. Via Web: Click "Add Member" button in Members tab
2. Via API: POST to /members endpoint
3. Via MySQL: INSERT into members table
**Managing Subscriptions:**
1. Via Web: Click "Subscriptions" button on member, toggle lists
2. Via API: POST/DELETE to /subscriptions endpoint
3. Via MySQL: INSERT/DELETE in list_members table
**Credential Updates:**
Update `SES_USER`/`SES_PASS` in docker-compose.yaml, restart container.
## Migration Considerations
When implementing the web frontend and database backend:
- Preserve existing `community@lists.sasalliance.org` list during migration
- Consider migration script to import current virtual aliases into database
- Plan for zero-downtime deployment with database-driven config generation
- Web interface should validate email addresses and handle duplicate members
1. Edit `.env` file with new credentials
2. Restart affected services:
- SES: `docker-compose restart postfix`
- MySQL: `docker-compose restart mysql`
- API: `docker-compose restart api`

277
BOUNCE_HANDLING_SETUP.md Normal file
View File

@@ -0,0 +1,277 @@
# SES SNS Bounce Handling Setup (Optional)
**⚠️ NOTICE: Bounce handling is optional and disabled by default.**
This document describes how to configure AWS SES and SNS to handle email bounces automatically in the Mail List Manager.
**Prerequisites:**
- SES production access (not available in sandbox mode)
- Valid HTTPS domain for webhook endpoint
- Bounce handling must be enabled in configuration
## Overview
The system can optionally use AWS Simple Notification Service (SNS) to receive real-time bounce notifications from AWS Simple Email Service (SES). When bounce handling is enabled and an email bounces:
1. SES sends a notification to an SNS topic
2. SNS forwards the notification to your webhook endpoint
3. The API processes the notification and updates the database
4. Members with hard bounces are automatically deactivated
5. Bounce history is tracked and displayed in the UI
## Bounce Status Types
- **Clean**: No bounces recorded
- **Soft Bounce**: Temporary delivery issues (e.g., mailbox full, temporary server issues)
- After 3 soft bounces, the member is marked with soft bounce status
- **Hard Bounce**: Permanent delivery failure (e.g., invalid email address, domain doesn't exist)
- Member is automatically deactivated and cannot receive emails
## Setup Instructions
### 1. Enable Bounce Handling
First, enable bounce handling in your `.env` file:
```bash
# Enable SNS webhook bounce handling
ENABLE_SNS_WEBHOOKS=true
ENABLE_BOUNCE_HANDLING=true
```
Restart the API container after making this change:
```bash
sudo docker-compose restart api
```
### 2. Prerequisites
- AWS account with SES configured and verified
- Your Mail List Manager deployed and accessible via HTTPS (required for SNS webhook)
- Domain or subdomain for webhook (e.g., `https://lists.yourdomain.com`)
### 3. Create SNS Topic
1. Log in to AWS Console and navigate to SNS
2. Click "Create topic"
3. Choose "Standard" topic type
4. Name: `ses-bounce-notifications` (or your preferred name)
5. Display name: `SES Bounce Notifications`
6. Click "Create topic"
7. **Save the Topic ARN** (you'll need it in step 4) arn:aws:sns:eu-west-2:827164363113:ses-bounces
### 4. Subscribe Your Webhook to SNS Topic
1. In the SNS topic details, click "Create subscription"
2. Protocol: `HTTPS`
3. Endpoint: `https://yourdomain.com:8000/webhooks/sns`
- Replace `yourdomain.com` with your actual domain
- The API must be accessible via HTTPS (SNS doesn't support HTTP)
4. Enable raw message delivery: **Unchecked**
5. Click "Create subscription"
6. The subscription will be in "PendingConfirmation" status
### 5. Confirm SNS Subscription
When you create the subscription, SNS will send a `SubscriptionConfirmation` request to your webhook endpoint. The Mail List Manager API automatically confirms this subscription.
1. Check your API logs:
```bash
sudo docker-compose logs -f api
```
2. You should see a log entry indicating the subscription was confirmed
3. In the AWS SNS console, refresh the subscriptions list
4. The status should change from "PendingConfirmation" to "Confirmed"
### 6. Configure SES to Send Bounce Notifications
1. Navigate to AWS SES console
2. Go to "Configuration Sets" (or "Verified identities" > select your domain > "Notifications")
3. For configuration sets:
- Create a new configuration set or select existing
- Add "Event destination"
- Event types: Select **"Bounce"** (and optionally "Complaint")
- Destination: SNS topic
- Select your SNS topic created in step 2
4. For verified identities:
- Select your sending domain/email
- Click "Edit" in the "Notifications" section
- Bounce feedback: Select your SNS topic
- Include original headers: Enabled (optional)
- Click "Save changes"
### 7. Verify Setup
#### Test with a Bounce Simulator
AWS SES provides bounce simulator addresses:
```bash
# From inside Postfix container
docker-compose exec postfix bash
echo "Test bounce" | mail -s "Test" bounce@simulator.amazonses.com
```
Or send to your mailing list with a test recipient:
1. Add `bounce@simulator.amazonses.com` as a member
2. Subscribe to a test list
3. Send an email to the list
#### Check the Results
1. Wait a few minutes for SES to process and send the notification
2. Check API logs:
```bash
sudo docker-compose logs api | grep -i bounce
```
3. Log in to the web UI
4. Go to Members tab
5. Find the test member and click the "Bounces" button
6. You should see the bounce event recorded
### 8. Security Considerations
#### SNS Signature Verification
The webhook endpoint automatically verifies SNS message signatures to ensure notifications are genuine AWS messages. This prevents unauthorized parties from sending fake bounce notifications.
#### HTTPS Requirement
SNS requires HTTPS for webhooks. You'll need:
- Valid SSL/TLS certificate for your domain
- Reverse proxy (e.g., Nginx, Apache) in front of the API container
- Or use AWS API Gateway as a proxy
#### Example Nginx Configuration
```nginx
server {
listen 443 ssl http2;
server_name lists.yourdomain.com;
ssl_certificate /path/to/cert.pem;
ssl_certificate_key /path/to/key.pem;
# Webhook endpoint
location /webhooks/sns {
proxy_pass http://localhost:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Optional: proxy API for web UI
location /api {
proxy_pass http://localhost:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
```
### 9. Managing Bounces in the UI
#### View Bounce Status
In the Members tab, bounced emails are indicated with:
- Warning badge showing bounce count
- Color-coded status (yellow for soft bounce, red for hard bounce)
- Last bounce timestamp
#### View Bounce History
1. Click the "Bounces" button next to a member
2. View detailed bounce history including:
- Bounce type (Permanent, Transient, Undetermined)
- Bounce subtype
- Diagnostic code from the receiving mail server
- Timestamp of each bounce
#### Reset Bounce Status
If a member's email has been corrected or verified:
1. Open the bounce history modal
2. Click "Reset Bounce Status"
3. Confirm the action
4. The member's bounce count is cleared and they can receive emails again
**Note**: Only users with write access (administrators and operators) can reset bounce status.
### 10. Monitoring and Maintenance
#### Check Bounce Logs
```bash
# View all bounces in database
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT * FROM bounce_logs ORDER BY timestamp DESC LIMIT 20;"
# Count bounces by type
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT bounce_type, COUNT(*) as count FROM bounce_logs GROUP BY bounce_type;"
# Find members with bounces
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT name, email, bounce_count, bounce_status FROM members WHERE bounce_count > 0;"
```
#### API Health Check
```bash
# Check if webhook is accessible
curl -X POST https://yourdomain.com:8000/webhooks/sns \
-H "Content-Type: application/json" \
-d '{"Type":"test"}'
```
#### Clean Up Old Bounce Records
Periodically review and clean up old bounce records:
```sql
-- Delete bounce logs older than 90 days
DELETE FROM bounce_logs WHERE created_at < DATE_SUB(NOW(), INTERVAL 90 DAY);
```
## Troubleshooting
### SNS Subscription Not Confirming
- Ensure the API container is running and accessible via HTTPS
- Check API logs for errors
- Verify firewall rules allow HTTPS traffic to port 8000
- Test the endpoint manually: `curl https://yourdomain.com:8000/health`
### Bounces Not Being Recorded
1. Verify SNS topic is receiving messages:
- Check SNS topic metrics in AWS Console
2. Verify subscription is active:
- Check subscription status in SNS console
3. Check API logs for webhook errors:
```bash
sudo docker-compose logs api | grep -i "sns\|bounce"
```
4. Test signature verification:
- Temporarily add debug logging to the webhook endpoint
### Members Not Being Deactivated
- Check if bounce type is "Permanent"
- Review member's bounce_status in database:
```bash
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT * FROM members WHERE email='problem@example.com';"
```
- Verify bounce processing logic in API logs
### SSL Certificate Issues
If using self-signed certificates, SNS will reject the webhook. You must use:
- Valid certificate from a trusted CA (Let's Encrypt, etc.)
- Or use AWS Certificate Manager with API Gateway
## Additional Resources
- [AWS SES Bounce Handling](https://docs.aws.amazon.com/ses/latest/dg/event-publishing-retrieving-sns.html)
- [AWS SNS HTTPS Subscriptions](https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html)
- [SES Bounce Types](https://docs.aws.amazon.com/ses/latest/dg/notification-contents.html#bounce-types)

240
CODE_REVIEW_FINDINGS.md Normal file
View File

@@ -0,0 +1,240 @@
# Code Review Findings & Fixes
## Date: 13 October 2025
## Critical Issues Found and Fixed
### 1. ✅ FIXED: Header Merging Bug in API Client
**Severity:** HIGH
**Location:** `web/static/js/api.js` - `request()` method
**Issue:**
The header merging was incorrect. When passing custom headers in options, they were being overridden by the default headers due to incorrect spread operator order:
```javascript
// BEFORE (BUGGY):
const config = {
headers: { ...this.headers },
...options // This overwrites headers!
};
```
**Fix:**
```javascript
// AFTER (FIXED):
const config = {
...options,
headers: {
...this.headers,
...(options.headers || {}) // Custom headers take precedence
}
};
```
**Impact:**
- The `login()` method was trying to remove the Authorization header for unauthenticated requests, but it was being overridden
- Could have caused authentication issues or unexpected behavior
---
### 2. ✅ FIXED: Missing API Client Methods
**Severity:** HIGH
**Location:** `web/static/js/api.js`
**Issue:** Multiple methods were being called by the UI but didn't exist in the API client:
- `login(username, password)` - ❌ Missing
- `logout()` - ❌ Missing
- `getCurrentUser()` - ❌ Missing
- `getUsers()` - ❌ Missing
- `createUser(userData)` - ❌ Missing
- `updateUser(userId, userData)` - ❌ Missing
- `deleteUser(userId)` - ❌ Missing
- `getMemberBounces(memberId)` - ❌ Missing
- `resetBounceStatus(memberId)` - ❌ Missing
**Fix:** Added all missing methods with proper implementation
---
### 3. ✅ FIXED: Incorrect API Base URL for Production
**Severity:** HIGH
**Location:** `web/static/js/api.js` - `getBaseURL()` method
**Issue:**
When running behind a reverse proxy (Caddy), the API client was trying to connect to `https://lists.sasalliance.org:8000` which:
- Port 8000 is not exposed through Caddy
- Would cause CORS issues
- Would fail all API requests
**Fix:**
```javascript
// Production detection
if (hostname === 'localhost' || hostname === '127.0.0.1') {
return `${protocol}//${hostname}:8000`; // Development
}
// Production - use reverse proxy path
return `${protocol}//${hostname}/api`;
```
---
## Potential Issues (Not Critical, But Worth Noting)
### 4. ⚠️ CORS Configuration
**Severity:** MEDIUM
**Location:** `api/main.py`
**Current Configuration:**
```python
app.add_middleware(
CORSMiddleware,
allow_origins=["*"], # Allows all origins
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
```
**Issue:**
- `allow_origins=["*"]` with `allow_credentials=True` is not allowed by browsers
- Currently working because Caddy reverse proxy makes same-origin requests
- If you ever need direct API access from different origins, this will fail
**Recommendation:**
```python
# For production
allow_origins=[
"https://lists.sasalliance.org",
"http://localhost:3000", # For development
],
```
---
### 5. ✅ GOOD: Authentication Token Storage
**Severity:** INFO
**Location:** `web/static/js/app.js`
**Current Implementation:**
```javascript
localStorage.setItem('authToken', response.access_token);
```
**Assessment:**
- ✅ Proper use of localStorage for JWT tokens
- ✅ Token is cleared on logout
- ✅ Token is validated on page load
- ✅ Expired tokens trigger automatic logout
**Note:** localStorage is appropriate for JWT tokens. HttpOnly cookies would be more secure but require different architecture.
---
### 6. ✅ GOOD: Error Handling
**Severity:** INFO
**Location:** `web/static/js/ui.js` - `handleError()` method
**Assessment:**
- ✅ Proper handling of 401 (auth) errors with automatic logout
- ✅ Proper handling of 403 (forbidden) errors
- ✅ Proper handling of 404 (not found) errors
- ✅ Proper handling of 500 (server) errors
- ✅ User-friendly error messages
---
### 7. ✅ GOOD: Role-Based Access Control
**Severity:** INFO
**Location:** Multiple files
**Assessment:**
- ✅ Backend enforces RBAC with decorators (`require_admin()`, `require_write_access()`, etc.)
- ✅ Frontend checks user roles before showing UI elements
- ✅ Three roles properly implemented: administrator, operator, read-only
- ✅ UI elements are hidden/shown based on permissions
---
### 8. ⚠️ Token Expiration Handling
**Severity:** LOW
**Location:** `api/main.py` and `web/static/js/app.js`
**Current Configuration:**
- JWT tokens expire in 30 minutes
- Session records expire in 24 hours (but not enforced in JWT)
**Potential Issue:**
- Users will be logged out after 30 minutes without warning
- No token refresh mechanism
**Recommendation (Future Enhancement):**
Consider implementing:
- Token refresh endpoint
- Warning before token expiration
- Silent token refresh in background
---
### 9. ✅ GOOD: SQL Injection Prevention
**Severity:** INFO
**Location:** `api/main.py`
**Assessment:**
- ✅ All SQL queries use parameterized statements with `%s` placeholders
- ✅ No string concatenation in SQL queries
- ✅ MySQL connector properly escapes parameters
---
### 10. ✅ GOOD: Password Security
**Severity:** INFO
**Location:** `api/main.py`
**Assessment:**
- ✅ Uses bcrypt for password hashing (`passlib.context.CryptContext`)
- ✅ Passwords are never stored in plain text
- ✅ Passwords are never logged or exposed in API responses
---
## Testing Recommendations
1. **Hard refresh browser cache** after updates (Ctrl+Shift+R)
2. **Test authentication flow:**
- Login with valid credentials
- Login with invalid credentials
- Token expiration after 30 minutes
- Logout functionality
3. **Test role-based access:**
- Administrator can see Users tab
- Operator can modify lists/members but not users
- Read-only can view but not modify
4. **Test bounce handling:**
- View bounce history
- Reset bounce status
5. **Test bulk import:**
- CSV upload
- Subscription assignment
---
## Summary
**3 Critical Bugs Fixed:**
1. Header merging bug in request method
2. Missing API client methods
3. Incorrect production API URL
**All Core Functionality Working:**
- Authentication and authorization
- CRUD operations for lists, members, and users
- Subscription management
- Bounce tracking
- Bulk import
⚠️ **Minor Improvements Suggested:**
1. Tighten CORS policy for production
2. Consider token refresh mechanism
3. Add user session timeout warnings
**Overall Assessment:** The codebase is now production-ready with proper security, error handling, and functionality. The critical bugs have been fixed and deployed.

View File

@@ -0,0 +1,243 @@
# Email-Based Bounce Handling Setup
This document explains the email-based bounce handling system implemented as an alternative to SNS webhooks for environments without SES production access.
## Overview
The system processes email bounces directly within the Postfix container by:
1. Rewriting return paths to direct bounces to a processing address
2. Processing bounce emails via Python script
3. Updating member bounce statistics in MySQL database
4. Automatically disabling members with excessive bounces
2. Setting up an alias that pipes bounce emails to a Python processing script
3. The script parses bounce emails, extracts bounced addresses, and updates the database
4. Members with hard bounces are automatically deactivated
5. Bounce history is tracked and displayed in the UI (same as SNS method)
## Advantages
- **Works with SES Sandbox**: No production SES access required
- **No External Dependencies**: Doesn't require SNS, webhooks, or HTTPS domains
- **Self-Contained**: All processing happens within the existing containers
- **Real-time Processing**: Bounces are processed as soon as emails arrive
- **Compatible**: Uses the same database schema and UI as SNS bounce handling
## Configuration
### 1. Enable Email Bounce Processing
In your `.env` file:
```bash
# Enable email-based bounce processing
ENABLE_EMAIL_BOUNCE_PROCESSING=true
# This will automatically enable bounce handling features
ENABLE_BOUNCE_HANDLING=true # Automatically set to true when email processing is enabled
```
### 2. Restart the System
```bash
sudo docker-compose down
sudo docker-compose up --build -d
```
### 3. Verify Configuration
Check that email bounce processing is enabled:
```bash
curl -s http://localhost:8000/config | jq .
```
Expected output:
```json
{
"bounce_handling_enabled": true,
"sns_webhooks_enabled": false,
"email_bounce_processing_enabled": true
}
```
## How It Works
### Postfix Configuration
The system configures Postfix with:
- `bounce_notice_recipient = bounces@lists.sasalliance.org`
- `2bounce_notice_recipient = bounces@lists.sasalliance.org`
- `error_notice_recipient = bounces@lists.sasalliance.org`
### Aliases Configuration
The `bounces` address is configured to pipe emails to the processing script:
```
bounces: "|/usr/local/bin/process-bounce.py"
```
### Bounce Processing Script
The Python script (`/usr/local/bin/process-bounce.py`):
1. **Reads bounce emails** from stdin (via pipe)
2. **Parses email content** using multiple regex patterns to extract bounced addresses
3. **Analyzes bounce type** based on SMTP error codes:
- 5xx codes = Permanent bounces
- 4xx codes = Transient bounces
- Unknown = Undetermined
4. **Updates database** using the same schema as SNS bounce handling:
- Logs bounce in `bounce_logs` table
- Updates member `bounce_count`, `bounce_status`, and `last_bounce_at`
- Deactivates members with permanent bounces
- Marks members with soft bounce status after 3 transient bounces
## Testing
### Test the Processing Script
You can test the bounce processing script in test mode:
```bash
# Test with sample bounce email
sudo docker-compose exec postfix /usr/local/bin/process-bounce.py --test
```
Expected output:
```
2025-10-14 15:49:16,041 - bounce-processor - INFO - Starting bounce processing
2025-10-14 15:49:16,041 - bounce-processor - INFO - Running in test mode with sample bounce email
2025-10-14 15:49:16,050 - bounce-processor - INFO - Extracted addresses: ['testuser@example.com']
2025-10-14 15:49:16,050 - bounce-processor - INFO - Test mode - would process 1 bounce(s):
2025-10-14 15:49:16,050 - bounce-processor - INFO - {'email': 'testuser@example.com', 'bounce_type': 'Permanent', 'bounce_subtype': 'General', 'diagnostic_code': '', 'timestamp': '2025-10-14 15:49:16'}
```
### Test with Real Bounce Email
To test with a real bounce:
1. Send an email to a non-existent address via your mailing list
2. Wait for the bounce to be processed
3. Check the database for bounce logs:
```bash
# Check bounce logs
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;"
# Check member bounce status
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT member_id, email, bounce_count, bounce_status, last_bounce_at FROM members WHERE bounce_count > 0;"
```
### View in Web Interface
1. Open http://localhost:3000
2. Navigate to the Members tab
3. Look for bounce badges and bounce counts next to member names
4. Click the "Bounces" button next to a member to view bounce history
## Supported Bounce Email Formats
The processing script recognizes these bounce patterns:
### SMTP Error Codes
- 550, 551, 553, 552, 554 (permanent failures)
- 450, 451, 452 (temporary failures)
### Delivery Status Notification (DSN)
- `Final-Recipient:` headers
- `Original-Recipient:` headers
### Common Bounce Messages
- "user unknown"
- "does not exist"
- "not found"
- "mailbox unavailable"
- "recipient rejected"
## Monitoring
### View Processing Logs
```bash
# View bounce processing logs
sudo docker-compose logs postfix | grep bounce-processor
# Follow logs in real-time
sudo docker-compose logs -f postfix | grep bounce-processor
```
### Check Aliases Configuration
```bash
# Verify aliases are configured correctly
sudo docker-compose exec postfix cat /etc/aliases
# Check alias database
sudo docker-compose exec postfix postmap -q bounces /etc/aliases
```
## Troubleshooting
### Bounces Not Being Processed
1. **Check aliases configuration:**
```bash
sudo docker-compose exec postfix cat /etc/aliases
```
2. **Verify script permissions:**
```bash
sudo docker-compose exec postfix ls -la /usr/local/bin/process-bounce.py
```
3. **Test script manually:**
```bash
sudo docker-compose exec postfix /usr/local/bin/process-bounce.py --test
```
4. **Check Postfix logs:**
```bash
sudo docker-compose logs postfix | grep -i bounce
```
### Database Connection Issues
1. **Check environment variables:**
```bash
sudo docker-compose exec postfix env | grep MYSQL
```
2. **Test database connection:**
```bash
sudo docker-compose exec postfix python3 -c "import pymysql; print('PyMySQL available')"
```
### Script Errors
View detailed error logs:
```bash
sudo docker-compose logs postfix | grep -A 10 -B 10 "bounce-processor.*ERROR"
```
## Comparison with SNS Webhooks
| Feature | Email-Based | SNS Webhooks |
|---------|-------------|--------------|
| **SES Requirement** | Sandbox OK | Production access required |
| **External Dependencies** | None | SNS, HTTPS domain |
| **Processing Speed** | Real-time | Real-time |
| **Setup Complexity** | Low | High |
| **Reliability** | High | High |
| **Bounce Detection** | Regex-based | AWS-provided |
| **Cost** | Free | SNS charges apply |
## Next Steps
1. **Monitor bounce processing** to ensure it's working correctly
2. **Review bounce patterns** in the logs to improve detection if needed
3. **Set up bounce notification alerts** (optional)
4. **Consider upgrading to SNS webhooks** when SES production access is available
Email-based bounce handling provides a robust alternative that works immediately with any SES configuration while providing the same bounce management features as the SNS webhook method.

View File

@@ -0,0 +1,212 @@
# Email-Based Bounce Handling Setup (Comprehensive)
This document explains the complete email-based bounce handling system implemented as an alternative to SNS webhooks for environments without SES production access.
## Overview
The system processes email bounces directly within the Postfix container by:
1. Rewriting return paths to direct bounces to a processing address
2. Processing bounce emails via Python script
3. Updating member bounce statistics in MySQL database
4. Automatically disabling members with excessive bounces
## Configuration
### Environment Variables (in .env)
```bash
# Bounce handling feature flags
ENABLE_BOUNCE_HANDLING=true # Master switch for bounce functionality
ENABLE_EMAIL_BOUNCE_PROCESSING=true # Enable email-based processing
ENABLE_SNS_WEBHOOKS=false # Disable SNS webhooks (optional)
# Database settings (required for bounce processing)
MYSQL_ROOT_PASSWORD=your_root_password
MYSQL_DATABASE=maillist
MYSQL_USER=maillist
MYSQL_PASSWORD=your_maillist_password
```
### Key Components
#### 1. Return Path Rewriting (`postfix/smtp_generic`)
Routes all bounces to the processing address:
```
@lists.sasalliance.org bounces@lists.sasalliance.org
@sasalliance.org bounces@lists.sasalliance.org
```
#### 2. Bounce Processing Script (`postfix/process-bounce.py`)
Python script that:
- Parses bounce emails for recipient addresses and bounce types
- Updates bounce counts in MySQL database
- Automatically disables members after 5 hard bounces
- Logs all bounce events
#### 3. Postfix Integration (`postfix/main.cf.template`)
```
# Return path rewriting for outbound mail
smtp_generic_maps = hash:/etc/postfix/smtp_generic
# Bounce processing
bounce_notice_recipient = bounces@lists.sasalliance.org
```
#### 4. Email Aliases (`postfix/entrypoint.sh`)
```
# Route bounces to processing script
bounces: "|/usr/local/bin/python3 /etc/postfix/process-bounce.py"
```
## How It Works
### Outbound Email Flow
1. User sends email to `community@lists.sasalliance.org`
2. Postfix expands to member list via MySQL
3. Email sent via SES with return path rewritten to `bounces@lists.sasalliance.org`
4. If delivery fails, bounce goes to bounce processing address
### Bounce Processing Flow
1. Bounce email arrives at `bounces@lists.sasalliance.org`
2. Postfix pipes email to `process-bounce.py` script
3. Script parses bounce for recipient and bounce type
4. Database updated with bounce information
5. Member automatically disabled if hard bounce threshold reached
### Bounce Types Detected
- **Hard Bounces**: Permanent failures (5.x.x SMTP codes)
- Invalid email addresses
- Domain doesn't exist
- Mailbox doesn't exist
- **Soft Bounces**: Temporary failures (4.x.x SMTP codes)
- Mailbox full
- Temporary server issues
- Rate limiting
## Database Schema
### Bounce Logs Table
```sql
CREATE TABLE bounce_logs (
id INT PRIMARY KEY AUTO_INCREMENT,
email VARCHAR(255) NOT NULL,
bounce_type ENUM('hard', 'soft', 'complaint') NOT NULL,
bounce_reason TEXT,
bounced_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
raw_message TEXT,
INDEX idx_email (email),
INDEX idx_bounced_at (bounced_at)
);
```
### Members Table (bounce tracking fields)
```sql
ALTER TABLE members ADD COLUMN bounce_count INT DEFAULT 0;
ALTER TABLE members ADD COLUMN last_bounce_at TIMESTAMP NULL;
ALTER TABLE members ADD COLUMN active BOOLEAN DEFAULT true;
```
## Testing Bounce Handling
### 1. Test Return Path Configuration
```bash
# Check that return path rewriting is working
sudo docker-compose exec postfix postconf smtp_generic_maps
sudo docker-compose exec postfix postmap -q "test@lists.sasalliance.org" hash:/etc/postfix/smtp_generic
```
### 2. Simulate Bounce Email
```bash
# Send test bounce to processing script
echo "Subject: Delivery Status Notification (Failure)
From: MAILER-DAEMON@ses.amazonaws.com
To: bounces@lists.sasalliance.org
The following message could not be delivered:
Recipient: test@example.com
Reason: 550 5.1.1 User unknown" | docker-compose exec -T postfix mail -s "Test Bounce" bounces@lists.sasalliance.org
```
### 3. Check Bounce Processing
```bash
# View bounce logs
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY bounced_at DESC LIMIT 5;"
# Check member bounce counts
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT email, bounce_count, last_bounce_at, active FROM members WHERE bounce_count > 0;"
```
## Monitoring and Maintenance
### View Processing Logs
```bash
# Monitor bounce processing
sudo docker-compose logs -f postfix | grep -E "(bounce|process-bounce)"
# Check API bounce handling status
curl -H "Authorization: Bearer $API_TOKEN" http://localhost:8000/config
```
### Reset Member Bounce Count
```bash
# Via API
curl -X POST http://localhost:8000/members/{member_id}/reset-bounces \
-H "Authorization: Bearer $API_TOKEN"
# Via Database
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "UPDATE members SET bounce_count=0, active=true WHERE email='user@example.com';"
```
## Troubleshooting
### Common Issues
#### Bounces Not Being Processed
1. Check that `bounces` alias exists: `sudo docker-compose exec postfix cat /etc/aliases | grep bounces`
2. Verify Python script permissions: `sudo docker-compose exec postfix ls -la /etc/postfix/process-bounce.py`
3. Test script manually: `sudo docker-compose exec postfix python3 /etc/postfix/process-bounce.py --test`
#### Return Path Not Rewritten
1. Check smtp_generic configuration: `sudo docker-compose exec postfix postconf smtp_generic_maps`
2. Verify map file exists: `sudo docker-compose exec postfix ls -la /etc/postfix/smtp_generic*`
3. Test mapping: `sudo docker-compose exec postfix postmap -q "test@lists.sasalliance.org" hash:/etc/postfix/smtp_generic`
#### Database Connection Issues
1. Check PyMySQL installation: `sudo docker-compose exec postfix python3 -c "import pymysql; print('OK')"`
2. Test database connection: `sudo docker-compose exec postfix python3 -c "import pymysql; pymysql.connect(host='mysql', user='maillist', password='your_password', database='maillist')"`
3. Verify network connectivity: `sudo docker-compose exec postfix ping mysql`
### Log Analysis
```bash
# Postfix logs
sudo docker-compose logs postfix | grep -E "(bounce|MAILER-DAEMON|process-bounce)"
# MySQL connection logs
sudo docker-compose logs postfix | grep -E "(pymysql|mysql)"
# SES relay logs
sudo docker-compose logs postfix | grep -E "(relay|sent|deferred)"
```
## Security Considerations
- Bounce processing script runs with limited privileges
- Database credentials secured in environment variables
- Bounce emails contain sensitive delivery information - logs are rotated
- Return path rewriting prevents bounce loops
- Processing script validates email format before database updates
## Performance Notes
- Bounce processing is asynchronous (doesn't block email delivery)
- Database queries are indexed for bounce lookups
- Bounce logs should be periodically archived for large volumes
- SMTP generic maps are cached by Postfix for performance
## Advantages over SNS Webhooks
- **Works with SES Sandbox**: No production SES access required
- **No External Dependencies**: Doesn't require SNS, webhooks, or HTTPS domains
- **Self-Contained**: All processing happens within existing containers
- **Real-time Processing**: Bounces processed as emails arrive
- **Compatible**: Uses same database schema and UI as SNS method

375
README.md
View File

@@ -1,18 +1,23 @@
# Mail List Manager
A containerized mailing list management system built around Postfix as an SMTP relay through Amazon SES.
A complete containerized mailing list management system with web interface, REST API, MySQL database, and Postfix mail server configured as an SMTP relay through Amazon SES.
## Architecture
**Current (Phase 1):** Static configuration with environment-based credentials
- Postfix container configured as SES relay
- Static virtual aliases for mailing list distribution
- Environment variable configuration for security
**Multi-Service Architecture:**
- **Web Frontend** - Modern responsive interface for managing lists and members (Port 3000)
- **REST API** - FastAPI backend with token authentication (Port 8000)
- **MySQL Database** - Stores lists, members, and subscriptions with real-time integration
- **Postfix Mail Server** - SMTP server with native MySQL integration for dynamic list expansion (Port 25)
- **Amazon SES** - Outbound mail relay for reliable delivery
**Planned (Phase 2+):** Web interface with SQL backend
- Web frontend for list management (view/add/remove members)
- SQL database for member storage
- Dynamic Postfix configuration generation
**Key Features:**
- Real-time list updates (no Postfix reload needed)
- Native Postfix MySQL queries for instant list expansion
- Token-based API authentication
- Sender whitelisting for authorized board members
- Private Docker network for security
- Complete web-based management interface
## Quick Start
@@ -21,72 +26,239 @@ A containerized mailing list management system built around Postfix as an SMTP r
cp .env.example .env
```
2. Edit `.env` with your SES credentials and configuration:
2. Edit `.env` with your credentials:
```bash
# Required: Your SES credentials
# SES Credentials (Required)
SES_USER=your_ses_access_key
SES_PASS=your_ses_secret_key
# MySQL Credentials (Required)
MYSQL_ROOT_PASSWORD=secure_root_password
MYSQL_PASSWORD=secure_maillist_password
# API Authentication (Required)
API_TOKEN=your_secure_api_token
# Optional: SMTP configuration (defaults to EU West 2)
SMTP_HOST=email-smtp.eu-west-2.amazonaws.com
SMTP_PORT=587
```
3. Build and start the mail server:
3. Build and start all services:
```bash
docker-compose up --build
sudo docker-compose up --build
```
4. Test mail delivery:
```bash
# From inside container
docker-compose exec postfix bash
echo "Test message" | mail -s "Subject" community@lists.sasalliance.org
# Check logs
docker-compose logs -f postfix
4. Access the web interface:
```
http://localhost:3000
```
Enter your API_TOKEN to authenticate and start managing lists!
5. Alternative: Use the REST API directly:
```bash
# Health check
curl http://localhost:8000/health
# Get all lists (requires authentication)
curl -H "Authorization: Bearer your_api_token" http://localhost:8000/lists
```
## Services
### Web Interface (Port 3000)
- Modern, responsive interface for managing mailing lists and members
- Token-based authentication
- Real-time updates and feedback
- See `web/README.md` for details
### REST API (Port 8000)
- Complete CRUD operations for lists, members, and subscriptions
- Bearer token authentication
- Interactive documentation at http://localhost:8000/docs
- See `api/README.md` for API reference
### MySQL Database (Internal Only)
- Stores all mailing list and member data
- Automatically initialized with schema
- Native Postfix integration for real-time queries
- See `database/README.md` for schema and management
### Postfix Mail Server (Port 25)
- Accepts mail for lists.sasalliance.org
- Queries MySQL for list expansion
- Relays through Amazon SES
- Sender whitelist: @sasalliance.org domain
- Changes take effect immediately (no restart needed)
## Configuration
### Adding Mailing Lists (Current)
### Managing Lists and Members
Edit `postfix/virtual_aliases.cf`:
```
newlist@lists.sasalliance.org recipient1@domain.com, recipient2@domain.com
```
**Via Web Interface (Recommended):**
1. Open http://localhost:3000
2. Enter your API token
3. Use the intuitive interface to:
- Create/edit/delete mailing lists
- Add/edit/remove members
- Manage subscriptions with toggle switches
- View member counts and list details
Then rebuild the container:
**Via REST API:**
```bash
docker-compose up --build
# Create a new list
curl -X POST http://localhost:8000/lists \
-H "Authorization: Bearer your_token" \
-H "Content-Type: application/json" \
-d '{
"list_name": "Developers",
"list_email": "dev@lists.sasalliance.org",
"description": "Developer discussions",
"active": true
}'
# Add a new member
curl -X POST http://localhost:8000/members \
-H "Authorization: Bearer your_token" \
-H "Content-Type: application/json" \
-d '{
"name": "John Doe",
"email": "john@example.com",
"active": true
}'
# Subscribe member to list
curl -X POST http://localhost:8000/subscriptions \
-H "Authorization: Bearer your_token" \
-H "Content-Type: application/json" \
-d '{
"list_email": "dev@lists.sasalliance.org",
"member_email": "john@example.com"
}'
```
### Domain Configuration
**Via Direct Database Access:**
```bash
# Connect to MySQL
docker-compose exec mysql mysql -u maillist -p maillist
The system is configured for:
- **Hostname**: `lists.sasalliance.org` (mailing lists)
# Run SQL queries (see database/README.md for examples)
```
### Mail Server Configuration
### Mail Server Configuration
The Postfix mail server is configured for:
- **Hostname**: `lists.sasalliance.org` (accepts mail for this domain)
- **Origin Domain**: `sasalliance.org`
- **SES Region**: EU West 2 (configurable via `SMTP_HOST`)
- **Sender Whitelist**: @sasalliance.org (only authorized board members can send to lists)
- **Dynamic Lists**: Postfix queries MySQL in real-time for list members
### Testing Mail Delivery
```bash
# From inside Postfix container
docker-compose exec postfix bash
echo "Test message" | mail -s "Test Subject" community@lists.sasalliance.org
# Check delivery logs
docker-compose logs -f postfix | grep -E "(sent|bounced|deferred)"
# Verify MySQL query works
docker-compose exec postfix postmap -q "community@lists.sasalliance.org" \
mysql:/etc/postfix/mysql_virtual_alias_maps.cf
```
### Bounce Handling (Optional)
**Email bounce handling is optional and disabled by default.**
**Three Configuration Options:**
1. **SNS Webhooks** (Requires SES Production Access):
```bash
# In .env file
ENABLE_SNS_WEBHOOKS=true
ENABLE_BOUNCE_HANDLING=true
```
- Real-time bounce notifications via AWS SNS webhooks
- Automatic member deactivation for hard bounces
- Bounce history tracking and management in web UI
- Requires valid HTTPS domain and SES production access
- See `BOUNCE_HANDLING_SETUP.md` for complete setup
2. **Email-Based Processing** (Works with SES Sandbox):
```bash
# In .env file
ENABLE_EMAIL_BOUNCE_PROCESSING=true
ENABLE_BOUNCE_HANDLING=true # Automatically enabled
```
- Automatic bounce processing via email parsing
- Same bounce management features as SNS webhooks
- Works with SES sandbox accounts (no production access needed)
- Self-contained processing within existing containers
- See `EMAIL_BOUNCE_HANDLING_SETUP.md` for complete setup
3. **Disabled** (Manual Processing Only):
```bash
# In .env file (or leave these commented out)
ENABLE_SNS_WEBHOOKS=false
ENABLE_BOUNCE_HANDLING=false
ENABLE_EMAIL_BOUNCE_PROCESSING=false
```
- No automatic bounce processing
- Manual bounce management via email notifications
- Bounce-related UI elements are hidden
**When bounce handling is disabled:**
- `/webhooks/sns` endpoint is not registered
- Bounce history endpoints return empty results
- Web UI hides bounce badges and bounce management buttons
- No automatic member deactivation occurs
## Security
- SES credentials are stored in `.env` (git-ignored)
- SASL password files have restricted permissions (600)
- TLS encryption enforced for SES relay
- Only localhost and configured hostname accepted for local delivery
- **Environment Variables**: All credentials stored in `.env` (git-ignored)
- **Private Network**: MySQL communicates with Postfix/API on internal Docker network only
- **Token Authentication**: API requires Bearer token for all write operations
- **Sender Whitelist**: Only @sasalliance.org addresses can send to mailing lists
- **TLS Encryption**: Enforced for SES relay connections
- **File Permissions**: SASL password files have restricted permissions (600)
- **No External MySQL**: Database is not exposed to host (no port mapping)
## Development
### Project Structure
```
├── docker-compose.yaml # Service orchestration
├── docker-compose.yaml # Multi-service orchestration
├── .env # Environment configuration (not in git)
├── postfix/
│ ├── Dockerfile # Postfix container build
│ ├── entrypoint.sh # Runtime configuration processing
├── main.cf.template # Postfix main configuration template
├── sasl_passwd.template # SES authentication template
└── virtual_aliases.cf # Static mailing list definitions
├── web/ # Web frontend (Nginx + HTML/CSS/JS)
│ ├── index.html
│ ├── static/
│ ├── css/style.css
│ └── js/
│ ├── api.js
│ │ ├── ui.js
│ │ └── app.js
│ ├── nginx.conf
│ └── Dockerfile
├── api/ # REST API (FastAPI)
│ ├── main.py
│ ├── requirements.txt
│ └── Dockerfile
├── database/ # MySQL schema and docs
│ ├── schema.sql
│ └── README.md
├── postfix/ # Mail server configuration
│ ├── Dockerfile
│ ├── entrypoint.sh
│ ├── main.cf.template
│ ├── mysql_virtual_alias_maps.cf
│ ├── sasl_passwd.template
│ └── sender_access
└── .github/
└── copilot-instructions.md # AI agent guidance
```
@@ -96,28 +268,131 @@ The system is configured for:
- `SES_PASS`: AWS SES secret access key
- `SMTP_HOST`: SMTP server hostname (default: email-smtp.eu-west-2.amazonaws.com)
- `SMTP_PORT`: SMTP server port (default: 587)
- `MYSQL_ROOT_PASSWORD`: MySQL root password
- `MYSQL_DATABASE`: Database name (default: maillist)
- `MYSQL_USER`: MySQL user (default: maillist)
- `MYSQL_PASSWORD`: MySQL user password
- `API_TOKEN`: Bearer token for API authentication
### Service Dependencies
```
mysql (healthcheck) → postfix, api → web
```
- MySQL must be healthy before Postfix/API start
- Web frontend depends on API being available
- All services communicate on private Docker network
### Debugging
Monitor mail delivery:
Monitor services:
```bash
# View all logs
# View all service logs
docker-compose logs -f
# Filter for delivery status
# View specific service
docker-compose logs -f postfix
docker-compose logs -f api
docker-compose logs -f web
docker-compose logs -f mysql
# Filter for mail delivery status
docker-compose logs postfix | grep -E "(sent|bounced|deferred)"
# Check Postfix queue
docker-compose exec postfix postqueue -p
# Test MySQL connectivity from Postfix
docker-compose exec postfix postmap -q "community@lists.sasalliance.org" \
mysql:/etc/postfix/mysql_virtual_alias_maps.cf
# Check API health
curl http://localhost:8000/health
# Access MySQL directly
docker-compose exec mysql mysql -u maillist -p maillist
```
### Development Workflow
1. **Make changes** to code or configuration
2. **Rebuild affected service**:
```bash
sudo docker-compose up --build -d web # Just web frontend
sudo docker-compose up --build -d api # Just API
sudo docker-compose up --build -d postfix # Just mail server
```
3. **Check logs** for errors:
```bash
docker-compose logs -f [service-name]
```
4. **Test changes** via web interface or API
### Common Tasks
**Reset database completely:**
```bash
docker-compose down -v # Removes volumes
docker-compose up -d # Reinitializes from schema.sql
```
**Update API token:**
1. Edit `.env` and change `API_TOKEN`
2. Restart API: `docker-compose restart api`
3. Use new token in web interface
**Add authorized sender domain:**
1. Edit `postfix/sender_access`
2. Add line: `newdomain.com OK`
3. Rebuild: `docker-compose up --build -d postfix`
## How It Works
1. **User manages lists** via web interface (http://localhost:3000)
2. **Web frontend** makes authenticated API calls to FastAPI backend
3. **API updates MySQL** database with new lists/members/subscriptions
4. **Email arrives** at Postfix for `someone@lists.sasalliance.org`
5. **Postfix queries MySQL** in real-time using native MySQL support
6. **MySQL returns** comma-separated list of active member emails
7. **Postfix expands** the alias and delivers to all members via SES
8. **Changes take effect immediately** - no restart or reload needed!
## Features
- ✅ **Real-time Updates** - Changes to lists/members take effect immediately
- ✅ **Native MySQL Integration** - Postfix queries database directly (no scripts)
- ✅ **Web Interface** - Modern, responsive UI for easy management
- ✅ **REST API** - Complete programmatic access with token auth
- ✅ **Sender Whitelist** - Only authorized domains can send to lists
- ✅ **SES Integration** - Reliable email delivery through Amazon SES
- ✅ **Bounce Handling** - Automatic tracking and management of email bounces via SNS
- ✅ **Secure** - Private Docker network, token auth, environment-based credentials
- ✅ **Flexible** - Manage via web, API, or direct database access
- ✅ **Scalable** - Database-driven architecture supports many lists and members
## Documentation
- **Web Interface**: See `web/README.md` for frontend features and usage
- **REST API**: See `api/README.md` for complete API reference
- **Database**: See `database/README.md` for schema and SQL examples
- **Bounce Handling**: See `BOUNCE_HANDLING_SETUP.md` for SNS configuration
- **AI Agents**: See `.github/copilot-instructions.md` for development guidance
## Roadmap
- [ ] Web frontend for mailing list management
- [ ] SQL database backend for member storage
- [ ] Dynamic configuration generation from database
- [ ] Multi-service Docker Compose architecture
- [ ] Migration tools for static → dynamic configuration
- [x] Web frontend for mailing list management
- [x] SQL database backend for member storage
- [x] Dynamic configuration with native Postfix MySQL
- [x] Multi-service Docker Compose architecture
- [x] REST API with authentication
- [x] Sender whitelist for authorized domains
- [x] Bounce handling with SES SNS integration
- [ ] Email verification workflow for new members
- [ ] Subscription confirmation (double opt-in)
- [ ] List archive functionality
- [ ] Unsubscribe links in emails
- [ ] Rate limiting and anti-spam measures
- [ ] HTTPS support for web interface
- [ ] Admin roles and permissions
## License

398
SES_BOUNCE_TESTING_GUIDE.md Normal file
View File

@@ -0,0 +1,398 @@
# Testing Bounces from SES Sandbox
AWS SES provides **built-in bounce simulator addresses** that work even in sandbox mode. This guide shows you how to use them to test your bounce handling.
## Quick Answer
Send email to these special AWS addresses to simulate different bounce types:
### Hard Bounce (Permanent - Invalid Address)
```bash
bounce@simulator.amazonses.com
```
### Soft Bounce (Temporary - Mailbox Full)
```bash
ooto@simulator.amazonses.com
```
### Complaint (Spam Report)
```bash
complaint@simulator.amazonses.com
```
### Successful Delivery (No Bounce)
```bash
success@simulator.amazonses.com
```
## Step-by-Step Testing Guide
### Option 1: Using Your Mailing Lists (Recommended)
This tests the complete flow: Postfix → SES → SNS → API
1. **Add the simulator address as a member:**
```bash
# Using the API
curl -X POST http://localhost:8000/members \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "Bounce Test",
"email": "bounce@simulator.amazonses.com",
"active": true
}'
```
Or use the Web UI:
- Go to http://localhost:3000
- Members tab → Add Member
- Name: `Bounce Test`
- Email: `bounce@simulator.amazonses.com`
2. **Subscribe to a test list:**
- Click "Subscriptions" button for the test member
- Toggle on one of your mailing lists
3. **Send email to the list:**
```bash
# From inside Postfix container
sudo docker compose exec postfix bash
echo "This will bounce" | mail -s "Test Bounce" community@lists.sasalliance.org
exit
```
Replace `community@lists.sasalliance.org` with your actual list email.
4. **Wait 30-60 seconds** for:
- Email to be sent via SES
- SES to process the bounce
- SNS to send notification to your webhook
5. **Check the results:**
**Watch API logs in real-time:**
```bash
sudo docker compose logs api -f
```
You should see:
```
============================================================
SNS Webhook Request Received
============================================================
Content-Type: text/plain; charset=UTF-8
User-Agent: Amazon Simple Notification Service Agent
✓ Notification Received
Notification Type: Bounce
✓ Processing Bounce
Bounce Type: Permanent
Recipients: ['bounce@simulator.amazonses.com']
✓ Bounce processed successfully
============================================================
```
**Check the database:**
```bash
# View bounce logs
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist \
-e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;"
# Check member status
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist \
-e "SELECT email, active, bounce_count, bounce_status FROM members WHERE email='bounce@simulator.amazonses.com';"
```
**View in Web UI:**
- Open http://localhost:3000
- Go to Members tab
- Find "Bounce Test" member
- Should show: ❌ Inactive (red), bounce badge, last bounce timestamp
- Click "Bounces" button to see detailed history
### Option 2: Direct Email via Postfix (Simpler)
Send directly to the simulator without going through your list:
```bash
# Enter Postfix container
sudo docker compose exec postfix bash
# Send test email
echo "Testing hard bounce" | mail -s "Hard Bounce Test" bounce@simulator.amazonses.com
# Or soft bounce
echo "Testing soft bounce" | mail -s "Soft Bounce Test" ooto@simulator.amazonses.com
# Exit container
exit
```
### Option 3: Using AWS SES Console (For Non-Sandbox Testing)
If you have SES production access:
1. Go to AWS SES Console
2. Click "Send test email"
3. To: `bounce@simulator.amazonses.com`
4. From: Your verified email/domain
5. Subject: "Bounce test"
6. Body: "Testing bounce handling"
7. Click "Send test email"
## Testing Different Bounce Types
### 1. Hard Bounce (Permanent Failure)
```bash
echo "Test" | mail -s "Test" bounce@simulator.amazonses.com
```
**Expected Result:**
- Member marked as `hard_bounce`
- Member deactivated (`active = 0`)
- `bounce_count` incremented
- Entry in `bounce_logs` table
### 2. Soft Bounce (Transient Failure)
```bash
echo "Test" | mail -s "Test" ooto@simulator.amazonses.com
```
**Expected Result:**
- Member marked as `clean` (first time)
- After 3 soft bounces → `soft_bounce` status
- `bounce_count` incremented
- Member stays active
### 3. Complaint (Spam Report)
```bash
echo "Test" | mail -s "Test" complaint@simulator.amazonses.com
```
**Expected Result:**
- API receives complaint notification
- Currently logged but not processed (you can extend the handler)
## Monitoring the Test
### Real-Time Monitoring (Recommended)
Open 3 terminal windows:
**Terminal 1 - API Logs:**
```bash
sudo docker compose logs api -f
```
**Terminal 2 - Postfix Logs:**
```bash
sudo docker compose logs postfix -f
```
**Terminal 3 - Send Test Email:**
```bash
sudo docker compose exec postfix bash
echo "Test" | mail -s "Bounce Test" bounce@simulator.amazonses.com
```
### Timeline
Here's what happens and when:
- **T+0s**: Email sent to Postfix
- **T+1-3s**: Postfix relays to SES
- **T+5-10s**: SES processes and generates bounce
- **T+10-30s**: SNS sends notification to your webhook
- **T+30-60s**: API processes bounce and updates database
## Verifying the Complete Flow
### 1. Check Postfix Logs
```bash
sudo docker compose logs postfix | grep bounce@simulator
```
Should show:
```
postfix/smtp[xxx]: ... to=<bounce@simulator.amazonses.com>, relay=email-smtp.eu-west-2.amazonaws.com[...], ... status=sent
```
### 2. Check SNS Subscription Status
- Go to AWS SNS Console
- Find your topic
- Check "Subscriptions" tab
- Status should be "Confirmed"
- Messages delivered should be > 0
### 3. Check API Logs
```bash
sudo docker compose logs api | grep -A 20 "SNS Webhook"
```
Should show successful processing.
### 4. Check Database
```bash
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist <<EOF
-- Show recent bounces
SELECT
b.bounce_id,
m.email,
b.bounce_type,
b.diagnostic_code,
b.timestamp
FROM bounce_logs b
JOIN members m ON b.member_id = m.member_id
ORDER BY b.timestamp DESC
LIMIT 5;
-- Show members with bounces
SELECT
email,
active,
bounce_count,
bounce_status,
last_bounce_at
FROM members
WHERE bounce_count > 0;
EOF
```
## Troubleshooting
### "Email not sent" or "Relay access denied"
**Problem**: Postfix not configured to relay via SES
**Check**:
```bash
sudo docker compose exec postfix postconf relayhost
sudo docker compose exec postfix postconf smtp_sasl_auth_enable
```
Should show:
```
relayhost = [email-smtp.eu-west-2.amazonaws.com]:587
smtp_sasl_auth_enable = yes
```
### "No bounce received after 5 minutes"
**Possible causes**:
1. **SNS subscription not confirmed**
- Check AWS SNS console
- Status should be "Confirmed", not "Pending"
2. **SNS topic not configured in SES**
- Check SES → Configuration Sets or Verified Identities → Notifications
- Bounce notifications should point to your SNS topic
3. **Webhook endpoint not accessible**
- SNS requires HTTPS
- Test: `curl https://your-domain.com:8000/health`
4. **API container not running**
```bash
sudo docker compose ps api
```
### "Bounce received but not in database"
**Check API logs for errors**:
```bash
sudo docker compose logs api | grep -i error
```
**Check database tables exist**:
```bash
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e "SHOW TABLES;"
```
Should include: `bounce_logs`, `members`
## Testing Multiple Bounces
To test the "3 soft bounces = soft_bounce status" logic:
```bash
sudo docker compose exec postfix bash
# Send 3 emails to soft bounce simulator
for i in {1..3}; do
echo "Soft bounce test $i" | mail -s "Test $i" ooto@simulator.amazonses.com
sleep 70 # Wait between sends for SNS processing
done
```
After the 3rd bounce:
- Member's `bounce_status` should change from `clean` to `soft_bounce`
- `bounce_count` should be 3
## Cleanup After Testing
Remove test bounce data:
```bash
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist <<EOF
-- Delete test bounce logs
DELETE FROM bounce_logs WHERE email IN ('bounce@simulator.amazonses.com', 'ooto@simulator.amazonses.com');
-- Reset test member
UPDATE members
SET bounce_count = 0,
bounce_status = 'clean',
last_bounce_at = NULL,
active = 1
WHERE email IN ('bounce@simulator.amazonses.com', 'ooto@simulator.amazonses.com');
EOF
```
Or use the Web UI:
- Go to Members tab
- Find the test member
- Click "Bounces" button
- Click "Reset Bounce Status"
## Alternative: Simulate Bounces Without SES
If SNS isn't set up yet, use the included script to simulate bounces directly in the database:
```bash
./simulate_bounce.sh
```
This is useful for:
- Testing the UI without AWS
- Development environments
- Demonstrating bounce handling to stakeholders
## Next Steps
Once bounce handling is working:
1. **Remove simulator addresses** from your member list
2. **Monitor real bounces** in production
3. **Set up alerts** for high bounce rates
4. **Review bounced members** regularly and update/remove invalid addresses
5. **Consider complaint handling** (similar to bounces, for spam reports)
## Summary Commands
**Quick test sequence**:
```bash
# 1. Watch logs
sudo docker compose logs api -f &
# 2. Send test bounce
echo "Test" | sudo docker compose exec -T postfix mail -s "Bounce Test" bounce@simulator.amazonses.com
# 3. Wait 60 seconds, then check database
sleep 60
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 1;"
```
That's it! The bounce simulator is the easiest way to test your bounce handling without needing real bounced emails.

132
SNS_DEBUG_GUIDE.md Normal file
View File

@@ -0,0 +1,132 @@
# SNS Webhook Debugging Guide
## Current Status
**Detailed logging is now active!** The API will log all incoming SNS requests with:
- Full headers (including SNS-specific headers)
- Request body content
- Parsed message structure
- Processing steps and results
## How to View Logs in Real-Time
```bash
# Watch API logs as SNS sends notifications
sudo docker compose logs api -f
```
## Testing with Real AWS SNS
### Step 1: Trigger a Test from SNS Console
1. Go to AWS SNS Console → Your Topic
2. Click "Publish message"
3. Send a test notification
### Step 2: Check the Logs
The logs will show you exactly what SNS sent:
```
============================================================
SNS Webhook Request Received
============================================================
Content-Type: text/plain; charset=UTF-8
User-Agent: Amazon Simple Notification Service Agent
X-Amz-SNS-Message-Type: Notification
X-Amz-SNS-Topic-Arn: arn:aws:sns:...
Message Type: Notification
Message Keys: ['Type', 'MessageId', 'TopicArn', 'Message', ...]
✓ Notification Received
Message (first 200 chars): {"notificationType":"Bounce",...
Notification Type: Bounce
✓ Processing Bounce
Bounce Type: Permanent
Recipients: ['bounce@example.com']
✓ Bounce processed successfully
============================================================
```
## What the Logs Tell You
### If you see this:
- **"SNS webhook received body: b''"** → SNS sent empty body (check SNS configuration)
- **"Missing SigningCertURL"** → Message missing required SNS fields
- **"Invalid certificate URL"** → Certificate URL not from amazonaws.com
- **"Invalid signature"** → Signature verification failed (message tampered or wrong cert)
### Successful Flow:
1. Headers logged → Shows SNS user agent and message type
2. Body logged → Shows the raw JSON
3. Message parsed → Shows the Type field
4. Notification processed → Shows bounce/complaint details
5. Database updated → Confirmation of processing
## Understanding Previous Errors
The error you saw earlier:
```
SNS webhook error: Expecting value: line 1 column 1 (char 0)
```
This means SNS was sending an **empty body** or the body was already consumed. Possible causes:
1. SNS subscription confirmation URL was opened in a browser (GET request, not POST)
2. Network issue causing body to be lost
3. SNS configuration issue
## Testing Bounce Handling
### Option 1: AWS SES Bounce Simulator
Send email to: `bounce@simulator.amazonses.com`
### Option 2: Verify Database Updates
After a bounce is processed:
```bash
# Check bounce logs
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;"
# Check member bounce status
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT member_id, email, bounce_count, bounce_status, last_bounce_at FROM members WHERE bounce_count > 0;"
```
## Common SNS Message Types
### 1. SubscriptionConfirmation
First message when you create the subscription. API auto-confirms by calling SubscribeURL.
### 2. Notification (Bounce)
```json
{
"Type": "Notification",
"Message": "{\"notificationType\":\"Bounce\",\"bounce\":{...}}"
}
```
### 3. Notification (Complaint)
```json
{
"Type": "Notification",
"Message": "{\"notificationType\":\"Complaint\",\"complaint\":{...}}"
}
```
## Next Steps
1. **Monitor logs while SNS sends**: `sudo docker compose logs api -f`
2. **Trigger a real bounce**: Send to `bounce@simulator.amazonses.com`
3. **Check the detailed logs** to see exactly what SNS is sending
4. **Verify database updates** to confirm bounces are being recorded
## If You Still See Errors
The enhanced logging will now show you:
- What headers SNS is sending
- What body content is arriving (or if it's empty)
- Where in the processing pipeline the error occurs
- Full stack traces for any exceptions
Copy the relevant log section and we can diagnose the exact issue!

16
api/Dockerfile Normal file
View File

@@ -0,0 +1,16 @@
FROM python:3.11-slim
WORKDIR /app
# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy application
COPY . .
# Expose port
EXPOSE 8000
# Run the application
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

231
api/README.md Normal file
View File

@@ -0,0 +1,231 @@
# Mailing List API Documentation
REST API for managing mailing lists and members with token-based authentication.
## Overview
This FastAPI-based REST API provides programmatic access to the mailing list system. It's designed for:
- **Automation**: Scripts and integrations that need to manage lists/members
- **Web Interface**: Powers the frontend at http://localhost:3000
- **Third-party Apps**: Any application that needs mailing list management
**For most users**, the web interface (http://localhost:3000) is more convenient. Use this API when you need programmatic access or automation.
## Base URL
```
http://localhost:8000
```
## Authentication
All endpoints (except `/` and `/health`) require Bearer token authentication.
Add the token to request headers:
```
Authorization: Bearer your_api_token_here
```
Set your API token in `.env`:
```
API_TOKEN=your_secure_token_here
```
## Endpoints
### Health Check
**GET** `/health`
- No authentication required
- Returns API and database status
```bash
curl http://localhost:8000/health
```
### Mailing Lists
**GET** `/lists`
- Get all mailing lists
```bash
curl -H "Authorization: Bearer your_token" http://localhost:8000/lists
```
**GET** `/lists/{list_id}`
- Get a specific mailing list
```bash
curl -H "Authorization: Bearer your_token" http://localhost:8000/lists/1
```
**POST** `/lists`
- Create a new mailing list
```bash
curl -X POST http://localhost:8000/lists \
-H "Authorization: Bearer your_token" \
-H "Content-Type: application/json" \
-d '{
"list_name": "Developers",
"list_email": "dev@lists.sasalliance.org",
"description": "Developer discussions",
"active": true
}'
```
**PATCH** `/lists/{list_id}`
- Update a mailing list
```bash
curl -X PATCH http://localhost:8000/lists/1 \
-H "Authorization: Bearer your_token" \
-H "Content-Type: application/json" \
-d '{
"description": "Updated description",
"active": false
}'
```
**DELETE** `/lists/{list_id}`
- Delete a mailing list
```bash
curl -X DELETE http://localhost:8000/lists/1 \
-H "Authorization: Bearer your_token"
```
### Members
**GET** `/members`
- Get all members
```bash
curl -H "Authorization: Bearer your_token" http://localhost:8000/members
```
**GET** `/members/{member_id}`
- Get a specific member
```bash
curl -H "Authorization: Bearer your_token" http://localhost:8000/members/1
```
**POST** `/members`
- Create a new member
```bash
curl -X POST http://localhost:8000/members \
-H "Authorization: Bearer your_token" \
-H "Content-Type: application/json" \
-d '{
"name": "John Doe",
"email": "john.doe@example.com",
"active": true
}'
```
**PATCH** `/members/{member_id}`
- Update a member
```bash
curl -X PATCH http://localhost:8000/members/1 \
-H "Authorization: Bearer your_token" \
-H "Content-Type: application/json" \
-d '{
"name": "John Q. Doe",
"active": true
}'
```
**DELETE** `/members/{member_id}`
- Delete a member
```bash
curl -X DELETE http://localhost:8000/members/1 \
-H "Authorization: Bearer your_token"
```
### Subscriptions
**GET** `/lists/{list_id}/members`
- Get all members of a specific list
```bash
curl -H "Authorization: Bearer your_token" http://localhost:8000/lists/1/members
```
**POST** `/subscriptions`
- Subscribe a member to a list
```bash
curl -X POST http://localhost:8000/subscriptions \
-H "Authorization: Bearer your_token" \
-H "Content-Type: application/json" \
-d '{
"list_email": "community@lists.sasalliance.org",
"member_email": "john.doe@example.com",
"active": true
}'
```
**DELETE** `/subscriptions?list_email=X&member_email=Y`
- Unsubscribe a member from a list
```bash
curl -X DELETE "http://localhost:8000/subscriptions?list_email=community@lists.sasalliance.org&member_email=john.doe@example.com" \
-H "Authorization: Bearer your_token"
```
## Interactive Documentation
Once the API is running, visit:
- **Swagger UI**: http://localhost:8000/docs
- **ReDoc**: http://localhost:8000/redoc
These provide interactive API documentation where you can test endpoints directly in your browser.
## Example Workflow
1. **Create a new member:**
```bash
curl -X POST http://localhost:8000/members \
-H "Authorization: Bearer your_token" \
-H "Content-Type: application/json" \
-d '{"name": "Jane Smith", "email": "jane@example.com", "active": true}'
```
2. **Create a new mailing list:**
```bash
curl -X POST http://localhost:8000/lists \
-H "Authorization: Bearer your_token" \
-H "Content-Type: application/json" \
-d '{"list_name": "Marketing", "list_email": "marketing@lists.sasalliance.org", "active": true}'
```
3. **Subscribe the member to the list:**
```bash
curl -X POST http://localhost:8000/subscriptions \
-H "Authorization: Bearer your_token" \
-H "Content-Type: application/json" \
-d '{"list_email": "marketing@lists.sasalliance.org", "member_email": "jane@example.com"}'
```
4. **Verify the subscription:**
```bash
curl -H "Authorization: Bearer your_token" http://localhost:8000/lists/1/members
```
## Error Responses
- `401 Unauthorized` - Invalid or missing authentication token
- `404 Not Found` - Resource not found
- `400 Bad Request` - Invalid request data
- `500 Internal Server Error` - Database or server error
## Notes
- All changes take effect immediately in Postfix (no reload needed)
- Email validation is enforced on all email fields
- Deleting a list or member also removes associated subscriptions (CASCADE)

1157
api/main.py Normal file

File diff suppressed because it is too large Load Diff

11
api/requirements.txt Normal file
View File

@@ -0,0 +1,11 @@
fastapi==0.104.1
uvicorn[standard]==0.24.0
mysql-connector-python==8.2.0
pydantic==2.5.0
pydantic-settings==2.1.0
python-multipart==0.0.6
email-validator==2.1.0
bcrypt==4.0.1
python-jose[cryptography]==3.3.0
passlib[bcrypt]==1.7.4
httpx==0.25.2

30
api/test_api.sh Executable file
View File

@@ -0,0 +1,30 @@
#!/bin/bash
# Quick test script for the Mailing List API
API_URL="http://localhost:8000"
TOKEN="your_api_token_here_change_this"
# Colors for output
GREEN='\033[0;32m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
echo -e "${BLUE}Testing Mailing List API${NC}\n"
echo -e "${GREEN}1. Health Check:${NC}"
curl -s $API_URL/health | jq .
echo -e "\n"
echo -e "${GREEN}2. Get All Lists:${NC}"
curl -s -H "Authorization: Bearer $TOKEN" $API_URL/lists | jq .
echo -e "\n"
echo -e "${GREEN}3. Get All Members:${NC}"
curl -s -H "Authorization: Bearer $TOKEN" $API_URL/members | jq .
echo -e "\n"
echo -e "${GREEN}4. Get Members of Community List:${NC}"
curl -s -H "Authorization: Bearer $TOKEN" $API_URL/lists/1/members | jq .
echo -e "\n"
echo -e "${BLUE}Visit http://localhost:8000/docs for interactive API documentation${NC}"

188
database/README.md Normal file
View File

@@ -0,0 +1,188 @@
# Database-Driven Mailing List Management
This mailing list system uses MySQL with **Postfix's native MySQL support** for real-time dynamic list management. Postfix queries the database directly for each email - no scripts or reloads needed.
## Management Options
**1. Web Interface (Recommended for Non-Technical Users)**
- Access: http://localhost:3000
- Visual interface with tables and forms
- Toggle-based subscription management
- No SQL knowledge required
**2. REST API (Recommended for Automation)**
- Access: http://localhost:8000/docs
- Full CRUD operations via HTTP
- Token authentication
- Perfect for scripts and integrations
**3. Direct MySQL (Recommended for Advanced Users)**
- Full SQL access for complex queries
- Bulk operations and reporting
- Database administration tasks
- Described in detail below
## Database Schema
Three-table design with many-to-many relationships:
### Tables
**`lists`** - Mailing list definitions
- `list_id` (primary key)
- `list_name` - Display name
- `list_email` - Full email address (e.g., `community@lists.sasalliance.org`)
- `description` - Optional description
- `active` - Boolean flag to enable/disable list
**`members`** - Member information
- `member_id` (primary key)
- `name` - Display name
- `email` - Email address
- `active` - Boolean flag to enable/disable member
**`list_members`** - Subscription junction table
- `list_id` + `member_id` (composite unique key)
- `active` - Boolean flag to enable/disable subscription
- Foreign keys to `lists` and `members`
## How It Works
1. **Incoming email** arrives for `community@lists.sasalliance.org`
2. **Postfix queries MySQL** using the config in `mysql_virtual_alias_maps.cf`
3. **Database returns** comma-separated list of active member emails
4. **Postfix expands** the alias and delivers to all members
5. **Changes take effect immediately** - no restart or reload needed!
## Managing Lists and Members
### Via Web Interface (Easiest)
1. Open http://localhost:3000 in your browser
2. Enter your API_TOKEN (from .env file)
3. Use the tabs to:
- **Lists Tab**: View, create, edit, delete mailing lists
- **Members Tab**: View, add, edit, remove members
- **Subscriptions**: Click "Subscriptions" button on any member to toggle their list memberships
### Via REST API (For Automation)
See `api/README.md` for complete API documentation, or visit http://localhost:8000/docs for interactive docs.
Quick examples:
```bash
# Get all lists
curl -H "Authorization: Bearer $API_TOKEN" http://localhost:8000/lists
# Create member
curl -X POST http://localhost:8000/members \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"name":"John Doe","email":"john@example.com","active":true}'
# Subscribe to list
curl -X POST http://localhost:8000/subscriptions \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"list_email":"community@lists.sasalliance.org","member_email":"john@example.com"}'
```
### Via MySQL Client (Advanced)
Connect to the database:
```bash
docker-compose exec mysql mysql -u maillist -p maillist
```
### Common Operations
**View all lists:**
```sql
SELECT list_id, list_name, list_email, active FROM lists;
```
**View all members:**
```sql
SELECT member_id, name, email, active FROM members;
```
**View subscriptions for a list:**
```sql
SELECT m.name, m.email
FROM members m
JOIN list_members lm ON m.member_id = lm.member_id
JOIN lists l ON lm.list_id = l.list_id
WHERE l.list_email = 'community@lists.sasalliance.org'
AND lm.active = TRUE AND m.active = TRUE;
```
**Add a new member:**
```sql
INSERT INTO members (name, email)
VALUES ('John Doe', 'john.doe@example.com');
```
**Subscribe member to list:**
```sql
-- Method 1: Using subqueries (one step)
INSERT INTO list_members (list_id, member_id)
VALUES (
(SELECT list_id FROM lists WHERE list_email = 'community@lists.sasalliance.org'),
(SELECT member_id FROM members WHERE email = 'john.doe@example.com')
);
```
**Unsubscribe member from list:**
```sql
DELETE FROM list_members
WHERE list_id = (SELECT list_id FROM lists WHERE list_email = 'community@lists.sasalliance.org')
AND member_id = (SELECT member_id FROM members WHERE email = 'john.doe@example.com');
```
**Create a new mailing list:**
```sql
INSERT INTO lists (list_name, list_email, description)
VALUES ('Developers', 'dev@lists.sasalliance.org', 'Developer discussions');
```
**Disable a list (keeps data, stops delivery):**
```sql
UPDATE lists SET active = FALSE WHERE list_email = 'community@lists.sasalliance.org';
```
**Re-enable a list:**
```sql
UPDATE lists SET active = TRUE WHERE list_email = 'community@lists.sasalliance.org';
```
## Verification
Test that Postfix can query the database:
```bash
docker-compose exec postfix postmap -q "community@lists.sasalliance.org" mysql:/etc/postfix/mysql_virtual_alias_maps.cf
```
This should return a comma-separated list of member email addresses.
## Database Initialization
The database is automatically initialized from `database/schema.sql` when the MySQL container first starts. Sample data includes:
- 4 mailing lists (community, board, members, announcements)
- 2 sample members
- Sample subscriptions
### Reset Database
To completely reset the database (deletes all data!):
```bash
docker-compose down -v # Remove volumes
docker-compose up -d # Reinitialize from schema.sql
```
## Performance
Postfix caches MySQL query results, so the database isn't queried for every single email. The cache TTL is configurable in `mysql_virtual_alias_maps.cf` if needed.

137
database/schema.sql Normal file
View File

@@ -0,0 +1,137 @@
-- Mail List Manager Database Schema
-- Table: users
-- Stores user authentication and authorization information
CREATE TABLE IF NOT EXISTS users (
user_id INT AUTO_INCREMENT PRIMARY KEY,
username VARCHAR(50) NOT NULL UNIQUE,
password_hash VARCHAR(255) NOT NULL, -- bcrypt hash
role ENUM('administrator', 'operator', 'read-only') NOT NULL DEFAULT 'read-only',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
last_login TIMESTAMP NULL,
active BOOLEAN DEFAULT TRUE,
INDEX idx_username (username),
INDEX idx_role (role),
INDEX idx_active (active)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- Table: user_sessions
-- Stores active user sessions for authentication
CREATE TABLE IF NOT EXISTS user_sessions (
session_id VARCHAR(64) PRIMARY KEY,
user_id INT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
expires_at TIMESTAMP NOT NULL,
ip_address VARCHAR(45), -- Supports both IPv4 and IPv6
user_agent TEXT,
active BOOLEAN DEFAULT TRUE,
FOREIGN KEY (user_id) REFERENCES users(user_id) ON DELETE CASCADE,
INDEX idx_user_id (user_id),
INDEX idx_expires_at (expires_at),
INDEX idx_active (active)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- Table: lists
-- Stores mailing list information
CREATE TABLE IF NOT EXISTS lists (
list_id INT AUTO_INCREMENT PRIMARY KEY,
list_name VARCHAR(100) NOT NULL UNIQUE,
list_email VARCHAR(255) NOT NULL UNIQUE,
description TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
active BOOLEAN DEFAULT TRUE,
INDEX idx_list_email (list_email),
INDEX idx_active (active)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- Table: members
-- Stores member information
CREATE TABLE IF NOT EXISTS members (
member_id INT AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(255) NOT NULL,
email VARCHAR(255) NOT NULL UNIQUE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
active BOOLEAN DEFAULT TRUE,
bounce_count INT DEFAULT 0,
last_bounce_at TIMESTAMP NULL,
bounce_status ENUM('clean', 'soft_bounce', 'hard_bounce') DEFAULT 'clean',
INDEX idx_email (email),
INDEX idx_active (active),
INDEX idx_bounce_status (bounce_status)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- Table: list_members
-- Junction table for many-to-many relationship between lists and members
CREATE TABLE IF NOT EXISTS list_members (
id INT AUTO_INCREMENT PRIMARY KEY,
list_id INT NOT NULL,
member_id INT NOT NULL,
subscribed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
active BOOLEAN DEFAULT TRUE,
FOREIGN KEY (list_id) REFERENCES lists(list_id) ON DELETE CASCADE,
FOREIGN KEY (member_id) REFERENCES members(member_id) ON DELETE CASCADE,
UNIQUE KEY unique_list_member (list_id, member_id),
INDEX idx_list_id (list_id),
INDEX idx_member_id (member_id),
INDEX idx_active (active)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- Table: bounce_logs
-- Stores bounce notifications from SES SNS
CREATE TABLE IF NOT EXISTS bounce_logs (
bounce_id INT AUTO_INCREMENT PRIMARY KEY,
member_id INT,
email VARCHAR(255) NOT NULL,
bounce_type ENUM('Permanent', 'Transient', 'Undetermined') NOT NULL,
bounce_subtype VARCHAR(50),
diagnostic_code TEXT,
timestamp TIMESTAMP NOT NULL,
sns_message_id VARCHAR(255),
feedback_id VARCHAR(255),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (member_id) REFERENCES members(member_id) ON DELETE SET NULL,
INDEX idx_member_id (member_id),
INDEX idx_email (email),
INDEX idx_timestamp (timestamp),
INDEX idx_bounce_type (bounce_type)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- Insert sample data
-- Create default admin user (password: 'password')
-- Hash generated with passlib CryptContext using bcrypt
INSERT INTO users (username, password_hash, role) VALUES
('admin', '$2b$12$6Lsp/i.clxyEE9O/4D9RLOfyFlxfjvJBhJ8VXT/t4H/QvrnKrE/YK', 'administrator');
INSERT INTO lists (list_name, list_email, description) VALUES
('Community', 'community@lists.sasalliance.org', 'General community announcements'),
('Board', 'board@lists.sasalliance.org', 'Board members only'),
('Members', 'members@lists.sasalliance.org', 'All members'),
('Announcements', 'announcements@lists.sasalliance.org', 'Important announcements');
INSERT INTO members (name, email) VALUES
('James Pattinson', 'james.pattinson@sasalliance.org'),
('James Pattinson (Personal)', 'james@pattinson.org');
-- Subscribe members to lists
-- Community list - both addresses
INSERT INTO list_members (list_id, member_id) VALUES
(1, 1), -- James (work) on Community
(1, 2); -- James (personal) on Community
-- Board list - work address only
INSERT INTO list_members (list_id, member_id) VALUES
(2, 1); -- James (work) on Board
-- Members list - both addresses
INSERT INTO list_members (list_id, member_id) VALUES
(3, 1), -- James (work) on Members
(3, 2); -- James (personal) on Members
-- Announcements list - both addresses
INSERT INTO list_members (list_id, member_id) VALUES
(4, 1), -- James (work) on Announcements
(4, 2); -- James (personal) on Announcements

View File

@@ -1,8 +1,62 @@
version: "3.9"
networks:
maillist-internal:
driver: bridge
services:
mysql:
image: mysql:8.0
container_name: maillist-mysql
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
MYSQL_DATABASE: ${MYSQL_DATABASE:-maillist}
MYSQL_USER: ${MYSQL_USER:-maillist}
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
volumes:
- mysql_data:/var/lib/mysql
- ./database/schema.sql:/docker-entrypoint-initdb.d/schema.sql
networks:
- maillist-internal
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
interval: 10s
timeout: 5s
retries: 5
postfix:
build: ./postfix
container_name: postfix
env_file: .env
depends_on:
mysql:
condition: service_healthy
ports:
- "25:25"
networks:
- maillist-internal
api:
build: ./api
container_name: maillist-api
env_file: .env
depends_on:
mysql:
condition: service_healthy
ports:
- "8000:8000"
networks:
- maillist-internal
web:
build: ./web
container_name: maillist-web
depends_on:
- api
ports:
- "3000:80"
networks:
- maillist-internal
volumes:
mysql_data:

View File

@@ -0,0 +1,59 @@
# Example nginx configuration for production with SNS webhook support
server {
listen 80;
server_name yourdomain.com;
# Redirect HTTP to HTTPS in production
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name yourdomain.com;
# SSL configuration (add your certificates)
# ssl_certificate /path/to/your/cert.pem;
# ssl_certificate_key /path/to/your/key.pem;
# Frontend (static files)
location / {
proxy_pass http://maillist-web:80;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# API endpoints (including SNS webhook)
location /api/ {
# Remove /api prefix when forwarding to backend
rewrite ^/api/(.*) /$1 break;
proxy_pass http://maillist-api:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Important for SNS webhooks - increase timeouts and body size
proxy_connect_timeout 60s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
client_max_body_size 10M;
}
# Direct SNS webhook endpoint (alternative path)
location /webhooks/sns {
proxy_pass http://maillist-api:8000/webhooks/sns;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# SNS specific settings
proxy_connect_timeout 30s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;
client_max_body_size 1M;
}
}

View File

@@ -4,22 +4,30 @@ FROM debian:stable-slim
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y \
postfix \
postfix-mysql \
libsasl2-modules \
mailutils \
gettext-base \
netcat-openbsd \
python3 \
python3-pymysql \
&& apt-get clean && rm -rf /var/lib/apt/lists/*
# Copy configs
COPY main.cf.template /etc/postfix/main.cf.template
COPY sasl_passwd.template /etc/postfix/sasl_passwd.template
COPY virtual_aliases.cf /etc/postfix/virtual_aliases.cf
COPY mysql_virtual_alias_maps.cf /etc/postfix/mysql_virtual_alias_maps.cf.template
COPY sender_access /etc/postfix/sender_access
COPY smtp_generic /etc/postfix/smtp_generic
COPY aliases /etc/aliases
COPY process-bounce.py /usr/local/bin/process-bounce.py
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
RUN chmod +x /entrypoint.sh /usr/local/bin/process-bounce.py
# Generate Postfix maps for virtual aliases and sender access
RUN postmap /etc/postfix/virtual_aliases.cf
RUN postmap /etc/postfix/sender_access
# Generate Postfix maps for sender access, sender canonical, and aliases
RUN postmap /etc/postfix/sender_access && \
postmap /etc/postfix/smtp_generic && \
newaliases
# Expose SMTP
EXPOSE 25

17
postfix/aliases Normal file
View File

@@ -0,0 +1,17 @@
# Postfix aliases for bounce handling
#
# This file defines how bounces are processed
# The bounces address pipes messages to our bounce processing script
# Bounce processing - pipe to Python script
bounces: "|/usr/bin/python3 /usr/local/bin/process-bounce.py"
# Standard aliases
postmaster: root
mailer-daemon: postmaster
abuse: postmaster
spam: postmaster
root: postmaster
# Default fallback
MAILER-DAEMON: bounces

View File

@@ -4,16 +4,56 @@ set -e
# Generate main.cf from template with environment variables
envsubst < /etc/postfix/main.cf.template > /etc/postfix/main.cf
# Generate MySQL virtual alias config from template
envsubst < /etc/postfix/mysql_virtual_alias_maps.cf.template > /etc/postfix/mysql_virtual_alias_maps.cf
# Generate SASL password file from environment variables
envsubst < /etc/postfix/sasl_passwd.template > /etc/postfix/sasl_passwd
# Wait for MySQL to be ready
echo "Waiting for MySQL to be ready..."
for i in $(seq 1 30); do
if nc -z ${MYSQL_HOST} ${MYSQL_PORT} 2>/dev/null; then
echo "MySQL is ready!"
break
fi
echo "Waiting for MySQL... ($i/30)"
sleep 2
done
# Generate Postfix hash databases
postmap /etc/postfix/sasl_passwd
chmod 600 /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db
# Regenerate sender_access database (in case of updates)
# Regenerate sender_access database
postmap /etc/postfix/sender_access
chmod 644 /etc/postfix/sender_access /etc/postfix/sender_access.db
# Set permissions on MySQL config
chmod 644 /etc/postfix/mysql_virtual_alias_maps.cf
# Configure bounce processing based on environment variable
if [ "${ENABLE_EMAIL_BOUNCE_PROCESSING:-false}" = "true" ]; then
echo "Email-based bounce processing enabled"
# Regenerate aliases database to enable bounce processing
newaliases
chmod 644 /etc/aliases /etc/aliases.db
# Generate smtp generic maps to ensure bounces come to our bounce address
postmap /etc/postfix/smtp_generic
chmod 644 /etc/postfix/smtp_generic /etc/postfix/smtp_generic.db
echo "Configured return path rewriting to bounces@lists.sasalliance.org"
else
echo "Email-based bounce processing disabled"
# Create minimal aliases without bounce processing
echo "postmaster: root" > /etc/aliases
echo "root: postmaster" >> /etc/aliases
newaliases
# Disable smtp generic maps
echo "# Email bounce processing disabled" > /etc/postfix/smtp_generic
postmap /etc/postfix/smtp_generic
fi
# Start Postfix in foreground
exec postfix start-fg

View File

@@ -16,8 +16,8 @@ smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
# Virtual aliases (static for now)
virtual_alias_maps = hash:/etc/postfix/virtual_aliases.cf
# Virtual aliases - dynamic MySQL lookup
virtual_alias_maps = mysql:/etc/postfix/mysql_virtual_alias_maps.cf
# Sender restrictions - enforce whitelist
smtpd_sender_restrictions =
@@ -33,3 +33,23 @@ smtpd_recipient_restrictions =
# Other recommended settings
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
# Bounce handling configuration for email-based processing
# Configure bounce notification recipients
bounce_notice_recipient = bounces@lists.sasalliance.org
2bounce_notice_recipient = bounces@lists.sasalliance.org
delay_notice_recipient =
error_notice_recipient = bounces@lists.sasalliance.org
# Bounce settings
bounce_size_limit = 50000
bounce_queue_lifetime = 5d
maximal_bounce_delay = 1d
# Return path configuration - CRITICAL for bounce handling
# This ensures that when we relay emails via mailing lists through SES,
# bounces come back to our bounce processing address
#
# Use smtp_generic_maps instead of sender_canonical_maps because we only want
# to rewrite the return path for outbound SMTP (via SES), not for local delivery
smtp_generic_maps = hash:/etc/postfix/smtp_generic

View File

@@ -0,0 +1,14 @@
# Postfix MySQL query for virtual aliases
# This file queries the database to expand mailing list addresses to member emails
# Database connection settings
hosts = ${MYSQL_HOST}
port = ${MYSQL_PORT}
user = ${MYSQL_USER}
password = ${MYSQL_PASSWORD}
dbname = ${MYSQL_DATABASE}
# Query to get recipients for a mailing list
# Input: full email address (e.g., community@lists.sasalliance.org)
# Output: comma-separated list of recipient emails
query = SELECT GROUP_CONCAT(m.email SEPARATOR ', ') FROM lists l INNER JOIN list_members lm ON l.list_id = lm.list_id INNER JOIN members m ON lm.member_id = m.member_id WHERE l.list_email = '%s' AND l.active = 1 AND m.active = 1 AND lm.active = 1 GROUP BY l.list_id

348
postfix/process-bounce.py Normal file
View File

@@ -0,0 +1,348 @@
#!/usr/bin/env python3
"""
Email-based bounce processing script for Postfix
Parses bounce emails and updates the database with bounce information
This script is called by Postfix when bounce emails are received.
It reads the email from stdin, parses it for bounce information,
and updates the database accordingly.
"""
import sys
import os
import re
import email
from email.message import Message
import logging
from datetime import datetime
from typing import List, Tuple, Optional, Dict
import pymysql
# Configure logging
log_file = '/var/log/bounce-processor.log'
log_handlers = [logging.StreamHandler()]
# Add file handler if we can write to the log directory
try:
log_handlers.append(logging.FileHandler(log_file))
except (PermissionError, FileNotFoundError):
pass # Just use stdout if we can't write to log file
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
handlers=log_handlers
)
logger = logging.getLogger('bounce-processor')
# Database configuration from environment
MYSQL_HOST = os.getenv('MYSQL_HOST', 'mysql')
MYSQL_PORT = int(os.getenv('MYSQL_PORT', 3306))
MYSQL_DATABASE = os.getenv('MYSQL_DATABASE', 'maillist')
MYSQL_USER = os.getenv('MYSQL_USER', 'maillist')
MYSQL_PASSWORD = os.getenv('MYSQL_PASSWORD', '')
class BounceProcessor:
"""Processes bounce emails and updates the database"""
def __init__(self):
self.bounce_patterns = self._compile_bounce_patterns()
def _compile_bounce_patterns(self) -> List[re.Pattern]:
"""Compile regex patterns for detecting bounce information"""
patterns = [
# Standard bounce formats
re.compile(r'550.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
re.compile(r'554.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
re.compile(r'553.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
re.compile(r'552.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
re.compile(r'551.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
# Delivery Status Notification (DSN) format
re.compile(r'Final-Recipient:.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
re.compile(r'Original-Recipient:.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
# Common bounce message patterns
re.compile(r'user.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?unknown', re.IGNORECASE),
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?does not exist', re.IGNORECASE),
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?not found', re.IGNORECASE),
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?mailbox.*?unavailable', re.IGNORECASE),
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?recipient.*?rejected', re.IGNORECASE),
# Generic email extraction (fallback)
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
]
return patterns
def parse_bounce_email(self, email_content: str) -> List[Dict]:
"""Parse bounce email and extract bounce information"""
try:
# Parse the email
msg = email.message_from_string(email_content)
bounces = []
# Get the email body
body = self._get_email_body(msg)
if not body:
logger.warning("No email body found")
return bounces
logger.info(f"Processing email body (first 500 chars): {body[:500]}")
# Extract bounced addresses
bounced_addresses = self._extract_bounced_addresses(body)
# Determine bounce type and create bounce records
for address in bounced_addresses:
bounce_info = self._analyze_bounce(body, address)
if bounce_info:
bounces.append(bounce_info)
return bounces
except Exception as e:
logger.error(f"Error parsing bounce email: {str(e)}")
return []
def _get_email_body(self, msg: Message) -> Optional[str]:
"""Extract the email body from the message"""
body = ""
if msg.is_multipart():
for part in msg.walk():
content_type = part.get_content_type()
if content_type in ['text/plain', 'text/html']:
payload = part.get_payload(decode=True)
if payload:
body += payload.decode('utf-8', errors='ignore') + "\n"
else:
payload = msg.get_payload(decode=True)
if payload:
body = payload.decode('utf-8', errors='ignore')
return body.strip() if body else None
def _extract_bounced_addresses(self, body: str) -> List[str]:
"""Extract bounced email addresses from the bounce message"""
addresses = []
for pattern in self.bounce_patterns:
matches = pattern.findall(body)
for match in matches:
email_addr = match.strip().lower()
if self._is_valid_email(email_addr) and email_addr not in addresses:
# Skip our own addresses
if not email_addr.endswith('@lists.sasalliance.org'):
addresses.append(email_addr)
logger.info(f"Extracted addresses: {addresses}")
return addresses
def _is_valid_email(self, email_addr: str) -> bool:
"""Validate email address format"""
email_pattern = re.compile(r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$')
return bool(email_pattern.match(email_addr))
def _analyze_bounce(self, body: str, email_addr: str) -> Optional[Dict]:
"""Analyze bounce message to determine bounce type and details"""
bounce_info = {
'email': email_addr,
'bounce_type': 'Undetermined',
'bounce_subtype': '',
'diagnostic_code': '',
'timestamp': datetime.now().strftime('%Y-%m-%d %H:%M:%S')
}
# Analyze bounce type based on SMTP codes and message content
if re.search(r'5[0-9]{2}', body): # 5xx codes are permanent failures
bounce_info['bounce_type'] = 'Permanent'
if re.search(r'550|551|553', body):
bounce_info['bounce_subtype'] = 'General'
elif re.search(r'552', body):
bounce_info['bounce_subtype'] = 'MailboxFull'
elif re.search(r'554', body):
bounce_info['bounce_subtype'] = 'MessageTooLarge'
elif re.search(r'4[0-9]{2}', body): # 4xx codes are temporary failures
bounce_info['bounce_type'] = 'Transient'
if re.search(r'450|451|452', body):
bounce_info['bounce_subtype'] = 'General'
elif re.search(r'452', body):
bounce_info['bounce_subtype'] = 'MailboxFull'
# Extract diagnostic code
smtp_code_match = re.search(r'([45][0-9]{2}.*?)[\r\n]', body)
if smtp_code_match:
bounce_info['diagnostic_code'] = smtp_code_match.group(1).strip()[:500] # Limit length
return bounce_info
def update_database(self, bounces: List[Dict]) -> None:
"""Update the database with bounce information"""
if not bounces:
logger.info("No bounces to process")
return
try:
connection = pymysql.connect(
host=MYSQL_HOST,
port=MYSQL_PORT,
database=MYSQL_DATABASE,
user=MYSQL_USER,
password=MYSQL_PASSWORD,
cursorclass=pymysql.cursors.DictCursor
)
cursor = connection.cursor()
for bounce in bounces:
try:
email_addr = bounce['email']
# Find member by email
cursor.execute("SELECT member_id FROM members WHERE email = %s", (email_addr,))
member = cursor.fetchone()
member_id = member['member_id'] if member else None
# Log the bounce
cursor.execute("""
INSERT INTO bounce_logs
(member_id, email, bounce_type, bounce_subtype, diagnostic_code, timestamp, created_at)
VALUES (%s, %s, %s, %s, %s, %s, NOW())
""", (member_id, email_addr, bounce['bounce_type'], bounce['bounce_subtype'],
bounce['diagnostic_code'], bounce['timestamp']))
logger.info(f"Logged bounce for {email_addr}: {bounce['bounce_type']}")
# Update member bounce status if member exists
if member_id:
if bounce['bounce_type'] == 'Permanent':
# Hard bounce - deactivate member
cursor.execute("""
UPDATE members
SET bounce_count = bounce_count + 1,
last_bounce_at = %s,
bounce_status = 'hard_bounce',
active = 0
WHERE member_id = %s
""", (bounce['timestamp'], member_id))
logger.info(f"Deactivated member {email_addr} due to hard bounce")
elif bounce['bounce_type'] == 'Transient':
# Soft bounce - increment counter and check threshold
cursor.execute("SELECT bounce_count, bounce_status FROM members WHERE member_id = %s", (member_id,))
current = cursor.fetchone()
if current and current['bounce_status'] != 'hard_bounce':
new_count = current['bounce_count'] + 1
new_status = 'soft_bounce' if new_count >= 3 else 'clean'
cursor.execute("""
UPDATE members
SET bounce_count = %s,
last_bounce_at = %s,
bounce_status = %s
WHERE member_id = %s
""", (new_count, bounce['timestamp'], new_status, member_id))
logger.info(f"Updated member {email_addr} bounce count to {new_count}, status: {new_status}")
else:
# Undetermined - just increment counter
cursor.execute("""
UPDATE members
SET bounce_count = bounce_count + 1,
last_bounce_at = %s
WHERE member_id = %s
""", (bounce['timestamp'], member_id))
logger.info(f"Updated member {email_addr} bounce count (undetermined)")
except Exception as e:
logger.error(f"Database error processing bounce for {bounce['email']}: {str(e)}")
continue
connection.commit()
cursor.close()
connection.close()
logger.info(f"Successfully processed {len(bounces)} bounces")
except pymysql.Error as e:
logger.error(f"Database connection error: {str(e)}")
except Exception as e:
logger.error(f"Unexpected error updating database: {str(e)}")
def main():
"""Main function - reads email from stdin and processes bounces"""
try:
logger.info("Starting bounce processing")
# Check if we're in test mode
test_mode = len(sys.argv) > 1 and sys.argv[1] == '--test'
if test_mode:
# Test mode - use sample bounce email
email_content = """From: Mail Delivery Subsystem <MAILER-DAEMON@example.com>
To: bounces@lists.sasalliance.org
Subject: Delivery Status Notification (Failure)
Content-Type: multipart/report; report-type=delivery-status; boundary="boundary123"
--boundary123
Content-Type: text/plain
This is a test bounce message.
The following address(es) failed:
testuser@example.com
SMTP error from remote mail server after RCPT TO:<testuser@example.com>:
550 5.1.1 User unknown
--boundary123
Content-Type: message/delivery-status
Reporting-MTA: dns; mail.example.com
Received-From-MTA: dns; localhost
Final-Recipient: rfc822; testuser@example.com
Action: failed
Status: 5.1.1
Diagnostic-Code: smtp; 550 5.1.1 User unknown
--boundary123--
"""
logger.info("Running in test mode with sample bounce email")
else:
# Read email from stdin
email_content = sys.stdin.read()
if not email_content.strip():
logger.warning("No email content received")
return
logger.info(f"Received email content ({len(email_content)} bytes)")
# Process the bounce
processor = BounceProcessor()
bounces = processor.parse_bounce_email(email_content)
if test_mode:
logger.info(f"Test mode - would process {len(bounces)} bounce(s):")
for bounce in bounces:
logger.info(f" {bounce}")
else:
if bounces:
processor.update_database(bounces)
logger.info(f"Processed {len(bounces)} bounce(s)")
else:
logger.info("No bounces detected in email")
except Exception as e:
logger.error(f"Error in main: {str(e)}")
import traceback
logger.error(f"Traceback: {traceback.format_exc()}")
sys.exit(1)
if __name__ == "__main__":
main()

23
postfix/smtp_generic Normal file
View File

@@ -0,0 +1,23 @@
# Postfix SMTP Generic Maps
# This file controls the return path (envelope sender) for outbound SMTP emails
# Only applies to emails being relayed through external SMTP (SES in our case)
#
# Format: original_sender rewritten_sender
#
# For mailing list emails, we want bounces to go to our bounce processing address
# This is critical for email-based bounce handling to work properly
# Rewrite envelope sender for all mailing list addresses to bounce address
# When emails are forwarded through mailing lists via SES, bounces come back to us
community@lists.sasalliance.org bounces@lists.sasalliance.org
board@lists.sasalliance.org bounces@lists.sasalliance.org
members@lists.sasalliance.org bounces@lists.sasalliance.org
announcements@lists.sasalliance.org bounces@lists.sasalliance.org
# Generic pattern - any @lists.sasalliance.org sender gets rewritten to bounce address
# This catches any new lists automatically
@lists.sasalliance.org bounces@lists.sasalliance.org
# Also rewrite any envelope sender that's sending through our system
# This ensures ALL outgoing mail via SES has our bounce address as return path
@sasalliance.org bounces@lists.sasalliance.org

View File

@@ -1 +0,0 @@
community@lists.sasalliance.org james@pattinson.org, james.pattinson@sasalliance.org

206
simulate_bounce.sh Executable file
View File

@@ -0,0 +1,206 @@
#!/bin/bash
# Simulate Bounce Events for Testing
# This script inserts bounce data directly into the database to test the UI
# without needing to set up AWS SNS
set -e
echo "=== Bounce Simulation Script ==="
echo
echo "This script will create test bounce events for existing members"
echo
# Get database password from .env
DB_PASSWORD=$(grep MYSQL_ROOT_PASSWORD /home/jamesp/docker/maillist/.env | cut -d'=' -f2)
# Check if containers are running
if ! sudo docker-compose ps | grep -q "maillist-mysql.*Up"; then
echo "❌ MySQL container is not running. Starting containers..."
sudo docker-compose up -d
echo "Waiting for MySQL to be ready..."
sleep 5
fi
echo "1. Fetching existing members..."
echo
# Get list of members
MEMBERS=$(sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -N -e "SELECT member_id, name, email FROM members LIMIT 5;")
if [ -z "$MEMBERS" ]; then
echo "❌ No members found in database. Please add members first."
exit 1
fi
echo "Available members:"
echo "$MEMBERS" | while read -r line; do
MEMBER_ID=$(echo "$line" | awk '{print $1}')
NAME=$(echo "$line" | awk '{print $2, $3}')
EMAIL=$(echo "$line" | awk '{$1=""; $2=""; print $0}' | xargs)
echo " [$MEMBER_ID] $NAME - $EMAIL"
done
echo
# Prompt for member ID
read -p "Enter member ID to simulate bounce for (or press Enter for first member): " MEMBER_ID
if [ -z "$MEMBER_ID" ]; then
MEMBER_ID=$(echo "$MEMBERS" | head -1 | awk '{print $1}')
echo "Using member ID: $MEMBER_ID"
fi
# Get member details
MEMBER_INFO=$(sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -N -e "SELECT member_id, name, email FROM members WHERE member_id = $MEMBER_ID;")
if [ -z "$MEMBER_INFO" ]; then
echo "❌ Member ID $MEMBER_ID not found"
exit 1
fi
MEMBER_EMAIL=$(echo "$MEMBER_INFO" | awk '{$1=""; $2=""; print $0}' | xargs)
MEMBER_NAME=$(echo "$MEMBER_INFO" | awk '{print $2, $3}')
echo
echo "Selected member: $MEMBER_NAME ($MEMBER_EMAIL)"
echo
# Bounce type selection
echo "Select bounce type to simulate:"
echo " 1) Soft Bounce (Transient - e.g., mailbox full)"
echo " 2) Hard Bounce (Permanent - e.g., invalid address)"
echo " 3) Multiple Soft Bounces (3 bounces to trigger soft_bounce status)"
echo " 4) Undetermined Bounce"
echo
read -p "Enter choice (1-4): " BOUNCE_CHOICE
case $BOUNCE_CHOICE in
1)
BOUNCE_TYPE="Transient"
BOUNCE_SUBTYPE="MailboxFull"
DIAGNOSTIC="smtp; 452 4.2.2 Mailbox full"
COUNT=1
;;
2)
BOUNCE_TYPE="Permanent"
BOUNCE_SUBTYPE="General"
DIAGNOSTIC="smtp; 550 5.1.1 User unknown"
COUNT=1
;;
3)
BOUNCE_TYPE="Transient"
BOUNCE_SUBTYPE="General"
DIAGNOSTIC="smtp; 451 4.4.1 Temporary failure"
COUNT=3
;;
4)
BOUNCE_TYPE="Undetermined"
BOUNCE_SUBTYPE=""
DIAGNOSTIC="Unknown error occurred"
COUNT=1
;;
*)
echo "Invalid choice"
exit 1
;;
esac
echo
echo "Simulating $COUNT bounce event(s)..."
echo
# Insert bounce events
for i in $(seq 1 $COUNT); do
TIMESTAMP=$(date -u -d "-$((i * 24)) hours" '+%Y-%m-%d %H:%M:%S')
FEEDBACK_ID="test-feedback-$(date +%s)-$i"
echo "Creating bounce event $i/$COUNT (timestamp: $TIMESTAMP)..."
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
-- Insert bounce log
INSERT INTO bounce_logs (member_id, email, bounce_type, bounce_subtype, diagnostic_code, timestamp, feedback_id)
VALUES ($MEMBER_ID, '$MEMBER_EMAIL', '$BOUNCE_TYPE', '$BOUNCE_SUBTYPE', '$DIAGNOSTIC', '$TIMESTAMP', '$FEEDBACK_ID');
EOF
# Update member bounce status
if [ "$BOUNCE_TYPE" = "Permanent" ]; then
echo " → Hard bounce: Deactivating member..."
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
UPDATE members
SET bounce_count = bounce_count + 1,
last_bounce_at = '$TIMESTAMP',
bounce_status = 'hard_bounce',
active = 0
WHERE member_id = $MEMBER_ID;
EOF
elif [ "$BOUNCE_TYPE" = "Transient" ]; then
# Check current bounce count
CURRENT_COUNT=$(sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -N -e "SELECT bounce_count FROM members WHERE member_id = $MEMBER_ID;")
NEW_COUNT=$((CURRENT_COUNT + 1))
if [ $NEW_COUNT -ge 3 ]; then
echo " → Soft bounce threshold reached: Marking as soft_bounce..."
BOUNCE_STATUS="soft_bounce"
else
echo " → Soft bounce: Incrementing counter ($NEW_COUNT)..."
BOUNCE_STATUS="clean"
fi
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
UPDATE members
SET bounce_count = $NEW_COUNT,
last_bounce_at = '$TIMESTAMP',
bounce_status = '$BOUNCE_STATUS'
WHERE member_id = $MEMBER_ID;
EOF
else
echo " → Undetermined bounce: Incrementing counter..."
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
UPDATE members
SET bounce_count = bounce_count + 1,
last_bounce_at = '$TIMESTAMP'
WHERE member_id = $MEMBER_ID;
EOF
fi
done
echo
echo "✅ Bounce simulation complete!"
echo
# Show updated member status
echo "Updated member status:"
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -e "
SELECT
member_id,
name,
email,
active,
bounce_count,
last_bounce_at,
bounce_status
FROM members
WHERE member_id = $MEMBER_ID\G
"
echo
echo "Bounce history for this member:"
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -e "
SELECT
bounce_id,
bounce_type,
bounce_subtype,
diagnostic_code,
timestamp
FROM bounce_logs
WHERE member_id = $MEMBER_ID
ORDER BY timestamp DESC;
"
echo
echo "🎉 You can now view this bounce in the web UI:"
echo " 1. Open http://localhost:3000"
echo " 2. Go to the Members tab"
echo " 3. Look for $MEMBER_NAME"
echo " 4. Click the 'Bounces' button to see the history"
echo
echo "To simulate more bounces, run this script again!"

View File

@@ -0,0 +1,46 @@
<!DOCTYPE html>
<html>
<head>
<title>CORS Understanding Test</title>
</head>
<body>
<h1>CORS Test</h1>
<div id="results"></div>
<script>
const results = document.getElementById('results');
// Test 1: Same-origin request (should work without CORS headers)
console.log('Current origin:', window.location.origin);
// Test what your API client actually does
const protocol = window.location.protocol;
const hostname = window.location.hostname;
let baseURL;
if (hostname === 'localhost' || hostname === '127.0.0.1') {
baseURL = `${protocol}//${hostname}:8000`;
results.innerHTML += `<p><strong>Development Mode Detected</strong></p>`;
results.innerHTML += `<p>Frontend: ${window.location.origin}</p>`;
results.innerHTML += `<p>API: ${baseURL}</p>`;
results.innerHTML += `<p>This is a <strong>CROSS-ORIGIN</strong> request - CORS applies!</p>`;
} else {
baseURL = `${protocol}//${hostname}/api`;
results.innerHTML += `<p><strong>Production Mode Detected</strong></p>`;
results.innerHTML += `<p>Frontend: ${window.location.origin}</p>`;
results.innerHTML += `<p>API: ${baseURL}</p>`;
results.innerHTML += `<p>This is a <strong>SAME-ORIGIN</strong> request - NO CORS needed!</p>`;
}
// Test the actual request
fetch(`${baseURL}/health`)
.then(response => response.json())
.then(data => {
results.innerHTML += `<p style="color: green;">✅ API Request Successful: ${JSON.stringify(data)}</p>`;
})
.catch(error => {
results.innerHTML += `<p style="color: red;">❌ API Request Failed: ${error.message}</p>`;
});
</script>
</body>
</html>

45
test_bounce_quick.sh Normal file
View File

@@ -0,0 +1,45 @@
#!/bin/bash
# Quick SES Bounce Test - One-liner to test bounce handling
echo "🚀 Testing SES Bounce Handling"
echo "================================"
echo ""
# Check if API and Postfix are running
if ! sudo docker compose ps | grep -q "maillist-api.*Up"; then
echo "❌ API container not running. Start with: sudo docker compose up -d"
exit 1
fi
if ! sudo docker compose ps | grep -q "maillist-postfix.*Up"; then
echo "❌ Postfix container not running. Start with: sudo docker compose up -d"
exit 1
fi
echo "✅ Containers are running"
echo ""
# Start watching logs in background
echo "📋 Opening log viewer (press Ctrl+C to stop)..."
echo ""
sleep 2
# Show the command they can run
echo "Run this command to send a test bounce:"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "echo 'Test bounce' | sudo docker compose exec -T postfix mail -s 'Bounce Test' bounce@simulator.amazonses.com"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo ""
echo "Then wait 30-60 seconds and check:"
echo ""
echo "1. API logs (in this window)"
echo "2. Database: sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e 'SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;'"
echo "3. Web UI: http://localhost:3000 → Members tab"
echo ""
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "Watching API logs now..."
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo ""
# Follow logs
sudo docker compose logs api -f --tail 20

76
test_bounce_webhook.sh Executable file
View File

@@ -0,0 +1,76 @@
#!/bin/bash
# Test SNS Bounce Webhook
# This script simulates an SNS bounce notification (without signature verification for testing)
API_URL="http://localhost:8000"
echo "Testing SNS Bounce Webhook..."
echo
# Test 1: Health check
echo "1. Testing API health..."
curl -s "$API_URL/health" | jq .
echo
# Test 2: Create a test subscription confirmation (the webhook will auto-confirm)
echo "2. Testing subscription confirmation handling..."
curl -s -X POST "$API_URL/webhooks/sns" \
-H "Content-Type: application/json" \
-d '{
"Type": "SubscriptionConfirmation",
"MessageId": "test-message-id",
"Token": "test-token",
"TopicArn": "arn:aws:sns:eu-west-2:123456789:test-topic",
"Message": "You have chosen to subscribe to the topic",
"SubscribeURL": "https://example.com/subscribe",
"Timestamp": "2025-01-01T12:00:00.000Z"
}' || echo "Note: Signature verification will fail for test messages (expected)"
echo
echo
# Test 3: Simulated bounce notification (will fail signature verification)
echo "3. Testing bounce notification structure..."
echo "Note: In production, AWS SNS will send properly signed messages."
echo "This test demonstrates the expected structure."
echo
cat << 'EOF' > /tmp/test_bounce.json
{
"Type": "Notification",
"MessageId": "test-notification-id",
"TopicArn": "arn:aws:sns:eu-west-2:123456789:test-topic",
"Subject": "Amazon SES Email Event Notification",
"Message": "{\"notificationType\":\"Bounce\",\"bounce\":{\"bounceType\":\"Permanent\",\"bounceSubType\":\"General\",\"bouncedRecipients\":[{\"emailAddress\":\"bounce@simulator.amazonses.com\",\"diagnosticCode\":\"smtp; 550 5.1.1 user unknown\"}],\"timestamp\":\"2025-01-01T12:00:00.000Z\",\"feedbackId\":\"test-feedback-id\"}}",
"Timestamp": "2025-01-01T12:00:00.000Z",
"SignatureVersion": "1",
"Signature": "test-signature",
"SigningCertURL": "https://sns.eu-west-2.amazonaws.com/test.pem"
}
EOF
cat /tmp/test_bounce.json | jq .
echo
echo "Expected behavior: Signature verification will fail without real AWS credentials"
echo "In production, AWS SNS will send properly signed messages that will be verified"
echo
# Test 4: Check if database schema has bounce tables
echo "4. Checking database schema for bounce tables..."
sudo docker-compose exec -T mysql mysql -u maillist -pmaillist maillist -e "SHOW TABLES LIKE '%bounce%';" 2>/dev/null
echo
# Test 5: Check members table for bounce columns
echo "5. Checking members table for bounce columns..."
sudo docker-compose exec -T mysql mysql -u maillist -pmaillist maillist -e "DESCRIBE members;" 2>/dev/null | grep -i bounce
echo
echo "✓ Setup complete!"
echo
echo "To test with real AWS SNS:"
echo "1. Set up SNS topic in AWS Console"
echo "2. Subscribe webhook: https://your-domain.com:8000/webhooks/sns"
echo "3. Configure SES to send bounce notifications to the SNS topic"
echo "4. Send test email to bounce@simulator.amazonses.com"
echo
echo "See BOUNCE_HANDLING_SETUP.md for detailed setup instructions"

14
web/Dockerfile Normal file
View File

@@ -0,0 +1,14 @@
# Use nginx to serve the static files
FROM nginx:alpine
# Copy the web files to nginx html directory
COPY . /usr/share/nginx/html/
# Copy custom nginx configuration
COPY nginx.conf /etc/nginx/conf.d/default.conf
# Expose port 80
EXPOSE 80
# Start nginx
CMD ["nginx", "-g", "daemon off;"]

171
web/README.md Normal file
View File

@@ -0,0 +1,171 @@
# Web Frontend for Mailing List Manager
This directory contains the web frontend for the Mailing List Manager - a clean, modern web interface for managing mailing lists and members.
## Features
- 🔐 **Token-based Authentication** - Secure access using API tokens
- 📧 **Mailing List Management** - Create, edit, and delete mailing lists
- 👥 **Member Management** - Add, edit, and remove members
- 🔗 **Subscription Management** - Subscribe/unsubscribe members to/from lists
- 📱 **Responsive Design** - Works on desktop, tablet, and mobile devices
- 🎨 **Clean UI** - Modern, professional interface with intuitive navigation
-**Real-time Updates** - Immediate feedback and data refresh
## Architecture
### Frontend Stack
- **HTML5** - Semantic markup with accessibility features
- **CSS3** - Modern styling with CSS Grid/Flexbox and custom properties
- **Vanilla JavaScript** - No frameworks, just clean ES6+ code
- **Nginx** - Static file serving with compression and caching
### File Structure
```
web/
├── index.html # Main HTML interface
├── static/
│ ├── css/
│ │ └── style.css # Complete styling system
│ └── js/
│ ├── api.js # API client for backend communication
│ ├── ui.js # UI helpers and modal management
│ └── app.js # Main application controller
├── Dockerfile # Container configuration
├── nginx.conf # Nginx server configuration
└── README.md # This file
```
## Usage
### Accessing the Interface
**Via Docker (Recommended):**
```bash
# Ensure all services are running
sudo docker-compose up -d
# Access the web interface
open http://localhost:3000
```
**For Development:**
You can also open `index.html` directly in a browser, but the API must be running on port 8000.
### Getting Started
1. **Authenticate**: Enter your API_TOKEN (from `.env` file) when prompted
2. **Manage Lists**: Click the "Lists" tab to view and manage mailing lists
3. **Manage Members**: Click the "Members" tab to add and edit member information
4. **Manage Subscriptions**: Click "Subscriptions" button on any member to toggle their list memberships
The interface saves your token in browser storage, so you won't need to re-enter it on subsequent visits.
## Features Overview
### Authentication
- Secure token-based authentication
- Token persistence in browser storage
- Automatic logout on authentication errors
### Subscription Management (New & Improved!)
#### Member-Centric Subscription Management
The subscription management has been completely redesigned for the best user experience:
**How to Use:**
1. Navigate to the **Members** tab
2. Find the member you want to manage
3. Click the **"Subscriptions"** button next to their name
4. A modal appears showing all available mailing lists with toggle switches
**Visual Interface:**
- **Green toggle** = Member is subscribed to this list
- **Gray toggle** = Member is not subscribed to this list
- **Click anywhere** on a list item to toggle subscription on/off
**Batch Operations:**
- Make multiple changes (subscribe to some lists, unsubscribe from others)
- The "Save Changes" button shows how many changes you've made (e.g., "Save 3 Changes")
- All changes are saved together when you click "Save Changes"
- Click "Cancel" to discard all changes
**Benefits:**
-**Fast** - Toggle multiple subscriptions at once
-**Visual** - See all subscriptions at a glance with color coding
-**Intuitive** - Works like modern mobile app switches
-**Smart** - Only saves actual changes, not unchanged items
-**Clear** - Shows exactly how many changes you're about to save
#### Alternative: Legacy Bulk Subscription
For power users who prefer the old approach:
1. Go to the **Subscriptions** tab
2. Select list and member from dropdowns
3. Click "Add Subscription"
This is still available but the member-centric approach is much more efficient for managing multiple subscriptions.
### Mailing Lists
- View all mailing lists in a clean table
- Create new lists with name, email, and description
- Edit existing list details
- Delete lists with confirmation
- View member count for each list
- Status indicators (Active/Inactive)
### Members
- View all members with their details
- Add new members with name and email
- Edit member information
- Delete members with confirmation
- See which lists each member is subscribed to
- Status management
### Subscriptions
- **Member-centric subscription management** - Click on any member to manage their subscriptions
- **Interactive toggle interface** - Check/uncheck lists with visual toggles
- **Batch changes** - Make multiple subscription changes and save them all at once
- **Real-time feedback** - See subscription status instantly with visual indicators
- **Subscription overview** - View all subscriptions organized by list
- **Quick add functionality** - Legacy bulk subscription interface for power users
### User Experience
- **Responsive Design** - Works seamlessly on all device sizes
- **Loading States** - Visual feedback during API operations
- **Error Handling** - Clear error messages and graceful degradation
- **Notifications** - Success/error notifications with auto-dismiss
- **Confirmation Dialogs** - Prevent accidental deletions
- **Keyboard Shortcuts** - Enter key support in forms
- **Accessibility** - Semantic HTML and ARIA labels
## API Integration
The frontend communicates with the FastAPI backend through a comprehensive API client:
- **Authentication** - Bearer token authentication
- **Error Handling** - Automatic error parsing and user-friendly messages
- **Loading States** - Integrated loading indicators
- **Data Validation** - Client-side form validation before API calls
## Customization
### Styling
The CSS uses CSS custom properties (variables) for easy theming:
```css
:root {
--primary-color: #2563eb;
--success-color: #10b981;
--danger-color: #ef4444;
/* ... more variables */
}
```
### Configuration
The API client automatically detects the backend URL based on the current hostname and can be configured for different environments.
## Browser Support
- Modern browsers (Chrome 60+, Firefox 55+, Safari 12+, Edge 79+)
- Mobile browsers (iOS Safari 12+, Chrome Mobile 60+)
- Progressive enhancement for older browsers

703
web/index.html Normal file
View File

@@ -0,0 +1,703 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Mailing List Manager</title>
<link rel="stylesheet" href="static/css/style.css">
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap" rel="stylesheet">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css">
</head>
<body>
<!-- Login Page -->
<div class="login-page" id="loginPage">
<div class="login-container">
<div class="login-header">
<div class="login-logo">
<i class="fas fa-envelope"></i>
</div>
<h1>Mailing List Manager</h1>
<p>Sign in to manage your mailing lists</p>
</div>
<form class="login-form" id="loginForm">
<!-- Login Error Message -->
<div class="login-error" id="loginError" style="display: none;">
<i class="fas fa-exclamation-circle"></i>
<span id="loginErrorMessage">Invalid username or password</span>
</div>
<div class="form-group">
<label for="username">
<i class="fas fa-user"></i>
Username
</label>
<input type="text" id="username" placeholder="Enter your username" required autocomplete="username">
</div>
<div class="form-group">
<label for="password">
<i class="fas fa-lock"></i>
Password
</label>
<input type="password" id="password" placeholder="Enter your password" required autocomplete="current-password">
</div>
<button type="submit" class="btn btn-primary btn-block" id="loginBtn">
<span>Sign In</span>
<i class="fas fa-arrow-right"></i>
</button>
</form>
<div class="login-footer">
<p class="text-muted text-sm">
<i class="fas fa-shield-alt"></i>
Secure authentication required
</p>
</div>
</div>
</div>
<!-- Menu Bar Header (shown after login) -->
<header class="menu-bar" id="mainHeader" style="display: none;">
<div class="menu-bar-content">
<div class="app-title">
<i class="fas fa-envelope"></i>
<span>Mailing List Manager</span>
</div>
<div class="menu-spacer"></div>
<div class="user-dropdown" id="userDropdown">
<button class="user-dropdown-trigger" id="userDropdownTrigger">
<div class="user-avatar">
<i class="fas fa-user"></i>
</div>
<div class="user-details">
<span class="user-name" id="currentUsername">User</span>
<span class="user-role" id="currentUserRole">role</span>
</div>
<i class="fas fa-chevron-down dropdown-arrow"></i>
</button>
<div class="user-dropdown-menu" id="userDropdownMenu">
<div class="dropdown-header">
<div class="dropdown-user-info">
<div class="dropdown-avatar">
<i class="fas fa-user"></i>
</div>
<div class="dropdown-details">
<div class="dropdown-name" id="dropdownUsername">User</div>
<div class="dropdown-role" id="dropdownUserRole">role</div>
</div>
</div>
</div>
<div class="dropdown-divider"></div>
<button class="dropdown-item" id="userManagementBtn" style="display: none;">
<i class="fas fa-user-shield"></i>
<span>User Management</span>
</button>
<div class="dropdown-divider" id="userManagementDivider" style="display: none;"></div>
<button class="dropdown-item" id="logoutBtn">
<i class="fas fa-sign-out-alt"></i>
<span>Sign Out</span>
</button>
</div>
</div>
</div>
</header>
<!-- Main Content -->
<main class="main-content" id="mainContent" style="display: none;">
<div class="container">
<!-- Navigation Tabs -->
<nav class="tab-nav" id="tabNav">
<button class="tab-btn active" data-tab="lists">
<i class="fas fa-list"></i>
Mailing Lists
</button>
<button class="tab-btn" data-tab="members">
<i class="fas fa-users"></i>
Members
</button>
</nav>
<!-- Notification Area -->
<div class="notification" id="notification" style="display: none;">
<span class="notification-message" id="notificationMessage"></span>
<button class="notification-close" id="notificationClose">
<i class="fas fa-times"></i>
</button>
</div>
<!-- Mailing Lists Tab -->
<div class="tab-content active" id="lists-tab">
<div class="section-header">
<h2>Mailing Lists</h2>
<button class="btn btn-primary" id="addListBtn">
<i class="fas fa-plus"></i>
Add List
</button>
</div>
<div class="data-table">
<table class="table" id="listsTable">
<thead>
<tr>
<th>Name</th>
<th>Email</th>
<th>Description</th>
<th>Members</th>
<th>Status</th>
<th>Actions</th>
</tr>
</thead>
<tbody id="listsTableBody">
<!-- Dynamic content -->
</tbody>
</table>
</div>
</div>
<!-- Members Tab -->
<div class="tab-content" id="members-tab">
<div class="section-header">
<h2>Members</h2>
<div class="button-group">
<button class="btn btn-primary" id="addMemberBtn">
<i class="fas fa-plus"></i>
Add Member
</button>
<button class="btn btn-secondary" id="showBulkImportBtn">
<i class="fas fa-upload"></i>
Bulk Import CSV
</button>
</div>
</div>
<!-- Member Search -->
<div class="search-section">
<div class="search-container">
<div class="search-input-wrapper">
<i class="fas fa-search search-icon"></i>
<input type="text"
id="memberSearchInput"
class="search-input"
placeholder="Search members by name or email..."
autocomplete="off">
<button class="search-clear" id="memberSearchClear" style="display: none;">
<i class="fas fa-times"></i>
</button>
</div>
<div class="search-results-info" id="memberSearchInfo" style="display: none;">
<span id="memberSearchCount">0</span> members found
</div>
</div>
</div>
<div class="data-table">
<table class="table" id="membersTable">
<thead>
<tr>
<th>Name</th>
<th>Email</th>
<th>Lists</th>
<th>Status</th>
<th>Actions</th>
</tr>
</thead>
<tbody id="membersTableBody">
<!-- Dynamic content -->
</tbody>
</table>
</div>
</div>
<!-- Users Tab (Admin Only) -->
<div class="tab-content" id="users-tab">
<div class="section-header">
<h2>User Management</h2>
<button class="btn btn-primary" id="addUserBtn">
<i class="fas fa-plus"></i>
Add User
</button>
</div>
<div class="data-table">
<table class="table" id="usersTable">
<thead>
<tr>
<th>Username</th>
<th>Role</th>
<th>Created</th>
<th>Last Login</th>
<th>Status</th>
<th>Actions</th>
</tr>
</thead>
<tbody id="usersTableBody">
<!-- Dynamic content -->
</tbody>
</table>
</div>
</div>
</div>
</main>
<!-- Loading Overlay -->
<div class="loading-overlay" id="loadingOverlay" style="display: none;">
<div class="loading-spinner">
<i class="fas fa-spinner fa-spin"></i>
<p>Loading...</p>
</div>
</div>
<!-- Modals -->
<!-- Add/Edit List Modal -->
<div class="modal" id="listModal">
<div class="modal-content">
<div class="modal-header">
<h3 id="listModalTitle">Add Mailing List</h3>
<button class="modal-close" id="listModalClose">
<i class="fas fa-times"></i>
</button>
</div>
<form id="listForm">
<div class="form-group">
<label for="listName">List Name *</label>
<input type="text" id="listName" name="listName" required>
</div>
<div class="form-group">
<label for="listEmail">List Email *</label>
<input type="email" id="listEmail" name="listEmail" required>
</div>
<div class="form-group">
<label for="listDescription">Description</label>
<textarea id="listDescription" name="listDescription" rows="3"></textarea>
</div>
<div class="form-group">
<label class="checkbox-label">
<input type="checkbox" id="listActive" name="listActive" checked>
<span class="checkmark"></span>
Active
</label>
</div>
<div class="modal-actions">
<button type="button" class="btn btn-secondary" id="listCancelBtn">Cancel</button>
<button type="submit" class="btn btn-primary" id="listSubmitBtn">Save</button>
</div>
</form>
</div>
</div>
<!-- Add/Edit Member Modal -->
<div class="modal" id="memberModal">
<div class="modal-content">
<div class="modal-header">
<h3 id="memberModalTitle">Add Member</h3>
<button class="modal-close" id="memberModalClose">
<i class="fas fa-times"></i>
</button>
</div>
<form id="memberForm">
<div class="form-group">
<label for="memberName">Member Name *</label>
<input type="text" id="memberName" name="memberName" required>
</div>
<div class="form-group">
<label for="memberEmail">Email Address *</label>
<input type="email" id="memberEmail" name="memberEmail" required>
</div>
<div class="form-group">
<label class="checkbox-label">
<input type="checkbox" id="memberActive" name="memberActive" checked>
<span class="checkmark"></span>
Active
</label>
</div>
<div class="modal-actions">
<button type="button" class="btn btn-secondary" id="memberCancelBtn">Cancel</button>
<button type="submit" class="btn btn-primary" id="memberSubmitBtn">Save</button>
</div>
</form>
</div>
</div>
<!-- Member Subscriptions Modal -->
<div class="modal" id="memberSubscriptionsModal">
<div class="modal-content">
<div class="modal-header">
<h3 id="memberSubscriptionsTitle">Manage Subscriptions</h3>
<button class="modal-close" id="memberSubscriptionsModalClose">
<i class="fas fa-times"></i>
</button>
</div>
<div class="modal-body">
<div class="member-info-banner" id="memberInfoBanner">
<div class="member-avatar">
<i class="fas fa-user"></i>
</div>
<div class="member-details">
<h4 id="memberSubscriptionsName">Member Name</h4>
<p id="memberSubscriptionsEmail">member@example.com</p>
</div>
</div>
<div class="subscriptions-section">
<h5>Mailing List Subscriptions</h5>
<p class="text-muted text-sm">Check the lists this member should be subscribed to:</p>
<div class="subscription-list" id="subscriptionCheckboxList">
<!-- Dynamic content will be inserted here -->
</div>
</div>
</div>
<div class="modal-actions">
<button type="button" class="btn btn-secondary" id="memberSubscriptionsCancelBtn">Cancel</button>
<button type="button" class="btn btn-primary" id="memberSubscriptionsSaveBtn">
<i class="fas fa-save"></i>
Save Changes
</button>
</div>
</div>
</div>
<!-- Add Subscription Modal (Legacy - keeping for bulk operations) -->
<div class="modal" id="subscriptionModal">
<div class="modal-content">
<div class="modal-header">
<h3>Add Subscription</h3>
<button class="modal-close" id="subscriptionModalClose">
<i class="fas fa-times"></i>
</button>
</div>
<form id="subscriptionForm">
<div class="form-group">
<label for="subscriptionList">Mailing List *</label>
<select id="subscriptionList" name="subscriptionList" required>
<option value="">Select a list...</option>
</select>
</div>
<div class="form-group">
<label for="subscriptionMember">Member *</label>
<select id="subscriptionMember" name="subscriptionMember" required>
<option value="">Select a member...</option>
</select>
</div>
<div class="modal-actions">
<button type="button" class="btn btn-secondary" id="subscriptionCancelBtn">Cancel</button>
<button type="submit" class="btn btn-primary">Add Subscription</button>
</div>
</form>
</div>
</div>
<!-- Bulk Import Modal -->
<div class="modal modal-large" id="bulkImportModal">
<div class="modal-content">
<div class="modal-header">
<h3>Bulk Import Members from CSV</h3>
<button class="modal-close" id="bulkImportModalClose">
<i class="fas fa-times"></i>
</button>
</div>
<div class="modal-body">
<!-- Step 1: File Upload -->
<div class="import-step" id="importStep1">
<h4><i class="fas fa-upload"></i> Step 1: Upload CSV File</h4>
<p class="text-muted">Upload a CSV file containing member information. The file must have 'Name' and 'Email' columns.</p>
<div class="file-upload-area" id="fileUploadArea">
<div class="file-upload-content">
<i class="fas fa-cloud-upload-alt"></i>
<p>Click to select a CSV file or drag and drop</p>
<input type="file" id="csvFileInput" accept=".csv" style="display: none;">
</div>
</div>
<div class="file-info" id="fileInfo" style="display: none;">
<div class="file-details">
<i class="fas fa-file-csv"></i>
<span id="fileName">file.csv</span>
<span id="fileSize" class="text-muted">(0 KB)</span>
</div>
<button type="button" class="btn btn-sm btn-secondary" id="removeFileBtn">
<i class="fas fa-times"></i> Remove
</button>
</div>
<div class="csv-format-help">
<h5><i class="fas fa-info-circle"></i> CSV Format</h5>
<p>Your CSV file should look like this:</p>
<div class="code-example">
<code>Name,Email,Delivery,Member since<br>
Ahmed Ajzan,ahmedajzan@doctors.org.uk,,2025-04-06T18:44:26.819454<br>
Alan Bailey,abgower@icloud.com,,2025-04-06T18:44:26.824446</code>
</div>
<p class="text-muted text-sm">Only the 'Name' and 'Email' columns are required. Additional columns will be ignored.</p>
</div>
</div>
<!-- Step 2: Preview -->
<div class="import-step" id="importStep2" style="display: none;">
<h4><i class="fas fa-eye"></i> Step 2: Preview Data</h4>
<p class="text-muted">Review the data that will be imported:</p>
<div class="preview-stats" id="previewStats">
<div class="stat-item">
<span class="stat-value" id="totalRowsCount">0</span>
<span class="stat-label">Total Rows</span>
</div>
<div class="stat-item">
<span class="stat-value" id="validRowsCount">0</span>
<span class="stat-label">Valid Rows</span>
</div>
<div class="stat-item">
<span class="stat-value" id="errorRowsCount">0</span>
<span class="stat-label">Errors</span>
</div>
</div>
<div class="preview-table-container" id="previewTableContainer">
<table class="table table-sm" id="previewTable">
<thead>
<tr>
<th>Row</th>
<th>Name</th>
<th>Email</th>
<th>Status</th>
</tr>
</thead>
<tbody id="previewTableBody">
<!-- Dynamic content -->
</tbody>
</table>
</div>
<div class="error-list" id="errorList" style="display: none;">
<div class="error-header">
<h5><i class="fas fa-exclamation-triangle"></i> Errors Found</h5>
<button type="button" class="btn btn-sm btn-secondary" id="toggleErrorsBtn">
<span id="errorToggleText">Show All</span>
<i class="fas fa-chevron-down" id="errorToggleIcon"></i>
</button>
</div>
<div class="error-summary" id="errorSummary">
<!-- Summary will be shown here -->
</div>
<div class="error-details" id="errorDetails" style="display: none;">
<div class="error-table-container">
<table class="table table-sm" id="errorTable">
<thead>
<tr>
<th>Row</th>
<th>Issue</th>
<th>Name</th>
<th>Email</th>
<th>Raw Data</th>
</tr>
</thead>
<tbody id="errorTableBody">
<!-- Dynamic content -->
</tbody>
</table>
</div>
</div>
</div>
</div>
<!-- Step 3: List Selection -->
<div class="import-step" id="importStep3" style="display: none;">
<h4><i class="fas fa-list"></i> Step 3: Select Mailing Lists</h4>
<p class="text-muted">Choose which mailing lists these members should be subscribed to:</p>
<div class="list-selection" id="listSelection">
<!-- Dynamic content -->
</div>
<div class="selection-info">
<p class="text-muted text-sm">
<i class="fas fa-info-circle"></i>
If a member already exists, they will be added to the selected lists (existing subscriptions will be preserved).
</p>
</div>
</div>
<!-- Step 4: Import Results -->
<div class="import-step" id="importStep4" style="display: none;">
<h4><i class="fas fa-check-circle"></i> Step 4: Import Complete</h4>
<div class="import-results" id="importResults">
<div class="result-stats">
<div class="stat-item">
<span class="stat-value" id="processedCount">0</span>
<span class="stat-label">Processed</span>
</div>
<div class="stat-item">
<span class="stat-value" id="createdCount">0</span>
<span class="stat-label">Created</span>
</div>
<div class="stat-item">
<span class="stat-value" id="updatedCount">0</span>
<span class="stat-label">Updated</span>
</div>
<div class="stat-item">
<span class="stat-value" id="subscriptionsCount">0</span>
<span class="stat-label">Subscriptions</span>
</div>
</div>
<div class="result-errors" id="resultErrors" style="display: none;">
<h5><i class="fas fa-exclamation-triangle"></i> Import Errors</h5>
<ul id="resultErrorList">
<!-- Dynamic content -->
</ul>
</div>
</div>
</div>
</div>
<div class="modal-actions">
<button type="button" class="btn btn-secondary" id="bulkImportBackBtn" style="display: none;">
<i class="fas fa-arrow-left"></i> Back
</button>
<button type="button" class="btn btn-secondary" id="bulkImportCancelBtn">Cancel</button>
<button type="button" class="btn btn-primary" id="bulkImportNextBtn" disabled>
Next <i class="fas fa-arrow-right"></i>
</button>
<button type="button" class="btn btn-primary" id="bulkImportBtn" style="display: none;">
<i class="fas fa-upload"></i> Import Members
</button>
<button type="button" class="btn btn-success" id="bulkImportDoneBtn" style="display: none;">
<i class="fas fa-check"></i> Done
</button>
</div>
</div>
</div>
<!-- Add/Edit User Modal -->
<div class="modal" id="userModal">
<div class="modal-content">
<div class="modal-header">
<h3 id="userModalTitle">Add User</h3>
<button class="modal-close" id="userModalClose">
<i class="fas fa-times"></i>
</button>
</div>
<form id="userForm">
<div class="form-group">
<label for="userName">Username *</label>
<input type="text" id="userName" name="userName" required>
</div>
<div class="form-group">
<label for="userPassword">Password *</label>
<input type="password" id="userPassword" name="userPassword" required>
<div class="form-help" id="passwordHelp">
Leave blank to keep current password (when editing)
</div>
</div>
<div class="form-group">
<label for="userRole">Role *</label>
<select id="userRole" name="userRole" required>
<option value="">Select a role...</option>
<option value="administrator">Administrator - Full access including user management</option>
<option value="operator">Operator - Read/write access to lists and members</option>
<option value="read-only">Read-Only - View-only access</option>
</select>
</div>
<div class="form-group">
<label class="checkbox-label">
<input type="checkbox" id="userActive" name="userActive" checked>
<span class="checkmark"></span>
Active
</label>
</div>
<div class="modal-actions">
<button type="button" class="btn btn-secondary" id="userCancelBtn">Cancel</button>
<button type="submit" class="btn btn-primary" id="userSubmitBtn">Save</button>
</div>
</form>
</div>
</div>
<!-- Bounce History Modal -->
<div class="modal" id="bounceHistoryModal">
<div class="modal-content">
<div class="modal-header">
<h3 id="bounceHistoryTitle">Bounce History</h3>
<button class="modal-close" id="bounceHistoryModalClose">
<i class="fas fa-times"></i>
</button>
</div>
<div class="modal-body">
<div class="member-info-banner" id="bounceHistoryMemberInfo">
<div class="member-avatar">
<i class="fas fa-user"></i>
</div>
<div class="member-details">
<h4 id="bounceHistoryMemberName">Member Name</h4>
<p id="bounceHistoryMemberEmail">member@example.com</p>
</div>
</div>
<div class="bounce-summary">
<div class="bounce-stat">
<i class="fas fa-exclamation-triangle"></i>
<div>
<div class="bounce-stat-label">Total Bounces</div>
<div class="bounce-stat-value" id="bounceTotalCount">0</div>
</div>
</div>
<div class="bounce-stat">
<i class="fas fa-clock"></i>
<div>
<div class="bounce-stat-label">Last Bounce</div>
<div class="bounce-stat-value" id="bounceLastDate">Never</div>
</div>
</div>
<div class="bounce-stat">
<i class="fas fa-info-circle"></i>
<div>
<div class="bounce-stat-label">Status</div>
<div class="bounce-stat-value" id="bounceStatusText">Clean</div>
</div>
</div>
</div>
<div class="bounce-history-section">
<h5>Bounce Events</h5>
<div class="bounce-history-list" id="bounceHistoryList">
<!-- Dynamic content will be inserted here -->
</div>
</div>
</div>
<div class="modal-actions">
<button type="button" class="btn btn-secondary" id="bounceHistoryCloseBtn">Close</button>
<button type="button" class="btn btn-warning" id="bounceHistoryResetBtn" data-requires-write>
<i class="fas fa-redo"></i>
Reset Bounce Status
</button>
</div>
</div>
</div>
<!-- Confirmation Modal -->
<div class="modal" id="confirmModal">
<div class="modal-content">
<div class="modal-header">
<h3>Confirm Action</h3>
<button class="modal-close" id="confirmModalClose">
<i class="fas fa-times"></i>
</button>
</div>
<div class="modal-body">
<p id="confirmMessage">Are you sure you want to perform this action?</p>
</div>
<div class="modal-actions">
<button type="button" class="btn btn-secondary" id="confirmCancelBtn">Cancel</button>
<button type="button" class="btn btn-danger" id="confirmOkBtn">Confirm</button>
</div>
</div>
</div>
<script src="static/js/api.js"></script>
<script src="static/js/app.js"></script>
<script src="static/js/ui.js"></script>
</body>
</html>

46
web/nginx.conf Normal file
View File

@@ -0,0 +1,46 @@
server {
listen 80;
server_name localhost;
# Cache static assets - this needs to come BEFORE the main location block
location ~* \.(css|js|png|jpg|jpeg|gif|ico|svg)$ {
root /usr/share/nginx/html;
expires -1; # Disable caching for development
add_header Cache-Control "no-cache, no-store, must-revalidate";
add_header Pragma "no-cache";
add_header Expires "0";
}
# Serve static files
location / {
root /usr/share/nginx/html;
index index.html;
try_files $uri $uri/ /index.html;
}
# Enable gzip compression
gzip on;
gzip_vary on;
gzip_min_length 1024;
gzip_proxied expired no-cache no-store private auth;
gzip_types
text/plain
text/css
text/xml
text/javascript
application/javascript
application/xml+rss
application/json;
# Security headers
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
# Prevent access to hidden files
location ~ /\. {
deny all;
}
}

1954
web/static/css/style.css Normal file

File diff suppressed because it is too large Load Diff

311
web/static/js/api.js Normal file
View File

@@ -0,0 +1,311 @@
/**
* API Client for Mailing List Manager
* Handles all communication with the FastAPI backend
*/
class APIClient {
constructor() {
this.baseURL = this.getBaseURL();
this.token = null;
this.headers = {
'Content-Type': 'application/json'
};
}
/**
* Get the base URL for API calls
* Automatically detects if running in container or development
*/
getBaseURL() {
const protocol = window.location.protocol;
const hostname = window.location.hostname;
// If running on localhost, assume development mode
if (hostname === 'localhost' || hostname === '127.0.0.1') {
return `${protocol}//${hostname}:8000`;
}
// If running in production behind a reverse proxy, use /api path
return `${protocol}//${hostname}/api`;
}
/**
* Set authentication token
*/
setToken(token) {
this.token = token;
this.headers['Authorization'] = `Bearer ${token}`;
}
/**
* Clear authentication token
*/
clearToken() {
this.token = null;
delete this.headers['Authorization'];
}
/**
* Make HTTP request to API
*/
async request(endpoint, options = {}) {
const url = `${this.baseURL}${endpoint}`;
// Merge options, with custom headers taking precedence
const config = {
...options,
headers: {
...this.headers,
...(options.headers || {})
}
};
try {
const response = await fetch(url, config);
// Handle different response types
if (response.status === 204) {
return null; // No content
}
const contentType = response.headers.get('content-type');
let data;
if (contentType && contentType.includes('application/json')) {
data = await response.json();
} else {
data = await response.text();
}
if (!response.ok) {
throw new APIError(
data.detail || data || `HTTP ${response.status}`,
response.status,
data
);
}
return data;
} catch (error) {
if (error instanceof APIError) {
throw error;
}
// Network or other errors
throw new APIError(
'Network error or API unavailable',
0,
{ originalError: error.message }
);
}
}
// Health and authentication methods
async checkHealth() {
return this.request('/health');
}
async getConfig() {
return this.request('/config');
}
async testAuth() {
return this.request('/');
}
async login(username, password) {
// Don't include Authorization header for login
const tempHeaders = { ...this.headers };
delete tempHeaders['Authorization'];
const response = await this.request('/auth/login', {
method: 'POST',
headers: tempHeaders,
body: JSON.stringify({ username, password })
});
// Set the token from the response
if (response.access_token) {
this.setToken(response.access_token);
}
return response;
}
async logout() {
try {
await this.request('/auth/logout', {
method: 'POST'
});
} finally {
// Clear token even if logout fails
this.clearToken();
}
}
async getCurrentUser() {
return this.request('/auth/me');
}
// Mailing Lists API
async getLists() {
return this.request('/lists');
}
async getList(listId) {
return this.request(`/lists/${listId}`);
}
async createList(listData) {
return this.request('/lists', {
method: 'POST',
body: JSON.stringify(listData)
});
}
async updateList(listId, listData) {
return this.request(`/lists/${listId}`, {
method: 'PATCH',
body: JSON.stringify(listData)
});
}
async deleteList(listId) {
return this.request(`/lists/${listId}`, {
method: 'DELETE'
});
}
// Members API
async getMembers() {
return this.request('/members');
}
async getMember(memberId) {
return this.request(`/members/${memberId}`);
}
async createMember(memberData) {
return this.request('/members', {
method: 'POST',
body: JSON.stringify(memberData)
});
}
async updateMember(memberId, memberData) {
return this.request(`/members/${memberId}`, {
method: 'PATCH',
body: JSON.stringify(memberData)
});
}
async deleteMember(memberId) {
return this.request(`/members/${memberId}`, {
method: 'DELETE'
});
}
// Subscriptions API
async getListMembers(listId) {
return this.request(`/lists/${listId}/members`);
}
async createSubscription(subscriptionData) {
return this.request('/subscriptions', {
method: 'POST',
body: JSON.stringify(subscriptionData)
});
}
async deleteSubscription(listEmail, memberEmail) {
const params = new URLSearchParams({
list_email: listEmail,
member_email: memberEmail
});
return this.request(`/subscriptions?${params}`, {
method: 'DELETE'
});
}
/**
* Bulk import members from CSV data
*/
async bulkImportMembers(csvData, listIds) {
return this.request('/bulk-import', {
method: 'POST',
body: JSON.stringify({
csv_data: csvData,
list_ids: listIds
})
});
}
// User Management API
async getUsers() {
return this.request('/users');
}
async createUser(userData) {
return this.request('/users', {
method: 'POST',
body: JSON.stringify(userData)
});
}
async updateUser(userId, userData) {
return this.request(`/users/${userId}`, {
method: 'PATCH',
body: JSON.stringify(userData)
});
}
async deleteUser(userId) {
return this.request(`/users/${userId}`, {
method: 'DELETE'
});
}
// Bounce Management API
async getMemberBounces(memberId) {
return this.request(`/members/${memberId}/bounces`);
}
async resetBounceStatus(memberId) {
return this.request(`/members/${memberId}/bounce-status`, {
method: 'PATCH'
});
}
}
/**
* Custom API Error class
*/
class APIError extends Error {
constructor(message, status = 0, details = null) {
super(message);
this.name = 'APIError';
this.status = status;
this.details = details;
}
isAuthError() {
return this.status === 401;
}
isNotFound() {
return this.status === 404;
}
isBadRequest() {
return this.status === 400;
}
isServerError() {
return this.status >= 500;
}
}
// Create global API client instance
window.apiClient = new APIClient();
window.APIError = APIError;

914
web/static/js/app.js Normal file
View File

@@ -0,0 +1,914 @@
/**
* Main Application Controller
* Handles authentication, data loading, and view rendering
*/
class MailingListApp {
constructor() {
this.isAuthenticated = false;
this.currentUser = null;
this.lists = [];
this.members = [];
this.users = [];
this.subscriptions = new Map(); // list_id -> members[]
this.initializeApp();
}
/**
* Initialize the application
*/
async initializeApp() {
this.setupEventListeners();
// Check for saved token
const savedToken = localStorage.getItem('authToken');
if (savedToken) {
apiClient.setToken(savedToken);
await this.checkCurrentUser();
}
}
/**
* Setup event listeners
*/
setupEventListeners() {
// Login form submission
document.getElementById('loginForm').addEventListener('submit', (e) => {
e.preventDefault();
this.handleLogin();
});
// Clear error when user starts typing
document.getElementById('username').addEventListener('input', () => {
this.clearLoginError();
});
document.getElementById('password').addEventListener('input', () => {
this.clearLoginError();
});
document.getElementById('logoutBtn').addEventListener('click', () => {
this.logout();
});
// User dropdown functionality
const userDropdownTrigger = document.getElementById('userDropdownTrigger');
const userDropdown = document.getElementById('userDropdown');
userDropdownTrigger.addEventListener('click', (e) => {
e.preventDefault();
e.stopPropagation();
userDropdown.classList.toggle('active');
userDropdownTrigger.classList.toggle('active');
});
// Close dropdown when clicking outside
document.addEventListener('click', (e) => {
if (!userDropdown.contains(e.target)) {
userDropdown.classList.remove('active');
userDropdownTrigger.classList.remove('active');
}
});
// Close dropdown on escape key
document.addEventListener('keydown', (e) => {
if (e.key === 'Escape') {
userDropdown.classList.remove('active');
userDropdownTrigger.classList.remove('active');
}
});
// Bulk import button
document.getElementById('showBulkImportBtn').addEventListener('click', () => {
uiManager.showBulkImportModal();
});
// Add user button (admin only)
document.getElementById('addUserBtn').addEventListener('click', () => {
uiManager.showUserModal();
});
// User management dropdown item
document.getElementById('userManagementBtn').addEventListener('click', () => {
// Close the dropdown
userDropdown.classList.remove('active');
userDropdownTrigger.classList.remove('active');
// Switch to users tab
this.switchToUsersTab();
});
// Member search functionality
const memberSearchInput = document.getElementById('memberSearchInput');
const memberSearchClear = document.getElementById('memberSearchClear');
memberSearchInput.addEventListener('input', (e) => {
this.filterMembers(e.target.value);
// Show/hide clear button
if (e.target.value.trim()) {
memberSearchClear.style.display = 'flex';
} else {
memberSearchClear.style.display = 'none';
}
});
memberSearchClear.addEventListener('click', () => {
memberSearchInput.value = '';
memberSearchClear.style.display = 'none';
this.filterMembers('');
memberSearchInput.focus();
});
// Clear search when switching tabs
document.querySelectorAll('.tab-btn').forEach(btn => {
btn.addEventListener('click', () => {
if (btn.dataset.tab !== 'members') {
memberSearchInput.value = '';
memberSearchClear.style.display = 'none';
}
});
});
}
/**
* Check if current user is still valid
*/
async checkCurrentUser() {
try {
const user = await apiClient.getCurrentUser();
this.currentUser = user;
this.isAuthenticated = true;
this.showAuthenticatedUI();
await this.loadData();
} catch (error) {
// Token is invalid, clear it
this.logout();
}
}
/**
* Handle login button click
*/
async handleLogin() {
const usernameInput = document.getElementById('username');
const passwordInput = document.getElementById('password');
const username = usernameInput.value.trim();
const password = passwordInput.value.trim();
// Clear previous error states
this.clearLoginError();
if (!username || !password) {
this.showLoginError('Please enter both username and password');
return;
}
await this.login(username, password);
}
/**
* Show login error message
*/
showLoginError(message) {
const errorDiv = document.getElementById('loginError');
const errorMessage = document.getElementById('loginErrorMessage');
const usernameInput = document.getElementById('username');
const passwordInput = document.getElementById('password');
errorMessage.textContent = message;
errorDiv.style.display = 'flex';
// Add error class to inputs
usernameInput.classList.add('error');
passwordInput.classList.add('error');
// Remove animation class and re-add to trigger animation
errorDiv.style.animation = 'none';
setTimeout(() => {
errorDiv.style.animation = '';
}, 10);
}
/**
* Clear login error message
*/
clearLoginError() {
const errorDiv = document.getElementById('loginError');
const usernameInput = document.getElementById('username');
const passwordInput = document.getElementById('password');
errorDiv.style.display = 'none';
usernameInput.classList.remove('error');
passwordInput.classList.remove('error');
}
/**
* Authenticate with API
*/
async login(username, password) {
try {
uiManager.setLoading(true);
// Login and get token
const response = await apiClient.login(username, password);
this.currentUser = response.user;
this.isAuthenticated = true;
// Save token
localStorage.setItem('authToken', response.access_token);
this.showAuthenticatedUI();
await this.loadData();
uiManager.showNotification(`Welcome back, ${this.currentUser.username}!`, 'success');
} catch (error) {
this.isAuthenticated = false;
this.currentUser = null;
apiClient.clearToken();
localStorage.removeItem('authToken');
// Show error on login form
let errorMessage = 'Login failed';
if (error.message) {
if (error.message.includes('401') || error.message.includes('Unauthorized') ||
error.message.includes('Invalid') || error.message.includes('credentials')) {
errorMessage = 'Invalid username or password';
} else if (error.message.includes('network') || error.message.includes('fetch')) {
errorMessage = 'Unable to connect to server';
} else {
errorMessage = error.message;
}
}
this.showLoginError(errorMessage);
} finally {
uiManager.setLoading(false);
}
}
/**
* Logout
*/
async logout() {
try {
await apiClient.logout();
} catch (error) {
// Ignore logout errors
}
this.isAuthenticated = false;
this.currentUser = null;
apiClient.clearToken();
localStorage.removeItem('authToken');
this.showUnauthenticatedUI();
uiManager.showNotification('Logged out successfully', 'info');
}
/**
* Show authenticated UI
*/
showAuthenticatedUI() {
document.getElementById('loginPage').style.display = 'none';
document.getElementById('mainHeader').style.display = 'block';
document.getElementById('mainContent').style.display = 'block';
// Clear login inputs
document.getElementById('username').value = '';
document.getElementById('password').value = '';
// Update user info display
if (this.currentUser) {
document.getElementById('currentUsername').textContent = this.currentUser.username;
document.getElementById('currentUserRole').textContent = this.currentUser.role;
document.getElementById('dropdownUsername').textContent = this.currentUser.username;
document.getElementById('dropdownUserRole').textContent = this.currentUser.role;
// Show/hide admin-only features
const isAdmin = this.currentUser.role === 'administrator';
document.getElementById('userManagementBtn').style.display = isAdmin ? 'block' : 'none';
document.getElementById('userManagementDivider').style.display = isAdmin ? 'block' : 'none';
// Show/hide write access features
const hasWriteAccess = this.currentUser.role === 'administrator' || this.currentUser.role === 'operator';
this.updateUIForPermissions(hasWriteAccess);
}
}
/**
* Show unauthenticated UI
*/
showUnauthenticatedUI() {
document.getElementById('loginPage').style.display = 'flex';
document.getElementById('mainHeader').style.display = 'none';
document.getElementById('mainContent').style.display = 'none';
}
/**
* Update UI elements based on user permissions
*/
updateUIForPermissions(hasWriteAccess) {
// Disable/enable write action buttons
const writeButtons = document.querySelectorAll('[data-requires-write]');
writeButtons.forEach(button => {
button.style.display = hasWriteAccess ? '' : 'none';
});
// Update button attributes for later reference
document.getElementById('addListBtn').setAttribute('data-requires-write', '');
document.getElementById('addMemberBtn').setAttribute('data-requires-write', '');
document.getElementById('showBulkImportBtn').setAttribute('data-requires-write', '');
}
/**
* Switch to users tab (triggered from user dropdown)
*/
switchToUsersTab() {
// Switch to users tab programmatically
const tabButtons = document.querySelectorAll('.tab-btn');
const tabContents = document.querySelectorAll('.tab-content');
// Remove active class from all tabs and contents
tabButtons.forEach(btn => btn.classList.remove('active'));
tabContents.forEach(content => content.classList.remove('active'));
// Show users tab content
const usersTab = document.getElementById('users-tab');
if (usersTab) {
usersTab.classList.add('active');
}
// Load users data if needed
this.loadUsers();
}
/**
* Load all data from API
*/
async loadData() {
if (!this.isAuthenticated) return;
try {
uiManager.setLoading(true);
// Load configuration, lists and members in parallel
const [config, lists, members] = await Promise.all([
apiClient.getConfig(),
apiClient.getLists(),
apiClient.getMembers()
]);
this.config = config;
this.lists = lists;
this.members = members;
// Load users if admin
if (this.currentUser && this.currentUser.role === 'administrator') {
try {
this.users = await apiClient.getUsers();
} catch (error) {
console.warn('Failed to load users:', error);
this.users = [];
}
}
// Load subscriptions for each list
await this.loadSubscriptions();
// Render all views
this.renderLists();
this.renderMembers();
if (this.currentUser && this.currentUser.role === 'administrator') {
this.renderUsers();
}
} catch (error) {
uiManager.handleError(error, 'Failed to load data');
} finally {
uiManager.setLoading(false);
}
}
/**
* Load subscription data for all lists
*/
async loadSubscriptions() {
this.subscriptions.clear();
const subscriptionPromises = this.lists.map(async (list) => {
try {
const members = await apiClient.getListMembers(list.list_id);
this.subscriptions.set(list.list_id, members);
} catch (error) {
console.warn(`Failed to load members for list ${list.list_id}:`, error);
this.subscriptions.set(list.list_id, []);
}
});
await Promise.all(subscriptionPromises);
}
/**
* Render mailing lists table
*/
renderLists() {
const tbody = document.getElementById('listsTableBody');
tbody.innerHTML = '';
if (this.lists.length === 0) {
tbody.innerHTML = `
<tr>
<td colspan="6" class="text-center text-muted">
No mailing lists found. <a href="#" id="createFirstList">Create your first list</a>
</td>
</tr>
`;
document.getElementById('createFirstList').addEventListener('click', (e) => {
e.preventDefault();
uiManager.showListModal();
});
return;
}
const hasWriteAccess = this.currentUser && (this.currentUser.role === 'administrator' || this.currentUser.role === 'operator');
this.lists.forEach(list => {
const row = document.createElement('tr');
const memberCount = this.subscriptions.get(list.list_id)?.length || 0;
row.innerHTML = `
<td>
<div class="font-medium">${uiManager.escapeHtml(list.list_name)}</div>
</td>
<td>
<a href="mailto:${list.list_email}" style="color: var(--primary-color)">
${uiManager.escapeHtml(list.list_email)}
</a>
</td>
<td>
<div class="text-sm text-muted">
${list.description ? uiManager.escapeHtml(list.description) : '<em>No description</em>'}
</div>
</td>
<td>
<span class="font-medium">${memberCount}</span>
${memberCount === 1 ? 'member' : 'members'}
</td>
<td></td>
<td>
<div class="action-buttons"></div>
</td>
`;
// Add status badge
const statusCell = row.cells[4];
statusCell.appendChild(uiManager.createStatusBadge(list.active));
// Add action buttons only for users with write access
const actionsCell = row.cells[5].querySelector('.action-buttons');
if (hasWriteAccess) {
const editBtn = uiManager.createActionButton('Edit', 'edit', 'btn-secondary', () => {
uiManager.showListModal(list);
});
const deleteBtn = uiManager.createActionButton('Delete', 'trash', 'btn-danger', () => {
uiManager.showConfirmation(
`Are you sure you want to delete the mailing list "${list.list_name}"? This action cannot be undone.`,
async () => {
await this.deleteList(list.list_id);
}
);
});
actionsCell.appendChild(editBtn);
actionsCell.appendChild(deleteBtn);
}
tbody.appendChild(row);
});
}
/**
* Render members table
*/
renderMembers() {
const tbody = document.getElementById('membersTableBody');
tbody.innerHTML = '';
if (this.members.length === 0) {
tbody.innerHTML = `
<tr>
<td colspan="5" class="text-center text-muted">
No members found. <a href="#" id="createFirstMember">Add your first member</a>
</td>
</tr>
`;
document.getElementById('createFirstMember').addEventListener('click', (e) => {
e.preventDefault();
uiManager.showMemberModal();
});
return;
}
const hasWriteAccess = this.currentUser && (this.currentUser.role === 'administrator' || this.currentUser.role === 'operator');
this.members.forEach(member => {
const row = document.createElement('tr');
// Find lists this member belongs to
const memberLists = [];
this.subscriptions.forEach((members, listId) => {
if (members.some(m => m.member_id === member.member_id)) {
const list = this.lists.find(l => l.list_id === listId);
if (list) {
memberLists.push(list.list_name);
}
}
});
row.innerHTML = `
<td>
<div class="font-medium">${uiManager.escapeHtml(member.name)}</div>
${this.config?.bounce_handling_enabled && member.bounce_count > 0 ? `<div class="text-xs text-muted" style="margin-top: 2px;"></div>` : ''}
</td>
<td>
<a href="mailto:${member.email}" style="color: var(--primary-color)">
${uiManager.escapeHtml(member.email)}
</a>
</td>
<td>
<div class="text-sm">
${memberLists.length > 0
? memberLists.map(name => `<span class="text-muted">${uiManager.escapeHtml(name)}</span>`).join(', ')
: '<em class="text-muted">No subscriptions</em>'
}
</div>
</td>
<td></td>
<td>
<div class="action-buttons"></div>
</td>
`;
// Add bounce badge if member has bounces (only if bounce handling is enabled)
if (this.config?.bounce_handling_enabled && member.bounce_count > 0) {
const bounceInfoDiv = row.cells[0].querySelector('.text-xs');
const bounceBadge = uiManager.createBounceStatusBadge(member.bounce_status, member.bounce_count);
if (bounceBadge) {
bounceInfoDiv.appendChild(bounceBadge);
}
}
// Add status badge
const statusCell = row.cells[3];
statusCell.appendChild(uiManager.createStatusBadge(member.active));
// Add action buttons
const actionsCell = row.cells[4].querySelector('.action-buttons');
// Create Lists button with proper subscription modal functionality
const subscriptionsBtn = document.createElement('button');
subscriptionsBtn.className = 'btn btn-sm btn-primary';
subscriptionsBtn.innerHTML = '<i class="fas fa-list"></i> Lists';
subscriptionsBtn.title = 'Manage Subscriptions';
subscriptionsBtn.addEventListener('click', () => {
uiManager.showMemberSubscriptionsModal(member);
});
// Create Bounces button (show if bounce handling is enabled and member has bounces or for admins/operators)
if (this.config?.bounce_handling_enabled && (member.bounce_count > 0 || hasWriteAccess)) {
const bouncesBtn = document.createElement('button');
bouncesBtn.className = `btn btn-sm ${member.bounce_count > 0 ? 'btn-warning' : 'btn-secondary'}`;
bouncesBtn.innerHTML = `<i class="fas fa-exclamation-triangle"></i> Bounces${member.bounce_count > 0 ? ` (${member.bounce_count})` : ''}`;
bouncesBtn.title = 'View Bounce History';
bouncesBtn.addEventListener('click', () => {
uiManager.showBounceHistoryModal(member);
});
actionsCell.appendChild(bouncesBtn);
}
// Only show edit/delete buttons for users with write access
if (hasWriteAccess) {
const editBtn = uiManager.createActionButton('Edit', 'edit', 'btn-secondary', () => {
uiManager.showMemberModal(member);
});
const deleteBtn = uiManager.createActionButton('Delete', 'trash', 'btn-danger', () => {
uiManager.showConfirmation(
`Are you sure you want to delete the member "${member.name}"? This will remove them from all mailing lists.`,
async () => {
await this.deleteMember(member.member_id);
}
);
});
actionsCell.appendChild(subscriptionsBtn);
actionsCell.appendChild(editBtn);
actionsCell.appendChild(deleteBtn);
} else {
actionsCell.appendChild(subscriptionsBtn);
}
tbody.appendChild(row);
});
}
/**
* Render users table (admin only)
*/
renderUsers() {
const tbody = document.getElementById('usersTableBody');
tbody.innerHTML = '';
if (this.users.length === 0) {
tbody.innerHTML = `
<tr>
<td colspan="6" class="text-center text-muted">
No users found. <a href="#" id="createFirstUser">Create your first user</a>
</td>
</tr>
`;
document.getElementById('createFirstUser').addEventListener('click', (e) => {
e.preventDefault();
uiManager.showUserModal();
});
return;
}
this.users.forEach(user => {
const row = document.createElement('tr');
// Format dates
const createdAt = new Date(user.created_at).toLocaleDateString();
const lastLogin = user.last_login ? new Date(user.last_login).toLocaleDateString() : 'Never';
row.innerHTML = `
<td>
<div class="font-medium">${uiManager.escapeHtml(user.username)}</div>
</td>
<td>
<span class="role-badge role-${user.role}">${user.role.replace('-', ' ')}</span>
</td>
<td>
<div class="text-sm text-muted">${createdAt}</div>
</td>
<td>
<div class="text-sm text-muted">${lastLogin}</div>
</td>
<td></td>
<td>
<div class="action-buttons"></div>
</td>
`;
// Add status badge
const statusCell = row.cells[4];
statusCell.appendChild(uiManager.createStatusBadge(user.active));
// Add action buttons
const actionsCell = row.cells[5].querySelector('.action-buttons');
const editBtn = uiManager.createActionButton('Edit', 'edit', 'btn-secondary', () => {
uiManager.showUserModal(user);
});
// Don't allow deletion of current user
if (user.user_id !== this.currentUser.user_id) {
const deleteBtn = uiManager.createActionButton('Delete', 'trash', 'btn-danger', () => {
uiManager.showConfirmation(
`Are you sure you want to delete the user "${user.username}"? This action cannot be undone.`,
async () => {
await this.deleteUser(user.user_id);
}
);
});
actionsCell.appendChild(editBtn);
actionsCell.appendChild(deleteBtn);
} else {
actionsCell.appendChild(editBtn);
}
tbody.appendChild(row);
});
}
/**
* Delete a mailing list
*/
async deleteList(listId) {
try {
uiManager.setLoading(true);
await apiClient.deleteList(listId);
uiManager.showNotification('Mailing list deleted successfully', 'success');
await this.loadData();
} catch (error) {
uiManager.handleError(error, 'Failed to delete mailing list');
} finally {
uiManager.setLoading(false);
}
}
/**
* Delete a member
*/
async deleteMember(memberId) {
try {
uiManager.setLoading(true);
await apiClient.deleteMember(memberId);
uiManager.showNotification('Member deleted successfully', 'success');
await this.loadData();
} catch (error) {
uiManager.handleError(error, 'Failed to delete member');
} finally {
uiManager.setLoading(false);
}
}
/**
* Unsubscribe member from list
*/
async unsubscribeMember(listEmail, memberEmail) {
try {
uiManager.setLoading(true);
await apiClient.deleteSubscription(listEmail, memberEmail);
uiManager.showNotification('Member unsubscribed successfully', 'success');
await this.loadData();
} catch (error) {
uiManager.handleError(error, 'Failed to unsubscribe member');
} finally {
uiManager.setLoading(false);
}
}
/**
* Delete a user
*/
async deleteUser(userId) {
try {
uiManager.setLoading(true);
await apiClient.deleteUser(userId);
uiManager.showNotification('User deleted successfully', 'success');
await this.loadData();
} catch (error) {
uiManager.handleError(error, 'Failed to delete user');
} finally {
uiManager.setLoading(false);
}
}
/**
* Filter members by search term (name or email)
*/
filterMembers(searchTerm) {
const searchInfo = document.getElementById('memberSearchInfo');
const searchCount = document.getElementById('memberSearchCount');
const tbody = document.getElementById('membersTableBody');
if (!this.members || !Array.isArray(this.members)) {
return;
}
// If no search term, show all members
if (!searchTerm || searchTerm.trim() === '') {
this.renderMembers();
searchInfo.style.display = 'none';
return;
}
const normalizedSearch = searchTerm.trim().toLowerCase();
// Filter members by name or email
const filteredMembers = this.members.filter(member => {
const name = (member.name || '').toLowerCase();
const email = (member.email || '').toLowerCase();
return name.includes(normalizedSearch) || email.includes(normalizedSearch);
});
// Update search results info
searchCount.textContent = filteredMembers.length;
searchInfo.style.display = 'block';
// Render filtered results
this.renderFilteredMembers(filteredMembers);
}
/**
* Render filtered members (similar to renderMembers but with filtered data)
*/
renderFilteredMembers(filteredMembers) {
const tbody = document.getElementById('membersTableBody');
tbody.innerHTML = '';
if (filteredMembers.length === 0) {
// Show no results message
const row = tbody.insertRow();
const cell = row.insertCell();
cell.colSpan = 5;
cell.className = 'no-results';
cell.innerHTML = `
<i class="fas fa-search"></i>
<h3>No members found</h3>
<p>Try adjusting your search terms or check the spelling.</p>
`;
return;
}
filteredMembers.forEach(member => {
const row = tbody.insertRow();
// Name
const nameCell = row.insertCell();
nameCell.textContent = member.name;
// Email
const emailCell = row.insertCell();
const emailLink = document.createElement('a');
emailLink.href = `mailto:${member.email}`;
emailLink.textContent = member.email;
emailLink.style.color = 'var(--primary-color)';
emailCell.appendChild(emailLink);
// Lists (show member's subscriptions)
const listsCell = row.insertCell();
const memberLists = [];
this.subscriptions.forEach((members, listId) => {
if (members.some(m => m.member_id === member.member_id)) {
const list = this.lists.find(l => l.list_id === listId);
if (list) {
memberLists.push(list.list_name);
}
}
});
listsCell.textContent = memberLists.length > 0 ? memberLists.join(', ') : 'None';
// Status
const statusCell = row.insertCell();
const statusBadge = document.createElement('span');
statusBadge.className = `status-badge ${member.active ? 'active' : 'inactive'}`;
statusBadge.innerHTML = `<i class="fas fa-${member.active ? 'check' : 'times'}"></i> ${member.active ? 'Active' : 'Inactive'}`;
// Add bounce status if exists and bounce handling is enabled
if (this.config?.bounce_handling_enabled && member.bounce_status && member.bounce_status !== 'clean') {
const bounceIndicator = document.createElement('span');
bounceIndicator.className = `bounce-badge bounce-${member.bounce_status === 'hard_bounce' ? 'hard' : 'soft'}`;
bounceIndicator.innerHTML = `<i class="fas fa-exclamation-triangle"></i> Bounces`;
bounceIndicator.title = `${member.bounce_count || 0} bounces - ${member.bounce_status}`;
statusCell.appendChild(document.createElement('br'));
statusCell.appendChild(bounceIndicator);
}
statusCell.appendChild(statusBadge);
// Actions
const actionsCell = row.insertCell();
actionsCell.className = 'action-buttons';
// Bounces button (if bounce handling is enabled and member has bounce data)
if (this.config?.bounce_handling_enabled && member.bounce_count > 0) {
const bouncesBtn = uiManager.createActionButton('Bounces', 'exclamation-triangle', 'btn-warning', () => {
uiManager.showBounceHistory(member);
});
actionsCell.appendChild(bouncesBtn);
}
const subscriptionsBtn = uiManager.createActionButton('Lists', 'list', 'btn-secondary', () => {
uiManager.showMemberSubscriptions(member);
});
actionsCell.appendChild(subscriptionsBtn);
const editBtn = uiManager.createActionButton('Edit', 'edit', 'btn-primary', () => {
uiManager.showMemberModal(member);
});
editBtn.setAttribute('data-requires-write', '');
actionsCell.appendChild(editBtn);
const deleteBtn = uiManager.createActionButton('Delete', 'trash', 'btn-danger', () => {
uiManager.showConfirmation(
`Are you sure you want to delete member "${member.name}"? This will also remove them from all mailing lists.`,
async () => {
await this.deleteMember(member.member_id);
}
);
});
deleteBtn.setAttribute('data-requires-write', '');
actionsCell.appendChild(deleteBtn);
});
// Update UI permissions for the filtered results
const hasWriteAccess = this.currentUser && (this.currentUser.role === 'administrator' || this.currentUser.role === 'operator');
this.updateUIForPermissions(hasWriteAccess);
}
}
// Initialize the application when DOM is loaded
document.addEventListener('DOMContentLoaded', () => {
window.app = new MailingListApp();
});

1460
web/static/js/ui.js Normal file

File diff suppressed because it is too large Load Diff