Compare commits
12 Commits
9b6a6dab06
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f3d7592e7d | ||
|
|
12a82c8d03 | ||
|
|
b34ea2ed84 | ||
|
|
8fd951fd1f | ||
|
|
ecbc38cf8e | ||
|
|
d37027ee5a | ||
|
|
72f3297a80 | ||
|
|
ac23638125 | ||
|
|
4fc1ed96dd | ||
|
|
459e16b26f | ||
|
|
d4b88e0952 | ||
|
|
f721be7280 |
25
.env.example
25
.env.example
@@ -19,4 +19,27 @@ MYSQL_PASSWORD=change_this_password
|
||||
MYSQL_ROOT_PASSWORD=change_this_root_password
|
||||
|
||||
# API Configuration
|
||||
API_TOKEN=change_this_to_a_secure_random_token
|
||||
API_TOKEN=change_this_to_a_secure_random_token
|
||||
|
||||
# Bounce Handling Configuration (Optional)
|
||||
# Choose one of the bounce handling methods below:
|
||||
|
||||
# Method 1: SNS Webhook Bounce Handling (Requires SES Production Access)
|
||||
# Set to 'true' to enable real-time SNS webhook bounce handling
|
||||
ENABLE_SNS_WEBHOOKS=false
|
||||
ENABLE_BOUNCE_HANDLING=false
|
||||
|
||||
# If ENABLE_SNS_WEBHOOKS=true, you need:
|
||||
# 1. AWS SNS topic configured
|
||||
# 2. SES configured to send notifications to SNS topic
|
||||
# 3. Valid HTTPS domain for webhook endpoint
|
||||
# 4. SNS subscription confirmed to your webhook endpoint
|
||||
|
||||
# Method 2: Email-Based Bounce Processing (Works with SES Sandbox)
|
||||
# Set to 'true' to enable email-based bounce processing
|
||||
# This processes bounce emails that are sent to bounces@lists.sasalliance.org
|
||||
ENABLE_EMAIL_BOUNCE_PROCESSING=false
|
||||
|
||||
# Note: You can enable both methods, but typically only one is needed
|
||||
# Email-based processing works with SES sandbox accounts
|
||||
# SNS webhooks provide real-time processing but require SES production access
|
||||
172
.github/copilot-instructions.md
vendored
172
.github/copilot-instructions.md
vendored
@@ -1,97 +1,169 @@
|
||||
# Copilot Instructions: Mail List Manager (Postfix + SES + Web Interface)
|
||||
# Copilot Instructions: Mail List Manager (Postfix + SES + MySQL + API + Web)
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
This is a containerized mailing list management system built around Postfix as an SMTP relay through Amazon SES. **Currently in Phase 1** with static configuration; **planned expansion** includes web frontend and SQL database for dynamic member management.
|
||||
This is a complete containerized mailing list management system with four services:
|
||||
|
||||
**Current Components (Phase 1):**
|
||||
- `docker-compose.yaml`: Single-service orchestration with SES credentials
|
||||
- `postfix/`: Complete Postfix container configuration
|
||||
- Static virtual aliases system for mailing list distribution
|
||||
1. **Web Frontend** (Port 3000) - Nginx serving modern HTML/CSS/JS interface
|
||||
2. **REST API** (Port 8000) - FastAPI backend with token authentication
|
||||
3. **MySQL Database** (Internal) - Stores lists, members, and subscriptions
|
||||
4. **Postfix Mail Server** (Port 25) - SMTP server with native MySQL integration
|
||||
|
||||
**Planned Architecture (Phase 2+):**
|
||||
- Web frontend for list management (view/add/remove members)
|
||||
- SQL database backend for member storage
|
||||
- Dynamic Postfix configuration generation
|
||||
- Multi-service Docker Compose setup
|
||||
**Key Architecture Principles:**
|
||||
- Real-time operation: Changes take effect immediately without Postfix reload
|
||||
- Native integration: Postfix queries MySQL directly (no Python scripts)
|
||||
- Security-first: Private Docker network, token auth, sender whitelist
|
||||
- User-friendly: Web interface for easy management by non-technical users
|
||||
|
||||
## Configuration Pattern
|
||||
|
||||
**Current (Static):** Environment variable substitution for secure credential management:
|
||||
**Dynamic Database-Driven Architecture:**
|
||||
|
||||
1. **Credentials Flow**: SES credentials passed via environment → `sasl_passwd.template` → runtime generation in `entrypoint.sh`
|
||||
2. **Config Files**: Static configs (`main.cf`, `virtual_aliases.cf`) + dynamic SASL auth file
|
||||
3. **Postfix Maps**: Hash databases generated at build time (virtual aliases) and runtime (SASL)
|
||||
1. **Data Flow**: Web UI → REST API → MySQL ← Postfix (real-time queries)
|
||||
2. **Credentials**: All sensitive data in `.env` file (SES, MySQL, API_TOKEN)
|
||||
3. **Mail Processing**:
|
||||
- Email arrives for `list@lists.sasalliance.org`
|
||||
- Postfix queries MySQL via `mysql_virtual_alias_maps.cf`
|
||||
- MySQL returns comma-separated member emails
|
||||
- Postfix expands alias and delivers via SES
|
||||
4. **Security**: Private Docker network (maillist-internal), sender whitelist (@sasalliance.org)
|
||||
|
||||
**Future (Dynamic):** Database-driven configuration:
|
||||
- Member lists stored in SQL database, with members able to join multiple lists
|
||||
- Web interface for CRUD operations on members
|
||||
- `virtual_aliases.cf` generated from database at runtime
|
||||
- Postfix reload triggered by configuration changes
|
||||
**No Static Configuration:**
|
||||
- Virtual aliases are stored in MySQL database, not config files
|
||||
- Changes take effect immediately without container restarts
|
||||
- Postfix caches query results for performance
|
||||
|
||||
## Key Files and Their Roles
|
||||
|
||||
- `main.cf`: Core Postfix config - relay through SES, domain settings, security
|
||||
- `sasl_passwd.template`: Template for SES authentication (uses `${SES_USER}:${SES_PASS}`)
|
||||
- `virtual_aliases.cf`: Static email forwarding rules (one mailing list currently)
|
||||
- `entrypoint.sh`: Runtime credential processing and Postfix startup
|
||||
### Docker & Environment
|
||||
- `docker-compose.yaml`: Multi-service orchestration (mysql, postfix, api, web)
|
||||
- `.env`: All credentials and configuration (SES, MySQL, API_TOKEN)
|
||||
|
||||
### Web Frontend (`web/`)
|
||||
- `index.html`: Single-page application interface
|
||||
- `static/js/api.js`: API client with authentication
|
||||
- `static/js/ui.js`: UI helpers and modal management
|
||||
- `static/js/app.js`: Main application controller
|
||||
- `static/css/style.css`: Complete styling system
|
||||
- `nginx.conf`: Nginx configuration for static file serving
|
||||
|
||||
### REST API (`api/`)
|
||||
- `main.py`: FastAPI application with all CRUD endpoints
|
||||
- `requirements.txt`: Python dependencies (fastapi, mysql-connector-python, etc.)
|
||||
- `Dockerfile`: Python 3.11 container build
|
||||
|
||||
### Database (`database/`)
|
||||
- `schema.sql`: Database initialization with tables and sample data
|
||||
- Tables: `lists`, `members`, `list_members` (many-to-many junction)
|
||||
|
||||
### Postfix (`postfix/`)
|
||||
- `main.cf.template`: Core Postfix config with MySQL virtual alias maps
|
||||
- `mysql_virtual_alias_maps.cf`: Native Postfix MySQL query configuration
|
||||
- `sender_access`: Sender whitelist (sasalliance.org domain allowed)
|
||||
- `sasl_passwd.template`: SES authentication template
|
||||
- `entrypoint.sh`: Container startup with MySQL health check and config generation
|
||||
|
||||
## Development Workflows
|
||||
|
||||
**Building and Running:**
|
||||
|
||||
Always use sudo for Docker commands. Don't bother opening the simple browser UI for Docker as it is not very useful.
|
||||
|
||||
```bash
|
||||
docker-compose up --build # Build and start mail server
|
||||
docker-compose logs -f # Monitor mail delivery logs
|
||||
sudo docker-compose up --build # Build and start all services
|
||||
sudo docker-compose logs -f # Monitor all service logs
|
||||
sudo docker-compose logs -f api # Monitor specific service
|
||||
```
|
||||
|
||||
**Adding Mailing Lists (Current):**
|
||||
1. Edit `virtual_aliases.cf` with new list → recipients mapping
|
||||
2. Rebuild container (postmap runs at build time)
|
||||
3. No restart needed for existing virtual alias changes
|
||||
**Managing Lists and Members:**
|
||||
|
||||
**Future Web Interface Workflow:**
|
||||
1. Access web frontend at configured port
|
||||
2. Use CRUD interface to manage mailing lists and members
|
||||
3. Changes automatically update database and regenerate Postfix configs
|
||||
**Via Web Interface (Recommended):**
|
||||
1. Open http://localhost:3000
|
||||
2. Enter API_TOKEN from .env file
|
||||
3. Use intuitive interface to manage lists, members, and subscriptions
|
||||
4. Click "Subscriptions" button on any member for toggle-based subscription management
|
||||
|
||||
**Via REST API:**
|
||||
```bash
|
||||
# Get all lists
|
||||
curl -H "Authorization: Bearer $API_TOKEN" http://localhost:8000/lists
|
||||
|
||||
# Create new list
|
||||
curl -X POST http://localhost:8000/lists \
|
||||
-H "Authorization: Bearer $API_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"list_name":"Test","list_email":"test@lists.sasalliance.org","active":true}'
|
||||
|
||||
# Subscribe member to list
|
||||
curl -X POST http://localhost:8000/subscriptions \
|
||||
-H "Authorization: Bearer $API_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"list_email":"test@lists.sasalliance.org","member_email":"user@example.com"}'
|
||||
```
|
||||
|
||||
**Via Direct Database Access:**
|
||||
```bash
|
||||
docker-compose exec mysql mysql -u maillist -p maillist
|
||||
# See database/README.md for SQL examples
|
||||
```
|
||||
|
||||
**Testing Mail Delivery:**
|
||||
```bash
|
||||
# From inside container
|
||||
docker-compose exec postfix bash
|
||||
echo "Test message" | mail -s "Subject" community@lists.sasalliance.org
|
||||
|
||||
# Check logs for SES relay status
|
||||
docker-compose logs postfix | grep -E "(sent|bounced|deferred)"
|
||||
|
||||
# Verify MySQL query works
|
||||
docker-compose exec postfix postmap -q "community@lists.sasalliance.org" \
|
||||
mysql:/etc/postfix/mysql_virtual_alias_maps.cf
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- SES credentials exposed in docker-compose.yaml (consider using Docker secrets)
|
||||
- All credentials stored in `.env` file (git-ignored)
|
||||
- MySQL on private Docker network (no host port mapping)
|
||||
- API requires Bearer token authentication
|
||||
- Sender whitelist restricts who can send to lists (@sasalliance.org only)
|
||||
- SASL password file permissions set to 600 in entrypoint
|
||||
- TLS encryption enforced for SES relay (`smtp_tls_security_level = encrypt`)
|
||||
- Only localhost and configured hostname accepted for local delivery
|
||||
- Web interface requires token for all operations
|
||||
|
||||
## Configuration Conventions
|
||||
|
||||
- **Hostname Pattern**: `lists.sasalliance.org` for mailing lists, origin domain `sasalliance.org`
|
||||
- **Virtual Aliases**: Simple format `listname@lists.domain → recipient1, recipient2`
|
||||
- **Virtual Aliases**: Dynamically queried from MySQL database in real-time
|
||||
- **SES Region**: EU West 2 (`email-smtp.eu-west-2.amazonaws.com`)
|
||||
- **Port Mapping**: Standard SMTP port 25 exposed to host
|
||||
- **Database Tables**: `lists`, `members`, `list_members` (many-to-many design)
|
||||
- **API Port**: 8000 for REST API
|
||||
- **Web Port**: 3000 for web interface
|
||||
|
||||
## Common Modifications
|
||||
|
||||
**Adding Recipients to Existing List (Current):**
|
||||
Edit `virtual_aliases.cf`, add comma-separated emails, rebuild container.
|
||||
**Adding Recipients to a List:**
|
||||
Use web interface or API - changes are immediate!
|
||||
|
||||
**New Mailing List (Current):**
|
||||
Add line to `virtual_aliases.cf`: `newlist@lists.sasalliance.org recipients...`
|
||||
**Creating New Mailing List:**
|
||||
1. Via Web: Click "Add List" button, fill in form
|
||||
2. Via API: POST to /lists endpoint with list details
|
||||
3. Via MySQL: INSERT into lists table
|
||||
|
||||
**Adding New Member:**
|
||||
1. Via Web: Click "Add Member" button in Members tab
|
||||
2. Via API: POST to /members endpoint
|
||||
3. Via MySQL: INSERT into members table
|
||||
|
||||
**Managing Subscriptions:**
|
||||
1. Via Web: Click "Subscriptions" button on member, toggle lists
|
||||
2. Via API: POST/DELETE to /subscriptions endpoint
|
||||
3. Via MySQL: INSERT/DELETE in list_members table
|
||||
|
||||
**Credential Updates:**
|
||||
Update `SES_USER`/`SES_PASS` in docker-compose.yaml, restart container.
|
||||
|
||||
## Migration Considerations
|
||||
|
||||
When implementing the web frontend and database backend:
|
||||
- Preserve existing `community@lists.sasalliance.org` list during migration
|
||||
- Consider migration script to import current virtual aliases into database
|
||||
- Plan for zero-downtime deployment with database-driven config generation
|
||||
- Web interface should validate email addresses and handle duplicate members
|
||||
1. Edit `.env` file with new credentials
|
||||
2. Restart affected services:
|
||||
- SES: `docker-compose restart postfix`
|
||||
- MySQL: `docker-compose restart mysql`
|
||||
- API: `docker-compose restart api`
|
||||
277
BOUNCE_HANDLING_SETUP.md
Normal file
277
BOUNCE_HANDLING_SETUP.md
Normal file
@@ -0,0 +1,277 @@
|
||||
# SES SNS Bounce Handling Setup (Optional)
|
||||
|
||||
**⚠️ NOTICE: Bounce handling is optional and disabled by default.**
|
||||
|
||||
This document describes how to configure AWS SES and SNS to handle email bounces automatically in the Mail List Manager.
|
||||
|
||||
**Prerequisites:**
|
||||
- SES production access (not available in sandbox mode)
|
||||
- Valid HTTPS domain for webhook endpoint
|
||||
- Bounce handling must be enabled in configuration
|
||||
|
||||
## Overview
|
||||
|
||||
The system can optionally use AWS Simple Notification Service (SNS) to receive real-time bounce notifications from AWS Simple Email Service (SES). When bounce handling is enabled and an email bounces:
|
||||
|
||||
1. SES sends a notification to an SNS topic
|
||||
2. SNS forwards the notification to your webhook endpoint
|
||||
3. The API processes the notification and updates the database
|
||||
4. Members with hard bounces are automatically deactivated
|
||||
5. Bounce history is tracked and displayed in the UI
|
||||
|
||||
## Bounce Status Types
|
||||
|
||||
- **Clean**: No bounces recorded
|
||||
- **Soft Bounce**: Temporary delivery issues (e.g., mailbox full, temporary server issues)
|
||||
- After 3 soft bounces, the member is marked with soft bounce status
|
||||
- **Hard Bounce**: Permanent delivery failure (e.g., invalid email address, domain doesn't exist)
|
||||
- Member is automatically deactivated and cannot receive emails
|
||||
|
||||
## Setup Instructions
|
||||
|
||||
### 1. Enable Bounce Handling
|
||||
|
||||
First, enable bounce handling in your `.env` file:
|
||||
|
||||
```bash
|
||||
# Enable SNS webhook bounce handling
|
||||
ENABLE_SNS_WEBHOOKS=true
|
||||
ENABLE_BOUNCE_HANDLING=true
|
||||
```
|
||||
|
||||
Restart the API container after making this change:
|
||||
```bash
|
||||
sudo docker-compose restart api
|
||||
```
|
||||
|
||||
### 2. Prerequisites
|
||||
|
||||
- AWS account with SES configured and verified
|
||||
- Your Mail List Manager deployed and accessible via HTTPS (required for SNS webhook)
|
||||
- Domain or subdomain for webhook (e.g., `https://lists.yourdomain.com`)
|
||||
|
||||
### 3. Create SNS Topic
|
||||
|
||||
1. Log in to AWS Console and navigate to SNS
|
||||
2. Click "Create topic"
|
||||
3. Choose "Standard" topic type
|
||||
4. Name: `ses-bounce-notifications` (or your preferred name)
|
||||
5. Display name: `SES Bounce Notifications`
|
||||
6. Click "Create topic"
|
||||
7. **Save the Topic ARN** (you'll need it in step 4) arn:aws:sns:eu-west-2:827164363113:ses-bounces
|
||||
|
||||
### 4. Subscribe Your Webhook to SNS Topic
|
||||
|
||||
1. In the SNS topic details, click "Create subscription"
|
||||
2. Protocol: `HTTPS`
|
||||
3. Endpoint: `https://yourdomain.com:8000/webhooks/sns`
|
||||
- Replace `yourdomain.com` with your actual domain
|
||||
- The API must be accessible via HTTPS (SNS doesn't support HTTP)
|
||||
4. Enable raw message delivery: **Unchecked**
|
||||
5. Click "Create subscription"
|
||||
6. The subscription will be in "PendingConfirmation" status
|
||||
|
||||
### 5. Confirm SNS Subscription
|
||||
|
||||
When you create the subscription, SNS will send a `SubscriptionConfirmation` request to your webhook endpoint. The Mail List Manager API automatically confirms this subscription.
|
||||
|
||||
1. Check your API logs:
|
||||
```bash
|
||||
sudo docker-compose logs -f api
|
||||
```
|
||||
2. You should see a log entry indicating the subscription was confirmed
|
||||
3. In the AWS SNS console, refresh the subscriptions list
|
||||
4. The status should change from "PendingConfirmation" to "Confirmed"
|
||||
|
||||
### 6. Configure SES to Send Bounce Notifications
|
||||
|
||||
1. Navigate to AWS SES console
|
||||
2. Go to "Configuration Sets" (or "Verified identities" > select your domain > "Notifications")
|
||||
3. For configuration sets:
|
||||
- Create a new configuration set or select existing
|
||||
- Add "Event destination"
|
||||
- Event types: Select **"Bounce"** (and optionally "Complaint")
|
||||
- Destination: SNS topic
|
||||
- Select your SNS topic created in step 2
|
||||
4. For verified identities:
|
||||
- Select your sending domain/email
|
||||
- Click "Edit" in the "Notifications" section
|
||||
- Bounce feedback: Select your SNS topic
|
||||
- Include original headers: Enabled (optional)
|
||||
- Click "Save changes"
|
||||
|
||||
### 7. Verify Setup
|
||||
|
||||
#### Test with a Bounce Simulator
|
||||
|
||||
AWS SES provides bounce simulator addresses:
|
||||
|
||||
```bash
|
||||
# From inside Postfix container
|
||||
docker-compose exec postfix bash
|
||||
echo "Test bounce" | mail -s "Test" bounce@simulator.amazonses.com
|
||||
```
|
||||
|
||||
Or send to your mailing list with a test recipient:
|
||||
|
||||
1. Add `bounce@simulator.amazonses.com` as a member
|
||||
2. Subscribe to a test list
|
||||
3. Send an email to the list
|
||||
|
||||
#### Check the Results
|
||||
|
||||
1. Wait a few minutes for SES to process and send the notification
|
||||
2. Check API logs:
|
||||
```bash
|
||||
sudo docker-compose logs api | grep -i bounce
|
||||
```
|
||||
3. Log in to the web UI
|
||||
4. Go to Members tab
|
||||
5. Find the test member and click the "Bounces" button
|
||||
6. You should see the bounce event recorded
|
||||
|
||||
### 8. Security Considerations
|
||||
|
||||
#### SNS Signature Verification
|
||||
|
||||
The webhook endpoint automatically verifies SNS message signatures to ensure notifications are genuine AWS messages. This prevents unauthorized parties from sending fake bounce notifications.
|
||||
|
||||
#### HTTPS Requirement
|
||||
|
||||
SNS requires HTTPS for webhooks. You'll need:
|
||||
- Valid SSL/TLS certificate for your domain
|
||||
- Reverse proxy (e.g., Nginx, Apache) in front of the API container
|
||||
- Or use AWS API Gateway as a proxy
|
||||
|
||||
#### Example Nginx Configuration
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name lists.yourdomain.com;
|
||||
|
||||
ssl_certificate /path/to/cert.pem;
|
||||
ssl_certificate_key /path/to/key.pem;
|
||||
|
||||
# Webhook endpoint
|
||||
location /webhooks/sns {
|
||||
proxy_pass http://localhost:8000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
|
||||
# Optional: proxy API for web UI
|
||||
location /api {
|
||||
proxy_pass http://localhost:8000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 9. Managing Bounces in the UI
|
||||
|
||||
#### View Bounce Status
|
||||
|
||||
In the Members tab, bounced emails are indicated with:
|
||||
- Warning badge showing bounce count
|
||||
- Color-coded status (yellow for soft bounce, red for hard bounce)
|
||||
- Last bounce timestamp
|
||||
|
||||
#### View Bounce History
|
||||
|
||||
1. Click the "Bounces" button next to a member
|
||||
2. View detailed bounce history including:
|
||||
- Bounce type (Permanent, Transient, Undetermined)
|
||||
- Bounce subtype
|
||||
- Diagnostic code from the receiving mail server
|
||||
- Timestamp of each bounce
|
||||
|
||||
#### Reset Bounce Status
|
||||
|
||||
If a member's email has been corrected or verified:
|
||||
|
||||
1. Open the bounce history modal
|
||||
2. Click "Reset Bounce Status"
|
||||
3. Confirm the action
|
||||
4. The member's bounce count is cleared and they can receive emails again
|
||||
|
||||
**Note**: Only users with write access (administrators and operators) can reset bounce status.
|
||||
|
||||
### 10. Monitoring and Maintenance
|
||||
|
||||
#### Check Bounce Logs
|
||||
|
||||
```bash
|
||||
# View all bounces in database
|
||||
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT * FROM bounce_logs ORDER BY timestamp DESC LIMIT 20;"
|
||||
|
||||
# Count bounces by type
|
||||
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT bounce_type, COUNT(*) as count FROM bounce_logs GROUP BY bounce_type;"
|
||||
|
||||
# Find members with bounces
|
||||
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT name, email, bounce_count, bounce_status FROM members WHERE bounce_count > 0;"
|
||||
```
|
||||
|
||||
#### API Health Check
|
||||
|
||||
```bash
|
||||
# Check if webhook is accessible
|
||||
curl -X POST https://yourdomain.com:8000/webhooks/sns \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"Type":"test"}'
|
||||
```
|
||||
|
||||
#### Clean Up Old Bounce Records
|
||||
|
||||
Periodically review and clean up old bounce records:
|
||||
|
||||
```sql
|
||||
-- Delete bounce logs older than 90 days
|
||||
DELETE FROM bounce_logs WHERE created_at < DATE_SUB(NOW(), INTERVAL 90 DAY);
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### SNS Subscription Not Confirming
|
||||
|
||||
- Ensure the API container is running and accessible via HTTPS
|
||||
- Check API logs for errors
|
||||
- Verify firewall rules allow HTTPS traffic to port 8000
|
||||
- Test the endpoint manually: `curl https://yourdomain.com:8000/health`
|
||||
|
||||
### Bounces Not Being Recorded
|
||||
|
||||
1. Verify SNS topic is receiving messages:
|
||||
- Check SNS topic metrics in AWS Console
|
||||
2. Verify subscription is active:
|
||||
- Check subscription status in SNS console
|
||||
3. Check API logs for webhook errors:
|
||||
```bash
|
||||
sudo docker-compose logs api | grep -i "sns\|bounce"
|
||||
```
|
||||
4. Test signature verification:
|
||||
- Temporarily add debug logging to the webhook endpoint
|
||||
|
||||
### Members Not Being Deactivated
|
||||
|
||||
- Check if bounce type is "Permanent"
|
||||
- Review member's bounce_status in database:
|
||||
```bash
|
||||
sudo docker-compose exec mysql mysql -u maillist -p maillist -e "SELECT * FROM members WHERE email='problem@example.com';"
|
||||
```
|
||||
- Verify bounce processing logic in API logs
|
||||
|
||||
### SSL Certificate Issues
|
||||
|
||||
If using self-signed certificates, SNS will reject the webhook. You must use:
|
||||
- Valid certificate from a trusted CA (Let's Encrypt, etc.)
|
||||
- Or use AWS Certificate Manager with API Gateway
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [AWS SES Bounce Handling](https://docs.aws.amazon.com/ses/latest/dg/event-publishing-retrieving-sns.html)
|
||||
- [AWS SNS HTTPS Subscriptions](https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html)
|
||||
- [SES Bounce Types](https://docs.aws.amazon.com/ses/latest/dg/notification-contents.html#bounce-types)
|
||||
240
CODE_REVIEW_FINDINGS.md
Normal file
240
CODE_REVIEW_FINDINGS.md
Normal file
@@ -0,0 +1,240 @@
|
||||
# Code Review Findings & Fixes
|
||||
|
||||
## Date: 13 October 2025
|
||||
|
||||
## Critical Issues Found and Fixed
|
||||
|
||||
### 1. ✅ FIXED: Header Merging Bug in API Client
|
||||
**Severity:** HIGH
|
||||
**Location:** `web/static/js/api.js` - `request()` method
|
||||
|
||||
**Issue:**
|
||||
The header merging was incorrect. When passing custom headers in options, they were being overridden by the default headers due to incorrect spread operator order:
|
||||
|
||||
```javascript
|
||||
// BEFORE (BUGGY):
|
||||
const config = {
|
||||
headers: { ...this.headers },
|
||||
...options // This overwrites headers!
|
||||
};
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
```javascript
|
||||
// AFTER (FIXED):
|
||||
const config = {
|
||||
...options,
|
||||
headers: {
|
||||
...this.headers,
|
||||
...(options.headers || {}) // Custom headers take precedence
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
**Impact:**
|
||||
- The `login()` method was trying to remove the Authorization header for unauthenticated requests, but it was being overridden
|
||||
- Could have caused authentication issues or unexpected behavior
|
||||
|
||||
---
|
||||
|
||||
### 2. ✅ FIXED: Missing API Client Methods
|
||||
**Severity:** HIGH
|
||||
**Location:** `web/static/js/api.js`
|
||||
|
||||
**Issue:** Multiple methods were being called by the UI but didn't exist in the API client:
|
||||
- `login(username, password)` - ❌ Missing
|
||||
- `logout()` - ❌ Missing
|
||||
- `getCurrentUser()` - ❌ Missing
|
||||
- `getUsers()` - ❌ Missing
|
||||
- `createUser(userData)` - ❌ Missing
|
||||
- `updateUser(userId, userData)` - ❌ Missing
|
||||
- `deleteUser(userId)` - ❌ Missing
|
||||
- `getMemberBounces(memberId)` - ❌ Missing
|
||||
- `resetBounceStatus(memberId)` - ❌ Missing
|
||||
|
||||
**Fix:** Added all missing methods with proper implementation
|
||||
|
||||
---
|
||||
|
||||
### 3. ✅ FIXED: Incorrect API Base URL for Production
|
||||
**Severity:** HIGH
|
||||
**Location:** `web/static/js/api.js` - `getBaseURL()` method
|
||||
|
||||
**Issue:**
|
||||
When running behind a reverse proxy (Caddy), the API client was trying to connect to `https://lists.sasalliance.org:8000` which:
|
||||
- Port 8000 is not exposed through Caddy
|
||||
- Would cause CORS issues
|
||||
- Would fail all API requests
|
||||
|
||||
**Fix:**
|
||||
```javascript
|
||||
// Production detection
|
||||
if (hostname === 'localhost' || hostname === '127.0.0.1') {
|
||||
return `${protocol}//${hostname}:8000`; // Development
|
||||
}
|
||||
// Production - use reverse proxy path
|
||||
return `${protocol}//${hostname}/api`;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Potential Issues (Not Critical, But Worth Noting)
|
||||
|
||||
### 4. ⚠️ CORS Configuration
|
||||
**Severity:** MEDIUM
|
||||
**Location:** `api/main.py`
|
||||
|
||||
**Current Configuration:**
|
||||
```python
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"], # Allows all origins
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
```
|
||||
|
||||
**Issue:**
|
||||
- `allow_origins=["*"]` with `allow_credentials=True` is not allowed by browsers
|
||||
- Currently working because Caddy reverse proxy makes same-origin requests
|
||||
- If you ever need direct API access from different origins, this will fail
|
||||
|
||||
**Recommendation:**
|
||||
```python
|
||||
# For production
|
||||
allow_origins=[
|
||||
"https://lists.sasalliance.org",
|
||||
"http://localhost:3000", # For development
|
||||
],
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. ✅ GOOD: Authentication Token Storage
|
||||
**Severity:** INFO
|
||||
**Location:** `web/static/js/app.js`
|
||||
|
||||
**Current Implementation:**
|
||||
```javascript
|
||||
localStorage.setItem('authToken', response.access_token);
|
||||
```
|
||||
|
||||
**Assessment:**
|
||||
- ✅ Proper use of localStorage for JWT tokens
|
||||
- ✅ Token is cleared on logout
|
||||
- ✅ Token is validated on page load
|
||||
- ✅ Expired tokens trigger automatic logout
|
||||
|
||||
**Note:** localStorage is appropriate for JWT tokens. HttpOnly cookies would be more secure but require different architecture.
|
||||
|
||||
---
|
||||
|
||||
### 6. ✅ GOOD: Error Handling
|
||||
**Severity:** INFO
|
||||
**Location:** `web/static/js/ui.js` - `handleError()` method
|
||||
|
||||
**Assessment:**
|
||||
- ✅ Proper handling of 401 (auth) errors with automatic logout
|
||||
- ✅ Proper handling of 403 (forbidden) errors
|
||||
- ✅ Proper handling of 404 (not found) errors
|
||||
- ✅ Proper handling of 500 (server) errors
|
||||
- ✅ User-friendly error messages
|
||||
|
||||
---
|
||||
|
||||
### 7. ✅ GOOD: Role-Based Access Control
|
||||
**Severity:** INFO
|
||||
**Location:** Multiple files
|
||||
|
||||
**Assessment:**
|
||||
- ✅ Backend enforces RBAC with decorators (`require_admin()`, `require_write_access()`, etc.)
|
||||
- ✅ Frontend checks user roles before showing UI elements
|
||||
- ✅ Three roles properly implemented: administrator, operator, read-only
|
||||
- ✅ UI elements are hidden/shown based on permissions
|
||||
|
||||
---
|
||||
|
||||
### 8. ⚠️ Token Expiration Handling
|
||||
**Severity:** LOW
|
||||
**Location:** `api/main.py` and `web/static/js/app.js`
|
||||
|
||||
**Current Configuration:**
|
||||
- JWT tokens expire in 30 minutes
|
||||
- Session records expire in 24 hours (but not enforced in JWT)
|
||||
|
||||
**Potential Issue:**
|
||||
- Users will be logged out after 30 minutes without warning
|
||||
- No token refresh mechanism
|
||||
|
||||
**Recommendation (Future Enhancement):**
|
||||
Consider implementing:
|
||||
- Token refresh endpoint
|
||||
- Warning before token expiration
|
||||
- Silent token refresh in background
|
||||
|
||||
---
|
||||
|
||||
### 9. ✅ GOOD: SQL Injection Prevention
|
||||
**Severity:** INFO
|
||||
**Location:** `api/main.py`
|
||||
|
||||
**Assessment:**
|
||||
- ✅ All SQL queries use parameterized statements with `%s` placeholders
|
||||
- ✅ No string concatenation in SQL queries
|
||||
- ✅ MySQL connector properly escapes parameters
|
||||
|
||||
---
|
||||
|
||||
### 10. ✅ GOOD: Password Security
|
||||
**Severity:** INFO
|
||||
**Location:** `api/main.py`
|
||||
|
||||
**Assessment:**
|
||||
- ✅ Uses bcrypt for password hashing (`passlib.context.CryptContext`)
|
||||
- ✅ Passwords are never stored in plain text
|
||||
- ✅ Passwords are never logged or exposed in API responses
|
||||
|
||||
---
|
||||
|
||||
## Testing Recommendations
|
||||
|
||||
1. **Hard refresh browser cache** after updates (Ctrl+Shift+R)
|
||||
2. **Test authentication flow:**
|
||||
- Login with valid credentials
|
||||
- Login with invalid credentials
|
||||
- Token expiration after 30 minutes
|
||||
- Logout functionality
|
||||
3. **Test role-based access:**
|
||||
- Administrator can see Users tab
|
||||
- Operator can modify lists/members but not users
|
||||
- Read-only can view but not modify
|
||||
4. **Test bounce handling:**
|
||||
- View bounce history
|
||||
- Reset bounce status
|
||||
5. **Test bulk import:**
|
||||
- CSV upload
|
||||
- Subscription assignment
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
✅ **3 Critical Bugs Fixed:**
|
||||
1. Header merging bug in request method
|
||||
2. Missing API client methods
|
||||
3. Incorrect production API URL
|
||||
|
||||
✅ **All Core Functionality Working:**
|
||||
- Authentication and authorization
|
||||
- CRUD operations for lists, members, and users
|
||||
- Subscription management
|
||||
- Bounce tracking
|
||||
- Bulk import
|
||||
|
||||
⚠️ **Minor Improvements Suggested:**
|
||||
1. Tighten CORS policy for production
|
||||
2. Consider token refresh mechanism
|
||||
3. Add user session timeout warnings
|
||||
|
||||
**Overall Assessment:** The codebase is now production-ready with proper security, error handling, and functionality. The critical bugs have been fixed and deployed.
|
||||
243
EMAIL_BOUNCE_HANDLING_SETUP.md
Normal file
243
EMAIL_BOUNCE_HANDLING_SETUP.md
Normal file
@@ -0,0 +1,243 @@
|
||||
# Email-Based Bounce Handling Setup
|
||||
|
||||
This document explains the email-based bounce handling system implemented as an alternative to SNS webhooks for environments without SES production access.
|
||||
|
||||
## Overview
|
||||
|
||||
The system processes email bounces directly within the Postfix container by:
|
||||
1. Rewriting return paths to direct bounces to a processing address
|
||||
2. Processing bounce emails via Python script
|
||||
3. Updating member bounce statistics in MySQL database
|
||||
4. Automatically disabling members with excessive bounces
|
||||
2. Setting up an alias that pipes bounce emails to a Python processing script
|
||||
3. The script parses bounce emails, extracts bounced addresses, and updates the database
|
||||
4. Members with hard bounces are automatically deactivated
|
||||
5. Bounce history is tracked and displayed in the UI (same as SNS method)
|
||||
|
||||
## Advantages
|
||||
|
||||
- **Works with SES Sandbox**: No production SES access required
|
||||
- **No External Dependencies**: Doesn't require SNS, webhooks, or HTTPS domains
|
||||
- **Self-Contained**: All processing happens within the existing containers
|
||||
- **Real-time Processing**: Bounces are processed as soon as emails arrive
|
||||
- **Compatible**: Uses the same database schema and UI as SNS bounce handling
|
||||
|
||||
## Configuration
|
||||
|
||||
### 1. Enable Email Bounce Processing
|
||||
|
||||
In your `.env` file:
|
||||
|
||||
```bash
|
||||
# Enable email-based bounce processing
|
||||
ENABLE_EMAIL_BOUNCE_PROCESSING=true
|
||||
|
||||
# This will automatically enable bounce handling features
|
||||
ENABLE_BOUNCE_HANDLING=true # Automatically set to true when email processing is enabled
|
||||
```
|
||||
|
||||
### 2. Restart the System
|
||||
|
||||
```bash
|
||||
sudo docker-compose down
|
||||
sudo docker-compose up --build -d
|
||||
```
|
||||
|
||||
### 3. Verify Configuration
|
||||
|
||||
Check that email bounce processing is enabled:
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:8000/config | jq .
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```json
|
||||
{
|
||||
"bounce_handling_enabled": true,
|
||||
"sns_webhooks_enabled": false,
|
||||
"email_bounce_processing_enabled": true
|
||||
}
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
### Postfix Configuration
|
||||
|
||||
The system configures Postfix with:
|
||||
|
||||
- `bounce_notice_recipient = bounces@lists.sasalliance.org`
|
||||
- `2bounce_notice_recipient = bounces@lists.sasalliance.org`
|
||||
- `error_notice_recipient = bounces@lists.sasalliance.org`
|
||||
|
||||
### Aliases Configuration
|
||||
|
||||
The `bounces` address is configured to pipe emails to the processing script:
|
||||
|
||||
```
|
||||
bounces: "|/usr/local/bin/process-bounce.py"
|
||||
```
|
||||
|
||||
### Bounce Processing Script
|
||||
|
||||
The Python script (`/usr/local/bin/process-bounce.py`):
|
||||
|
||||
1. **Reads bounce emails** from stdin (via pipe)
|
||||
2. **Parses email content** using multiple regex patterns to extract bounced addresses
|
||||
3. **Analyzes bounce type** based on SMTP error codes:
|
||||
- 5xx codes = Permanent bounces
|
||||
- 4xx codes = Transient bounces
|
||||
- Unknown = Undetermined
|
||||
4. **Updates database** using the same schema as SNS bounce handling:
|
||||
- Logs bounce in `bounce_logs` table
|
||||
- Updates member `bounce_count`, `bounce_status`, and `last_bounce_at`
|
||||
- Deactivates members with permanent bounces
|
||||
- Marks members with soft bounce status after 3 transient bounces
|
||||
|
||||
## Testing
|
||||
|
||||
### Test the Processing Script
|
||||
|
||||
You can test the bounce processing script in test mode:
|
||||
|
||||
```bash
|
||||
# Test with sample bounce email
|
||||
sudo docker-compose exec postfix /usr/local/bin/process-bounce.py --test
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
2025-10-14 15:49:16,041 - bounce-processor - INFO - Starting bounce processing
|
||||
2025-10-14 15:49:16,041 - bounce-processor - INFO - Running in test mode with sample bounce email
|
||||
2025-10-14 15:49:16,050 - bounce-processor - INFO - Extracted addresses: ['testuser@example.com']
|
||||
2025-10-14 15:49:16,050 - bounce-processor - INFO - Test mode - would process 1 bounce(s):
|
||||
2025-10-14 15:49:16,050 - bounce-processor - INFO - {'email': 'testuser@example.com', 'bounce_type': 'Permanent', 'bounce_subtype': 'General', 'diagnostic_code': '', 'timestamp': '2025-10-14 15:49:16'}
|
||||
```
|
||||
|
||||
### Test with Real Bounce Email
|
||||
|
||||
To test with a real bounce:
|
||||
|
||||
1. Send an email to a non-existent address via your mailing list
|
||||
2. Wait for the bounce to be processed
|
||||
3. Check the database for bounce logs:
|
||||
|
||||
```bash
|
||||
# Check bounce logs
|
||||
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;"
|
||||
|
||||
# Check member bounce status
|
||||
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT member_id, email, bounce_count, bounce_status, last_bounce_at FROM members WHERE bounce_count > 0;"
|
||||
```
|
||||
|
||||
### View in Web Interface
|
||||
|
||||
1. Open http://localhost:3000
|
||||
2. Navigate to the Members tab
|
||||
3. Look for bounce badges and bounce counts next to member names
|
||||
4. Click the "Bounces" button next to a member to view bounce history
|
||||
|
||||
## Supported Bounce Email Formats
|
||||
|
||||
The processing script recognizes these bounce patterns:
|
||||
|
||||
### SMTP Error Codes
|
||||
- 550, 551, 553, 552, 554 (permanent failures)
|
||||
- 450, 451, 452 (temporary failures)
|
||||
|
||||
### Delivery Status Notification (DSN)
|
||||
- `Final-Recipient:` headers
|
||||
- `Original-Recipient:` headers
|
||||
|
||||
### Common Bounce Messages
|
||||
- "user unknown"
|
||||
- "does not exist"
|
||||
- "not found"
|
||||
- "mailbox unavailable"
|
||||
- "recipient rejected"
|
||||
|
||||
## Monitoring
|
||||
|
||||
### View Processing Logs
|
||||
|
||||
```bash
|
||||
# View bounce processing logs
|
||||
sudo docker-compose logs postfix | grep bounce-processor
|
||||
|
||||
# Follow logs in real-time
|
||||
sudo docker-compose logs -f postfix | grep bounce-processor
|
||||
```
|
||||
|
||||
### Check Aliases Configuration
|
||||
|
||||
```bash
|
||||
# Verify aliases are configured correctly
|
||||
sudo docker-compose exec postfix cat /etc/aliases
|
||||
|
||||
# Check alias database
|
||||
sudo docker-compose exec postfix postmap -q bounces /etc/aliases
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Bounces Not Being Processed
|
||||
|
||||
1. **Check aliases configuration:**
|
||||
```bash
|
||||
sudo docker-compose exec postfix cat /etc/aliases
|
||||
```
|
||||
|
||||
2. **Verify script permissions:**
|
||||
```bash
|
||||
sudo docker-compose exec postfix ls -la /usr/local/bin/process-bounce.py
|
||||
```
|
||||
|
||||
3. **Test script manually:**
|
||||
```bash
|
||||
sudo docker-compose exec postfix /usr/local/bin/process-bounce.py --test
|
||||
```
|
||||
|
||||
4. **Check Postfix logs:**
|
||||
```bash
|
||||
sudo docker-compose logs postfix | grep -i bounce
|
||||
```
|
||||
|
||||
### Database Connection Issues
|
||||
|
||||
1. **Check environment variables:**
|
||||
```bash
|
||||
sudo docker-compose exec postfix env | grep MYSQL
|
||||
```
|
||||
|
||||
2. **Test database connection:**
|
||||
```bash
|
||||
sudo docker-compose exec postfix python3 -c "import pymysql; print('PyMySQL available')"
|
||||
```
|
||||
|
||||
### Script Errors
|
||||
|
||||
View detailed error logs:
|
||||
```bash
|
||||
sudo docker-compose logs postfix | grep -A 10 -B 10 "bounce-processor.*ERROR"
|
||||
```
|
||||
|
||||
## Comparison with SNS Webhooks
|
||||
|
||||
| Feature | Email-Based | SNS Webhooks |
|
||||
|---------|-------------|--------------|
|
||||
| **SES Requirement** | Sandbox OK | Production access required |
|
||||
| **External Dependencies** | None | SNS, HTTPS domain |
|
||||
| **Processing Speed** | Real-time | Real-time |
|
||||
| **Setup Complexity** | Low | High |
|
||||
| **Reliability** | High | High |
|
||||
| **Bounce Detection** | Regex-based | AWS-provided |
|
||||
| **Cost** | Free | SNS charges apply |
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Monitor bounce processing** to ensure it's working correctly
|
||||
2. **Review bounce patterns** in the logs to improve detection if needed
|
||||
3. **Set up bounce notification alerts** (optional)
|
||||
4. **Consider upgrading to SNS webhooks** when SES production access is available
|
||||
|
||||
Email-based bounce handling provides a robust alternative that works immediately with any SES configuration while providing the same bounce management features as the SNS webhook method.
|
||||
212
EMAIL_BOUNCE_HANDLING_SETUP_COMPREHENSIVE.md
Normal file
212
EMAIL_BOUNCE_HANDLING_SETUP_COMPREHENSIVE.md
Normal file
@@ -0,0 +1,212 @@
|
||||
# Email-Based Bounce Handling Setup (Comprehensive)
|
||||
|
||||
This document explains the complete email-based bounce handling system implemented as an alternative to SNS webhooks for environments without SES production access.
|
||||
|
||||
## Overview
|
||||
|
||||
The system processes email bounces directly within the Postfix container by:
|
||||
1. Rewriting return paths to direct bounces to a processing address
|
||||
2. Processing bounce emails via Python script
|
||||
3. Updating member bounce statistics in MySQL database
|
||||
4. Automatically disabling members with excessive bounces
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables (in .env)
|
||||
|
||||
```bash
|
||||
# Bounce handling feature flags
|
||||
ENABLE_BOUNCE_HANDLING=true # Master switch for bounce functionality
|
||||
ENABLE_EMAIL_BOUNCE_PROCESSING=true # Enable email-based processing
|
||||
ENABLE_SNS_WEBHOOKS=false # Disable SNS webhooks (optional)
|
||||
|
||||
# Database settings (required for bounce processing)
|
||||
MYSQL_ROOT_PASSWORD=your_root_password
|
||||
MYSQL_DATABASE=maillist
|
||||
MYSQL_USER=maillist
|
||||
MYSQL_PASSWORD=your_maillist_password
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
#### 1. Return Path Rewriting (`postfix/smtp_generic`)
|
||||
Routes all bounces to the processing address:
|
||||
```
|
||||
@lists.sasalliance.org bounces@lists.sasalliance.org
|
||||
@sasalliance.org bounces@lists.sasalliance.org
|
||||
```
|
||||
|
||||
#### 2. Bounce Processing Script (`postfix/process-bounce.py`)
|
||||
Python script that:
|
||||
- Parses bounce emails for recipient addresses and bounce types
|
||||
- Updates bounce counts in MySQL database
|
||||
- Automatically disables members after 5 hard bounces
|
||||
- Logs all bounce events
|
||||
|
||||
#### 3. Postfix Integration (`postfix/main.cf.template`)
|
||||
```
|
||||
# Return path rewriting for outbound mail
|
||||
smtp_generic_maps = hash:/etc/postfix/smtp_generic
|
||||
|
||||
# Bounce processing
|
||||
bounce_notice_recipient = bounces@lists.sasalliance.org
|
||||
```
|
||||
|
||||
#### 4. Email Aliases (`postfix/entrypoint.sh`)
|
||||
```
|
||||
# Route bounces to processing script
|
||||
bounces: "|/usr/local/bin/python3 /etc/postfix/process-bounce.py"
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
### Outbound Email Flow
|
||||
1. User sends email to `community@lists.sasalliance.org`
|
||||
2. Postfix expands to member list via MySQL
|
||||
3. Email sent via SES with return path rewritten to `bounces@lists.sasalliance.org`
|
||||
4. If delivery fails, bounce goes to bounce processing address
|
||||
|
||||
### Bounce Processing Flow
|
||||
1. Bounce email arrives at `bounces@lists.sasalliance.org`
|
||||
2. Postfix pipes email to `process-bounce.py` script
|
||||
3. Script parses bounce for recipient and bounce type
|
||||
4. Database updated with bounce information
|
||||
5. Member automatically disabled if hard bounce threshold reached
|
||||
|
||||
### Bounce Types Detected
|
||||
- **Hard Bounces**: Permanent failures (5.x.x SMTP codes)
|
||||
- Invalid email addresses
|
||||
- Domain doesn't exist
|
||||
- Mailbox doesn't exist
|
||||
- **Soft Bounces**: Temporary failures (4.x.x SMTP codes)
|
||||
- Mailbox full
|
||||
- Temporary server issues
|
||||
- Rate limiting
|
||||
|
||||
## Database Schema
|
||||
|
||||
### Bounce Logs Table
|
||||
```sql
|
||||
CREATE TABLE bounce_logs (
|
||||
id INT PRIMARY KEY AUTO_INCREMENT,
|
||||
email VARCHAR(255) NOT NULL,
|
||||
bounce_type ENUM('hard', 'soft', 'complaint') NOT NULL,
|
||||
bounce_reason TEXT,
|
||||
bounced_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
raw_message TEXT,
|
||||
INDEX idx_email (email),
|
||||
INDEX idx_bounced_at (bounced_at)
|
||||
);
|
||||
```
|
||||
|
||||
### Members Table (bounce tracking fields)
|
||||
```sql
|
||||
ALTER TABLE members ADD COLUMN bounce_count INT DEFAULT 0;
|
||||
ALTER TABLE members ADD COLUMN last_bounce_at TIMESTAMP NULL;
|
||||
ALTER TABLE members ADD COLUMN active BOOLEAN DEFAULT true;
|
||||
```
|
||||
|
||||
## Testing Bounce Handling
|
||||
|
||||
### 1. Test Return Path Configuration
|
||||
```bash
|
||||
# Check that return path rewriting is working
|
||||
sudo docker-compose exec postfix postconf smtp_generic_maps
|
||||
sudo docker-compose exec postfix postmap -q "test@lists.sasalliance.org" hash:/etc/postfix/smtp_generic
|
||||
```
|
||||
|
||||
### 2. Simulate Bounce Email
|
||||
```bash
|
||||
# Send test bounce to processing script
|
||||
echo "Subject: Delivery Status Notification (Failure)
|
||||
From: MAILER-DAEMON@ses.amazonaws.com
|
||||
To: bounces@lists.sasalliance.org
|
||||
|
||||
The following message could not be delivered:
|
||||
Recipient: test@example.com
|
||||
Reason: 550 5.1.1 User unknown" | docker-compose exec -T postfix mail -s "Test Bounce" bounces@lists.sasalliance.org
|
||||
```
|
||||
|
||||
### 3. Check Bounce Processing
|
||||
```bash
|
||||
# View bounce logs
|
||||
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY bounced_at DESC LIMIT 5;"
|
||||
|
||||
# Check member bounce counts
|
||||
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT email, bounce_count, last_bounce_at, active FROM members WHERE bounce_count > 0;"
|
||||
```
|
||||
|
||||
## Monitoring and Maintenance
|
||||
|
||||
### View Processing Logs
|
||||
```bash
|
||||
# Monitor bounce processing
|
||||
sudo docker-compose logs -f postfix | grep -E "(bounce|process-bounce)"
|
||||
|
||||
# Check API bounce handling status
|
||||
curl -H "Authorization: Bearer $API_TOKEN" http://localhost:8000/config
|
||||
```
|
||||
|
||||
### Reset Member Bounce Count
|
||||
```bash
|
||||
# Via API
|
||||
curl -X POST http://localhost:8000/members/{member_id}/reset-bounces \
|
||||
-H "Authorization: Bearer $API_TOKEN"
|
||||
|
||||
# Via Database
|
||||
sudo docker-compose exec mysql mysql -u maillist -pmaillist maillist -e "UPDATE members SET bounce_count=0, active=true WHERE email='user@example.com';"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### Bounces Not Being Processed
|
||||
1. Check that `bounces` alias exists: `sudo docker-compose exec postfix cat /etc/aliases | grep bounces`
|
||||
2. Verify Python script permissions: `sudo docker-compose exec postfix ls -la /etc/postfix/process-bounce.py`
|
||||
3. Test script manually: `sudo docker-compose exec postfix python3 /etc/postfix/process-bounce.py --test`
|
||||
|
||||
#### Return Path Not Rewritten
|
||||
1. Check smtp_generic configuration: `sudo docker-compose exec postfix postconf smtp_generic_maps`
|
||||
2. Verify map file exists: `sudo docker-compose exec postfix ls -la /etc/postfix/smtp_generic*`
|
||||
3. Test mapping: `sudo docker-compose exec postfix postmap -q "test@lists.sasalliance.org" hash:/etc/postfix/smtp_generic`
|
||||
|
||||
#### Database Connection Issues
|
||||
1. Check PyMySQL installation: `sudo docker-compose exec postfix python3 -c "import pymysql; print('OK')"`
|
||||
2. Test database connection: `sudo docker-compose exec postfix python3 -c "import pymysql; pymysql.connect(host='mysql', user='maillist', password='your_password', database='maillist')"`
|
||||
3. Verify network connectivity: `sudo docker-compose exec postfix ping mysql`
|
||||
|
||||
### Log Analysis
|
||||
```bash
|
||||
# Postfix logs
|
||||
sudo docker-compose logs postfix | grep -E "(bounce|MAILER-DAEMON|process-bounce)"
|
||||
|
||||
# MySQL connection logs
|
||||
sudo docker-compose logs postfix | grep -E "(pymysql|mysql)"
|
||||
|
||||
# SES relay logs
|
||||
sudo docker-compose logs postfix | grep -E "(relay|sent|deferred)"
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Bounce processing script runs with limited privileges
|
||||
- Database credentials secured in environment variables
|
||||
- Bounce emails contain sensitive delivery information - logs are rotated
|
||||
- Return path rewriting prevents bounce loops
|
||||
- Processing script validates email format before database updates
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Bounce processing is asynchronous (doesn't block email delivery)
|
||||
- Database queries are indexed for bounce lookups
|
||||
- Bounce logs should be periodically archived for large volumes
|
||||
- SMTP generic maps are cached by Postfix for performance
|
||||
|
||||
## Advantages over SNS Webhooks
|
||||
|
||||
- **Works with SES Sandbox**: No production SES access required
|
||||
- **No External Dependencies**: Doesn't require SNS, webhooks, or HTTPS domains
|
||||
- **Self-Contained**: All processing happens within existing containers
|
||||
- **Real-time Processing**: Bounces processed as emails arrive
|
||||
- **Compatible**: Uses same database schema and UI as SNS method
|
||||
375
README.md
375
README.md
@@ -1,18 +1,23 @@
|
||||
# Mail List Manager
|
||||
|
||||
A containerized mailing list management system built around Postfix as an SMTP relay through Amazon SES.
|
||||
A complete containerized mailing list management system with web interface, REST API, MySQL database, and Postfix mail server configured as an SMTP relay through Amazon SES.
|
||||
|
||||
## Architecture
|
||||
|
||||
**Current (Phase 1):** Static configuration with environment-based credentials
|
||||
- Postfix container configured as SES relay
|
||||
- Static virtual aliases for mailing list distribution
|
||||
- Environment variable configuration for security
|
||||
**Multi-Service Architecture:**
|
||||
- **Web Frontend** - Modern responsive interface for managing lists and members (Port 3000)
|
||||
- **REST API** - FastAPI backend with token authentication (Port 8000)
|
||||
- **MySQL Database** - Stores lists, members, and subscriptions with real-time integration
|
||||
- **Postfix Mail Server** - SMTP server with native MySQL integration for dynamic list expansion (Port 25)
|
||||
- **Amazon SES** - Outbound mail relay for reliable delivery
|
||||
|
||||
**Planned (Phase 2+):** Web interface with SQL backend
|
||||
- Web frontend for list management (view/add/remove members)
|
||||
- SQL database for member storage
|
||||
- Dynamic Postfix configuration generation
|
||||
**Key Features:**
|
||||
- Real-time list updates (no Postfix reload needed)
|
||||
- Native Postfix MySQL queries for instant list expansion
|
||||
- Token-based API authentication
|
||||
- Sender whitelisting for authorized board members
|
||||
- Private Docker network for security
|
||||
- Complete web-based management interface
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -21,72 +26,239 @@ A containerized mailing list management system built around Postfix as an SMTP r
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
2. Edit `.env` with your SES credentials and configuration:
|
||||
2. Edit `.env` with your credentials:
|
||||
```bash
|
||||
# Required: Your SES credentials
|
||||
# SES Credentials (Required)
|
||||
SES_USER=your_ses_access_key
|
||||
SES_PASS=your_ses_secret_key
|
||||
|
||||
# MySQL Credentials (Required)
|
||||
MYSQL_ROOT_PASSWORD=secure_root_password
|
||||
MYSQL_PASSWORD=secure_maillist_password
|
||||
|
||||
# API Authentication (Required)
|
||||
API_TOKEN=your_secure_api_token
|
||||
|
||||
# Optional: SMTP configuration (defaults to EU West 2)
|
||||
SMTP_HOST=email-smtp.eu-west-2.amazonaws.com
|
||||
SMTP_PORT=587
|
||||
```
|
||||
|
||||
3. Build and start the mail server:
|
||||
3. Build and start all services:
|
||||
```bash
|
||||
docker-compose up --build
|
||||
sudo docker-compose up --build
|
||||
```
|
||||
|
||||
4. Test mail delivery:
|
||||
```bash
|
||||
# From inside container
|
||||
docker-compose exec postfix bash
|
||||
echo "Test message" | mail -s "Subject" community@lists.sasalliance.org
|
||||
|
||||
# Check logs
|
||||
docker-compose logs -f postfix
|
||||
4. Access the web interface:
|
||||
```
|
||||
http://localhost:3000
|
||||
```
|
||||
|
||||
Enter your API_TOKEN to authenticate and start managing lists!
|
||||
|
||||
5. Alternative: Use the REST API directly:
|
||||
```bash
|
||||
# Health check
|
||||
curl http://localhost:8000/health
|
||||
|
||||
# Get all lists (requires authentication)
|
||||
curl -H "Authorization: Bearer your_api_token" http://localhost:8000/lists
|
||||
```
|
||||
|
||||
## Services
|
||||
|
||||
### Web Interface (Port 3000)
|
||||
- Modern, responsive interface for managing mailing lists and members
|
||||
- Token-based authentication
|
||||
- Real-time updates and feedback
|
||||
- See `web/README.md` for details
|
||||
|
||||
### REST API (Port 8000)
|
||||
- Complete CRUD operations for lists, members, and subscriptions
|
||||
- Bearer token authentication
|
||||
- Interactive documentation at http://localhost:8000/docs
|
||||
- See `api/README.md` for API reference
|
||||
|
||||
### MySQL Database (Internal Only)
|
||||
- Stores all mailing list and member data
|
||||
- Automatically initialized with schema
|
||||
- Native Postfix integration for real-time queries
|
||||
- See `database/README.md` for schema and management
|
||||
|
||||
### Postfix Mail Server (Port 25)
|
||||
- Accepts mail for lists.sasalliance.org
|
||||
- Queries MySQL for list expansion
|
||||
- Relays through Amazon SES
|
||||
- Sender whitelist: @sasalliance.org domain
|
||||
- Changes take effect immediately (no restart needed)
|
||||
|
||||
## Configuration
|
||||
|
||||
### Adding Mailing Lists (Current)
|
||||
### Managing Lists and Members
|
||||
|
||||
Edit `postfix/virtual_aliases.cf`:
|
||||
```
|
||||
newlist@lists.sasalliance.org recipient1@domain.com, recipient2@domain.com
|
||||
```
|
||||
**Via Web Interface (Recommended):**
|
||||
1. Open http://localhost:3000
|
||||
2. Enter your API token
|
||||
3. Use the intuitive interface to:
|
||||
- Create/edit/delete mailing lists
|
||||
- Add/edit/remove members
|
||||
- Manage subscriptions with toggle switches
|
||||
- View member counts and list details
|
||||
|
||||
Then rebuild the container:
|
||||
**Via REST API:**
|
||||
```bash
|
||||
docker-compose up --build
|
||||
# Create a new list
|
||||
curl -X POST http://localhost:8000/lists \
|
||||
-H "Authorization: Bearer your_token" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"list_name": "Developers",
|
||||
"list_email": "dev@lists.sasalliance.org",
|
||||
"description": "Developer discussions",
|
||||
"active": true
|
||||
}'
|
||||
|
||||
# Add a new member
|
||||
curl -X POST http://localhost:8000/members \
|
||||
-H "Authorization: Bearer your_token" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "John Doe",
|
||||
"email": "john@example.com",
|
||||
"active": true
|
||||
}'
|
||||
|
||||
# Subscribe member to list
|
||||
curl -X POST http://localhost:8000/subscriptions \
|
||||
-H "Authorization: Bearer your_token" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"list_email": "dev@lists.sasalliance.org",
|
||||
"member_email": "john@example.com"
|
||||
}'
|
||||
```
|
||||
|
||||
### Domain Configuration
|
||||
**Via Direct Database Access:**
|
||||
```bash
|
||||
# Connect to MySQL
|
||||
docker-compose exec mysql mysql -u maillist -p maillist
|
||||
|
||||
The system is configured for:
|
||||
- **Hostname**: `lists.sasalliance.org` (mailing lists)
|
||||
# Run SQL queries (see database/README.md for examples)
|
||||
```
|
||||
|
||||
### Mail Server Configuration
|
||||
|
||||
### Mail Server Configuration
|
||||
|
||||
The Postfix mail server is configured for:
|
||||
- **Hostname**: `lists.sasalliance.org` (accepts mail for this domain)
|
||||
- **Origin Domain**: `sasalliance.org`
|
||||
- **SES Region**: EU West 2 (configurable via `SMTP_HOST`)
|
||||
- **Sender Whitelist**: @sasalliance.org (only authorized board members can send to lists)
|
||||
- **Dynamic Lists**: Postfix queries MySQL in real-time for list members
|
||||
|
||||
### Testing Mail Delivery
|
||||
|
||||
```bash
|
||||
# From inside Postfix container
|
||||
docker-compose exec postfix bash
|
||||
echo "Test message" | mail -s "Test Subject" community@lists.sasalliance.org
|
||||
|
||||
# Check delivery logs
|
||||
docker-compose logs -f postfix | grep -E "(sent|bounced|deferred)"
|
||||
|
||||
# Verify MySQL query works
|
||||
docker-compose exec postfix postmap -q "community@lists.sasalliance.org" \
|
||||
mysql:/etc/postfix/mysql_virtual_alias_maps.cf
|
||||
```
|
||||
|
||||
### Bounce Handling (Optional)
|
||||
|
||||
**Email bounce handling is optional and disabled by default.**
|
||||
|
||||
**Three Configuration Options:**
|
||||
|
||||
1. **SNS Webhooks** (Requires SES Production Access):
|
||||
```bash
|
||||
# In .env file
|
||||
ENABLE_SNS_WEBHOOKS=true
|
||||
ENABLE_BOUNCE_HANDLING=true
|
||||
```
|
||||
- Real-time bounce notifications via AWS SNS webhooks
|
||||
- Automatic member deactivation for hard bounces
|
||||
- Bounce history tracking and management in web UI
|
||||
- Requires valid HTTPS domain and SES production access
|
||||
- See `BOUNCE_HANDLING_SETUP.md` for complete setup
|
||||
|
||||
2. **Email-Based Processing** (Works with SES Sandbox):
|
||||
```bash
|
||||
# In .env file
|
||||
ENABLE_EMAIL_BOUNCE_PROCESSING=true
|
||||
ENABLE_BOUNCE_HANDLING=true # Automatically enabled
|
||||
```
|
||||
- Automatic bounce processing via email parsing
|
||||
- Same bounce management features as SNS webhooks
|
||||
- Works with SES sandbox accounts (no production access needed)
|
||||
- Self-contained processing within existing containers
|
||||
- See `EMAIL_BOUNCE_HANDLING_SETUP.md` for complete setup
|
||||
|
||||
3. **Disabled** (Manual Processing Only):
|
||||
```bash
|
||||
# In .env file (or leave these commented out)
|
||||
ENABLE_SNS_WEBHOOKS=false
|
||||
ENABLE_BOUNCE_HANDLING=false
|
||||
ENABLE_EMAIL_BOUNCE_PROCESSING=false
|
||||
```
|
||||
- No automatic bounce processing
|
||||
- Manual bounce management via email notifications
|
||||
- Bounce-related UI elements are hidden
|
||||
|
||||
**When bounce handling is disabled:**
|
||||
- `/webhooks/sns` endpoint is not registered
|
||||
- Bounce history endpoints return empty results
|
||||
- Web UI hides bounce badges and bounce management buttons
|
||||
- No automatic member deactivation occurs
|
||||
|
||||
## Security
|
||||
|
||||
- SES credentials are stored in `.env` (git-ignored)
|
||||
- SASL password files have restricted permissions (600)
|
||||
- TLS encryption enforced for SES relay
|
||||
- Only localhost and configured hostname accepted for local delivery
|
||||
- **Environment Variables**: All credentials stored in `.env` (git-ignored)
|
||||
- **Private Network**: MySQL communicates with Postfix/API on internal Docker network only
|
||||
- **Token Authentication**: API requires Bearer token for all write operations
|
||||
- **Sender Whitelist**: Only @sasalliance.org addresses can send to mailing lists
|
||||
- **TLS Encryption**: Enforced for SES relay connections
|
||||
- **File Permissions**: SASL password files have restricted permissions (600)
|
||||
- **No External MySQL**: Database is not exposed to host (no port mapping)
|
||||
|
||||
## Development
|
||||
|
||||
### Project Structure
|
||||
```
|
||||
├── docker-compose.yaml # Service orchestration
|
||||
├── docker-compose.yaml # Multi-service orchestration
|
||||
├── .env # Environment configuration (not in git)
|
||||
├── postfix/
|
||||
│ ├── Dockerfile # Postfix container build
|
||||
│ ├── entrypoint.sh # Runtime configuration processing
|
||||
│ ├── main.cf.template # Postfix main configuration template
|
||||
│ ├── sasl_passwd.template # SES authentication template
|
||||
│ └── virtual_aliases.cf # Static mailing list definitions
|
||||
├── web/ # Web frontend (Nginx + HTML/CSS/JS)
|
||||
│ ├── index.html
|
||||
│ ├── static/
|
||||
│ │ ├── css/style.css
|
||||
│ │ └── js/
|
||||
│ │ ├── api.js
|
||||
│ │ ├── ui.js
|
||||
│ │ └── app.js
|
||||
│ ├── nginx.conf
|
||||
│ └── Dockerfile
|
||||
├── api/ # REST API (FastAPI)
|
||||
│ ├── main.py
|
||||
│ ├── requirements.txt
|
||||
│ └── Dockerfile
|
||||
├── database/ # MySQL schema and docs
|
||||
│ ├── schema.sql
|
||||
│ └── README.md
|
||||
├── postfix/ # Mail server configuration
|
||||
│ ├── Dockerfile
|
||||
│ ├── entrypoint.sh
|
||||
│ ├── main.cf.template
|
||||
│ ├── mysql_virtual_alias_maps.cf
|
||||
│ ├── sasl_passwd.template
|
||||
│ └── sender_access
|
||||
└── .github/
|
||||
└── copilot-instructions.md # AI agent guidance
|
||||
```
|
||||
@@ -96,28 +268,131 @@ The system is configured for:
|
||||
- `SES_PASS`: AWS SES secret access key
|
||||
- `SMTP_HOST`: SMTP server hostname (default: email-smtp.eu-west-2.amazonaws.com)
|
||||
- `SMTP_PORT`: SMTP server port (default: 587)
|
||||
- `MYSQL_ROOT_PASSWORD`: MySQL root password
|
||||
- `MYSQL_DATABASE`: Database name (default: maillist)
|
||||
- `MYSQL_USER`: MySQL user (default: maillist)
|
||||
- `MYSQL_PASSWORD`: MySQL user password
|
||||
- `API_TOKEN`: Bearer token for API authentication
|
||||
|
||||
### Service Dependencies
|
||||
```
|
||||
mysql (healthcheck) → postfix, api → web
|
||||
```
|
||||
- MySQL must be healthy before Postfix/API start
|
||||
- Web frontend depends on API being available
|
||||
- All services communicate on private Docker network
|
||||
|
||||
### Debugging
|
||||
|
||||
Monitor mail delivery:
|
||||
Monitor services:
|
||||
```bash
|
||||
# View all logs
|
||||
# View all service logs
|
||||
docker-compose logs -f
|
||||
|
||||
# Filter for delivery status
|
||||
# View specific service
|
||||
docker-compose logs -f postfix
|
||||
docker-compose logs -f api
|
||||
docker-compose logs -f web
|
||||
docker-compose logs -f mysql
|
||||
|
||||
# Filter for mail delivery status
|
||||
docker-compose logs postfix | grep -E "(sent|bounced|deferred)"
|
||||
|
||||
# Check Postfix queue
|
||||
docker-compose exec postfix postqueue -p
|
||||
|
||||
# Test MySQL connectivity from Postfix
|
||||
docker-compose exec postfix postmap -q "community@lists.sasalliance.org" \
|
||||
mysql:/etc/postfix/mysql_virtual_alias_maps.cf
|
||||
|
||||
# Check API health
|
||||
curl http://localhost:8000/health
|
||||
|
||||
# Access MySQL directly
|
||||
docker-compose exec mysql mysql -u maillist -p maillist
|
||||
```
|
||||
|
||||
### Development Workflow
|
||||
|
||||
1. **Make changes** to code or configuration
|
||||
2. **Rebuild affected service**:
|
||||
```bash
|
||||
sudo docker-compose up --build -d web # Just web frontend
|
||||
sudo docker-compose up --build -d api # Just API
|
||||
sudo docker-compose up --build -d postfix # Just mail server
|
||||
```
|
||||
3. **Check logs** for errors:
|
||||
```bash
|
||||
docker-compose logs -f [service-name]
|
||||
```
|
||||
4. **Test changes** via web interface or API
|
||||
|
||||
### Common Tasks
|
||||
|
||||
**Reset database completely:**
|
||||
```bash
|
||||
docker-compose down -v # Removes volumes
|
||||
docker-compose up -d # Reinitializes from schema.sql
|
||||
```
|
||||
|
||||
**Update API token:**
|
||||
1. Edit `.env` and change `API_TOKEN`
|
||||
2. Restart API: `docker-compose restart api`
|
||||
3. Use new token in web interface
|
||||
|
||||
**Add authorized sender domain:**
|
||||
1. Edit `postfix/sender_access`
|
||||
2. Add line: `newdomain.com OK`
|
||||
3. Rebuild: `docker-compose up --build -d postfix`
|
||||
|
||||
## How It Works
|
||||
|
||||
1. **User manages lists** via web interface (http://localhost:3000)
|
||||
2. **Web frontend** makes authenticated API calls to FastAPI backend
|
||||
3. **API updates MySQL** database with new lists/members/subscriptions
|
||||
4. **Email arrives** at Postfix for `someone@lists.sasalliance.org`
|
||||
5. **Postfix queries MySQL** in real-time using native MySQL support
|
||||
6. **MySQL returns** comma-separated list of active member emails
|
||||
7. **Postfix expands** the alias and delivers to all members via SES
|
||||
8. **Changes take effect immediately** - no restart or reload needed!
|
||||
|
||||
## Features
|
||||
|
||||
- ✅ **Real-time Updates** - Changes to lists/members take effect immediately
|
||||
- ✅ **Native MySQL Integration** - Postfix queries database directly (no scripts)
|
||||
- ✅ **Web Interface** - Modern, responsive UI for easy management
|
||||
- ✅ **REST API** - Complete programmatic access with token auth
|
||||
- ✅ **Sender Whitelist** - Only authorized domains can send to lists
|
||||
- ✅ **SES Integration** - Reliable email delivery through Amazon SES
|
||||
- ✅ **Bounce Handling** - Automatic tracking and management of email bounces via SNS
|
||||
- ✅ **Secure** - Private Docker network, token auth, environment-based credentials
|
||||
- ✅ **Flexible** - Manage via web, API, or direct database access
|
||||
- ✅ **Scalable** - Database-driven architecture supports many lists and members
|
||||
|
||||
## Documentation
|
||||
|
||||
- **Web Interface**: See `web/README.md` for frontend features and usage
|
||||
- **REST API**: See `api/README.md` for complete API reference
|
||||
- **Database**: See `database/README.md` for schema and SQL examples
|
||||
- **Bounce Handling**: See `BOUNCE_HANDLING_SETUP.md` for SNS configuration
|
||||
- **AI Agents**: See `.github/copilot-instructions.md` for development guidance
|
||||
|
||||
## Roadmap
|
||||
|
||||
- [ ] Web frontend for mailing list management
|
||||
- [ ] SQL database backend for member storage
|
||||
- [ ] Dynamic configuration generation from database
|
||||
- [ ] Multi-service Docker Compose architecture
|
||||
- [ ] Migration tools for static → dynamic configuration
|
||||
- [x] Web frontend for mailing list management
|
||||
- [x] SQL database backend for member storage
|
||||
- [x] Dynamic configuration with native Postfix MySQL
|
||||
- [x] Multi-service Docker Compose architecture
|
||||
- [x] REST API with authentication
|
||||
- [x] Sender whitelist for authorized domains
|
||||
- [x] Bounce handling with SES SNS integration
|
||||
- [ ] Email verification workflow for new members
|
||||
- [ ] Subscription confirmation (double opt-in)
|
||||
- [ ] List archive functionality
|
||||
- [ ] Unsubscribe links in emails
|
||||
- [ ] Rate limiting and anti-spam measures
|
||||
- [ ] HTTPS support for web interface
|
||||
- [ ] Admin roles and permissions
|
||||
|
||||
## License
|
||||
|
||||
|
||||
398
SES_BOUNCE_TESTING_GUIDE.md
Normal file
398
SES_BOUNCE_TESTING_GUIDE.md
Normal file
@@ -0,0 +1,398 @@
|
||||
# Testing Bounces from SES Sandbox
|
||||
|
||||
AWS SES provides **built-in bounce simulator addresses** that work even in sandbox mode. This guide shows you how to use them to test your bounce handling.
|
||||
|
||||
## Quick Answer
|
||||
|
||||
Send email to these special AWS addresses to simulate different bounce types:
|
||||
|
||||
### Hard Bounce (Permanent - Invalid Address)
|
||||
```bash
|
||||
bounce@simulator.amazonses.com
|
||||
```
|
||||
|
||||
### Soft Bounce (Temporary - Mailbox Full)
|
||||
```bash
|
||||
ooto@simulator.amazonses.com
|
||||
```
|
||||
|
||||
### Complaint (Spam Report)
|
||||
```bash
|
||||
complaint@simulator.amazonses.com
|
||||
```
|
||||
|
||||
### Successful Delivery (No Bounce)
|
||||
```bash
|
||||
success@simulator.amazonses.com
|
||||
```
|
||||
|
||||
## Step-by-Step Testing Guide
|
||||
|
||||
### Option 1: Using Your Mailing Lists (Recommended)
|
||||
|
||||
This tests the complete flow: Postfix → SES → SNS → API
|
||||
|
||||
1. **Add the simulator address as a member:**
|
||||
```bash
|
||||
# Using the API
|
||||
curl -X POST http://localhost:8000/members \
|
||||
-H "Authorization: Bearer $API_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "Bounce Test",
|
||||
"email": "bounce@simulator.amazonses.com",
|
||||
"active": true
|
||||
}'
|
||||
```
|
||||
|
||||
Or use the Web UI:
|
||||
- Go to http://localhost:3000
|
||||
- Members tab → Add Member
|
||||
- Name: `Bounce Test`
|
||||
- Email: `bounce@simulator.amazonses.com`
|
||||
|
||||
2. **Subscribe to a test list:**
|
||||
- Click "Subscriptions" button for the test member
|
||||
- Toggle on one of your mailing lists
|
||||
|
||||
3. **Send email to the list:**
|
||||
```bash
|
||||
# From inside Postfix container
|
||||
sudo docker compose exec postfix bash
|
||||
echo "This will bounce" | mail -s "Test Bounce" community@lists.sasalliance.org
|
||||
exit
|
||||
```
|
||||
|
||||
Replace `community@lists.sasalliance.org` with your actual list email.
|
||||
|
||||
4. **Wait 30-60 seconds** for:
|
||||
- Email to be sent via SES
|
||||
- SES to process the bounce
|
||||
- SNS to send notification to your webhook
|
||||
|
||||
5. **Check the results:**
|
||||
|
||||
**Watch API logs in real-time:**
|
||||
```bash
|
||||
sudo docker compose logs api -f
|
||||
```
|
||||
|
||||
You should see:
|
||||
```
|
||||
============================================================
|
||||
SNS Webhook Request Received
|
||||
============================================================
|
||||
Content-Type: text/plain; charset=UTF-8
|
||||
User-Agent: Amazon Simple Notification Service Agent
|
||||
|
||||
✓ Notification Received
|
||||
Notification Type: Bounce
|
||||
|
||||
✓ Processing Bounce
|
||||
Bounce Type: Permanent
|
||||
Recipients: ['bounce@simulator.amazonses.com']
|
||||
✓ Bounce processed successfully
|
||||
============================================================
|
||||
```
|
||||
|
||||
**Check the database:**
|
||||
```bash
|
||||
# View bounce logs
|
||||
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist \
|
||||
-e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;"
|
||||
|
||||
# Check member status
|
||||
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist \
|
||||
-e "SELECT email, active, bounce_count, bounce_status FROM members WHERE email='bounce@simulator.amazonses.com';"
|
||||
```
|
||||
|
||||
**View in Web UI:**
|
||||
- Open http://localhost:3000
|
||||
- Go to Members tab
|
||||
- Find "Bounce Test" member
|
||||
- Should show: ❌ Inactive (red), bounce badge, last bounce timestamp
|
||||
- Click "Bounces" button to see detailed history
|
||||
|
||||
### Option 2: Direct Email via Postfix (Simpler)
|
||||
|
||||
Send directly to the simulator without going through your list:
|
||||
|
||||
```bash
|
||||
# Enter Postfix container
|
||||
sudo docker compose exec postfix bash
|
||||
|
||||
# Send test email
|
||||
echo "Testing hard bounce" | mail -s "Hard Bounce Test" bounce@simulator.amazonses.com
|
||||
|
||||
# Or soft bounce
|
||||
echo "Testing soft bounce" | mail -s "Soft Bounce Test" ooto@simulator.amazonses.com
|
||||
|
||||
# Exit container
|
||||
exit
|
||||
```
|
||||
|
||||
### Option 3: Using AWS SES Console (For Non-Sandbox Testing)
|
||||
|
||||
If you have SES production access:
|
||||
|
||||
1. Go to AWS SES Console
|
||||
2. Click "Send test email"
|
||||
3. To: `bounce@simulator.amazonses.com`
|
||||
4. From: Your verified email/domain
|
||||
5. Subject: "Bounce test"
|
||||
6. Body: "Testing bounce handling"
|
||||
7. Click "Send test email"
|
||||
|
||||
## Testing Different Bounce Types
|
||||
|
||||
### 1. Hard Bounce (Permanent Failure)
|
||||
```bash
|
||||
echo "Test" | mail -s "Test" bounce@simulator.amazonses.com
|
||||
```
|
||||
|
||||
**Expected Result:**
|
||||
- Member marked as `hard_bounce`
|
||||
- Member deactivated (`active = 0`)
|
||||
- `bounce_count` incremented
|
||||
- Entry in `bounce_logs` table
|
||||
|
||||
### 2. Soft Bounce (Transient Failure)
|
||||
```bash
|
||||
echo "Test" | mail -s "Test" ooto@simulator.amazonses.com
|
||||
```
|
||||
|
||||
**Expected Result:**
|
||||
- Member marked as `clean` (first time)
|
||||
- After 3 soft bounces → `soft_bounce` status
|
||||
- `bounce_count` incremented
|
||||
- Member stays active
|
||||
|
||||
### 3. Complaint (Spam Report)
|
||||
```bash
|
||||
echo "Test" | mail -s "Test" complaint@simulator.amazonses.com
|
||||
```
|
||||
|
||||
**Expected Result:**
|
||||
- API receives complaint notification
|
||||
- Currently logged but not processed (you can extend the handler)
|
||||
|
||||
## Monitoring the Test
|
||||
|
||||
### Real-Time Monitoring (Recommended)
|
||||
|
||||
Open 3 terminal windows:
|
||||
|
||||
**Terminal 1 - API Logs:**
|
||||
```bash
|
||||
sudo docker compose logs api -f
|
||||
```
|
||||
|
||||
**Terminal 2 - Postfix Logs:**
|
||||
```bash
|
||||
sudo docker compose logs postfix -f
|
||||
```
|
||||
|
||||
**Terminal 3 - Send Test Email:**
|
||||
```bash
|
||||
sudo docker compose exec postfix bash
|
||||
echo "Test" | mail -s "Bounce Test" bounce@simulator.amazonses.com
|
||||
```
|
||||
|
||||
### Timeline
|
||||
|
||||
Here's what happens and when:
|
||||
|
||||
- **T+0s**: Email sent to Postfix
|
||||
- **T+1-3s**: Postfix relays to SES
|
||||
- **T+5-10s**: SES processes and generates bounce
|
||||
- **T+10-30s**: SNS sends notification to your webhook
|
||||
- **T+30-60s**: API processes bounce and updates database
|
||||
|
||||
## Verifying the Complete Flow
|
||||
|
||||
### 1. Check Postfix Logs
|
||||
```bash
|
||||
sudo docker compose logs postfix | grep bounce@simulator
|
||||
```
|
||||
|
||||
Should show:
|
||||
```
|
||||
postfix/smtp[xxx]: ... to=<bounce@simulator.amazonses.com>, relay=email-smtp.eu-west-2.amazonaws.com[...], ... status=sent
|
||||
```
|
||||
|
||||
### 2. Check SNS Subscription Status
|
||||
- Go to AWS SNS Console
|
||||
- Find your topic
|
||||
- Check "Subscriptions" tab
|
||||
- Status should be "Confirmed"
|
||||
- Messages delivered should be > 0
|
||||
|
||||
### 3. Check API Logs
|
||||
```bash
|
||||
sudo docker compose logs api | grep -A 20 "SNS Webhook"
|
||||
```
|
||||
|
||||
Should show successful processing.
|
||||
|
||||
### 4. Check Database
|
||||
```bash
|
||||
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist <<EOF
|
||||
-- Show recent bounces
|
||||
SELECT
|
||||
b.bounce_id,
|
||||
m.email,
|
||||
b.bounce_type,
|
||||
b.diagnostic_code,
|
||||
b.timestamp
|
||||
FROM bounce_logs b
|
||||
JOIN members m ON b.member_id = m.member_id
|
||||
ORDER BY b.timestamp DESC
|
||||
LIMIT 5;
|
||||
|
||||
-- Show members with bounces
|
||||
SELECT
|
||||
email,
|
||||
active,
|
||||
bounce_count,
|
||||
bounce_status,
|
||||
last_bounce_at
|
||||
FROM members
|
||||
WHERE bounce_count > 0;
|
||||
EOF
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Email not sent" or "Relay access denied"
|
||||
|
||||
**Problem**: Postfix not configured to relay via SES
|
||||
|
||||
**Check**:
|
||||
```bash
|
||||
sudo docker compose exec postfix postconf relayhost
|
||||
sudo docker compose exec postfix postconf smtp_sasl_auth_enable
|
||||
```
|
||||
|
||||
Should show:
|
||||
```
|
||||
relayhost = [email-smtp.eu-west-2.amazonaws.com]:587
|
||||
smtp_sasl_auth_enable = yes
|
||||
```
|
||||
|
||||
### "No bounce received after 5 minutes"
|
||||
|
||||
**Possible causes**:
|
||||
|
||||
1. **SNS subscription not confirmed**
|
||||
- Check AWS SNS console
|
||||
- Status should be "Confirmed", not "Pending"
|
||||
|
||||
2. **SNS topic not configured in SES**
|
||||
- Check SES → Configuration Sets or Verified Identities → Notifications
|
||||
- Bounce notifications should point to your SNS topic
|
||||
|
||||
3. **Webhook endpoint not accessible**
|
||||
- SNS requires HTTPS
|
||||
- Test: `curl https://your-domain.com:8000/health`
|
||||
|
||||
4. **API container not running**
|
||||
```bash
|
||||
sudo docker compose ps api
|
||||
```
|
||||
|
||||
### "Bounce received but not in database"
|
||||
|
||||
**Check API logs for errors**:
|
||||
```bash
|
||||
sudo docker compose logs api | grep -i error
|
||||
```
|
||||
|
||||
**Check database tables exist**:
|
||||
```bash
|
||||
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e "SHOW TABLES;"
|
||||
```
|
||||
|
||||
Should include: `bounce_logs`, `members`
|
||||
|
||||
## Testing Multiple Bounces
|
||||
|
||||
To test the "3 soft bounces = soft_bounce status" logic:
|
||||
|
||||
```bash
|
||||
sudo docker compose exec postfix bash
|
||||
|
||||
# Send 3 emails to soft bounce simulator
|
||||
for i in {1..3}; do
|
||||
echo "Soft bounce test $i" | mail -s "Test $i" ooto@simulator.amazonses.com
|
||||
sleep 70 # Wait between sends for SNS processing
|
||||
done
|
||||
```
|
||||
|
||||
After the 3rd bounce:
|
||||
- Member's `bounce_status` should change from `clean` to `soft_bounce`
|
||||
- `bounce_count` should be 3
|
||||
|
||||
## Cleanup After Testing
|
||||
|
||||
Remove test bounce data:
|
||||
|
||||
```bash
|
||||
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist <<EOF
|
||||
-- Delete test bounce logs
|
||||
DELETE FROM bounce_logs WHERE email IN ('bounce@simulator.amazonses.com', 'ooto@simulator.amazonses.com');
|
||||
|
||||
-- Reset test member
|
||||
UPDATE members
|
||||
SET bounce_count = 0,
|
||||
bounce_status = 'clean',
|
||||
last_bounce_at = NULL,
|
||||
active = 1
|
||||
WHERE email IN ('bounce@simulator.amazonses.com', 'ooto@simulator.amazonses.com');
|
||||
EOF
|
||||
```
|
||||
|
||||
Or use the Web UI:
|
||||
- Go to Members tab
|
||||
- Find the test member
|
||||
- Click "Bounces" button
|
||||
- Click "Reset Bounce Status"
|
||||
|
||||
## Alternative: Simulate Bounces Without SES
|
||||
|
||||
If SNS isn't set up yet, use the included script to simulate bounces directly in the database:
|
||||
|
||||
```bash
|
||||
./simulate_bounce.sh
|
||||
```
|
||||
|
||||
This is useful for:
|
||||
- Testing the UI without AWS
|
||||
- Development environments
|
||||
- Demonstrating bounce handling to stakeholders
|
||||
|
||||
## Next Steps
|
||||
|
||||
Once bounce handling is working:
|
||||
|
||||
1. **Remove simulator addresses** from your member list
|
||||
2. **Monitor real bounces** in production
|
||||
3. **Set up alerts** for high bounce rates
|
||||
4. **Review bounced members** regularly and update/remove invalid addresses
|
||||
5. **Consider complaint handling** (similar to bounces, for spam reports)
|
||||
|
||||
## Summary Commands
|
||||
|
||||
**Quick test sequence**:
|
||||
```bash
|
||||
# 1. Watch logs
|
||||
sudo docker compose logs api -f &
|
||||
|
||||
# 2. Send test bounce
|
||||
echo "Test" | sudo docker compose exec -T postfix mail -s "Bounce Test" bounce@simulator.amazonses.com
|
||||
|
||||
# 3. Wait 60 seconds, then check database
|
||||
sleep 60
|
||||
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 1;"
|
||||
```
|
||||
|
||||
That's it! The bounce simulator is the easiest way to test your bounce handling without needing real bounced emails.
|
||||
132
SNS_DEBUG_GUIDE.md
Normal file
132
SNS_DEBUG_GUIDE.md
Normal file
@@ -0,0 +1,132 @@
|
||||
# SNS Webhook Debugging Guide
|
||||
|
||||
## Current Status
|
||||
|
||||
✅ **Detailed logging is now active!** The API will log all incoming SNS requests with:
|
||||
- Full headers (including SNS-specific headers)
|
||||
- Request body content
|
||||
- Parsed message structure
|
||||
- Processing steps and results
|
||||
|
||||
## How to View Logs in Real-Time
|
||||
|
||||
```bash
|
||||
# Watch API logs as SNS sends notifications
|
||||
sudo docker compose logs api -f
|
||||
```
|
||||
|
||||
## Testing with Real AWS SNS
|
||||
|
||||
### Step 1: Trigger a Test from SNS Console
|
||||
|
||||
1. Go to AWS SNS Console → Your Topic
|
||||
2. Click "Publish message"
|
||||
3. Send a test notification
|
||||
|
||||
### Step 2: Check the Logs
|
||||
|
||||
The logs will show you exactly what SNS sent:
|
||||
|
||||
```
|
||||
============================================================
|
||||
SNS Webhook Request Received
|
||||
============================================================
|
||||
Content-Type: text/plain; charset=UTF-8
|
||||
User-Agent: Amazon Simple Notification Service Agent
|
||||
X-Amz-SNS-Message-Type: Notification
|
||||
X-Amz-SNS-Topic-Arn: arn:aws:sns:...
|
||||
|
||||
Message Type: Notification
|
||||
Message Keys: ['Type', 'MessageId', 'TopicArn', 'Message', ...]
|
||||
|
||||
✓ Notification Received
|
||||
Message (first 200 chars): {"notificationType":"Bounce",...
|
||||
Notification Type: Bounce
|
||||
|
||||
✓ Processing Bounce
|
||||
Bounce Type: Permanent
|
||||
Recipients: ['bounce@example.com']
|
||||
✓ Bounce processed successfully
|
||||
============================================================
|
||||
```
|
||||
|
||||
## What the Logs Tell You
|
||||
|
||||
### If you see this:
|
||||
- **"SNS webhook received body: b''"** → SNS sent empty body (check SNS configuration)
|
||||
- **"Missing SigningCertURL"** → Message missing required SNS fields
|
||||
- **"Invalid certificate URL"** → Certificate URL not from amazonaws.com
|
||||
- **"Invalid signature"** → Signature verification failed (message tampered or wrong cert)
|
||||
|
||||
### Successful Flow:
|
||||
1. Headers logged → Shows SNS user agent and message type
|
||||
2. Body logged → Shows the raw JSON
|
||||
3. Message parsed → Shows the Type field
|
||||
4. Notification processed → Shows bounce/complaint details
|
||||
5. Database updated → Confirmation of processing
|
||||
|
||||
## Understanding Previous Errors
|
||||
|
||||
The error you saw earlier:
|
||||
```
|
||||
SNS webhook error: Expecting value: line 1 column 1 (char 0)
|
||||
```
|
||||
|
||||
This means SNS was sending an **empty body** or the body was already consumed. Possible causes:
|
||||
1. SNS subscription confirmation URL was opened in a browser (GET request, not POST)
|
||||
2. Network issue causing body to be lost
|
||||
3. SNS configuration issue
|
||||
|
||||
## Testing Bounce Handling
|
||||
|
||||
### Option 1: AWS SES Bounce Simulator
|
||||
Send email to: `bounce@simulator.amazonses.com`
|
||||
|
||||
### Option 2: Verify Database Updates
|
||||
After a bounce is processed:
|
||||
|
||||
```bash
|
||||
# Check bounce logs
|
||||
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;"
|
||||
|
||||
# Check member bounce status
|
||||
sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e "SELECT member_id, email, bounce_count, bounce_status, last_bounce_at FROM members WHERE bounce_count > 0;"
|
||||
```
|
||||
|
||||
## Common SNS Message Types
|
||||
|
||||
### 1. SubscriptionConfirmation
|
||||
First message when you create the subscription. API auto-confirms by calling SubscribeURL.
|
||||
|
||||
### 2. Notification (Bounce)
|
||||
```json
|
||||
{
|
||||
"Type": "Notification",
|
||||
"Message": "{\"notificationType\":\"Bounce\",\"bounce\":{...}}"
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Notification (Complaint)
|
||||
```json
|
||||
{
|
||||
"Type": "Notification",
|
||||
"Message": "{\"notificationType\":\"Complaint\",\"complaint\":{...}}"
|
||||
}
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Monitor logs while SNS sends**: `sudo docker compose logs api -f`
|
||||
2. **Trigger a real bounce**: Send to `bounce@simulator.amazonses.com`
|
||||
3. **Check the detailed logs** to see exactly what SNS is sending
|
||||
4. **Verify database updates** to confirm bounces are being recorded
|
||||
|
||||
## If You Still See Errors
|
||||
|
||||
The enhanced logging will now show you:
|
||||
- What headers SNS is sending
|
||||
- What body content is arriving (or if it's empty)
|
||||
- Where in the processing pipeline the error occurs
|
||||
- Full stack traces for any exceptions
|
||||
|
||||
Copy the relevant log section and we can diagnose the exact issue!
|
||||
@@ -2,6 +2,15 @@
|
||||
|
||||
REST API for managing mailing lists and members with token-based authentication.
|
||||
|
||||
## Overview
|
||||
|
||||
This FastAPI-based REST API provides programmatic access to the mailing list system. It's designed for:
|
||||
- **Automation**: Scripts and integrations that need to manage lists/members
|
||||
- **Web Interface**: Powers the frontend at http://localhost:3000
|
||||
- **Third-party Apps**: Any application that needs mailing list management
|
||||
|
||||
**For most users**, the web interface (http://localhost:3000) is more convenient. Use this API when you need programmatic access or automation.
|
||||
|
||||
## Base URL
|
||||
|
||||
```
|
||||
|
||||
831
api/main.py
831
api/main.py
@@ -2,24 +2,58 @@
|
||||
Mailing List Management API
|
||||
FastAPI-based REST API for managing mailing lists and members
|
||||
"""
|
||||
from fastapi import FastAPI, HTTPException, Depends, Header
|
||||
from fastapi import FastAPI, HTTPException, Depends, Header, status, Request
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
from fastapi.responses import PlainTextResponse
|
||||
from pydantic import BaseModel, EmailStr
|
||||
from typing import List, Optional
|
||||
from typing import List, Optional, Annotated
|
||||
import mysql.connector
|
||||
from mysql.connector import Error
|
||||
import os
|
||||
import csv
|
||||
import io
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timedelta
|
||||
import secrets
|
||||
import bcrypt
|
||||
from jose import JWTError, jwt
|
||||
from passlib.context import CryptContext
|
||||
from enum import Enum
|
||||
import json
|
||||
import base64
|
||||
from urllib.parse import urlparse
|
||||
import httpx
|
||||
from cryptography import x509
|
||||
from cryptography.hazmat.backends import default_backend
|
||||
from cryptography.hazmat.primitives import hashes
|
||||
from cryptography.hazmat.primitives.asymmetric import padding
|
||||
from cryptography.hazmat.primitives.serialization import load_pem_public_key
|
||||
|
||||
# Configuration
|
||||
API_TOKEN = os.getenv('API_TOKEN', 'change-this-token')
|
||||
API_TOKEN = os.getenv('API_TOKEN', 'change-this-token') # Keep for backward compatibility during transition
|
||||
JWT_SECRET_KEY = os.getenv('JWT_SECRET_KEY', 'your-secret-key-change-this-in-production')
|
||||
JWT_ALGORITHM = "HS256"
|
||||
JWT_ACCESS_TOKEN_EXPIRE_MINUTES = 30
|
||||
SESSION_EXPIRE_HOURS = 24
|
||||
|
||||
MYSQL_HOST = os.getenv('MYSQL_HOST', 'mysql')
|
||||
MYSQL_PORT = int(os.getenv('MYSQL_PORT', 3306))
|
||||
MYSQL_DATABASE = os.getenv('MYSQL_DATABASE', 'maillist')
|
||||
MYSQL_USER = os.getenv('MYSQL_USER', 'maillist')
|
||||
MYSQL_PASSWORD = os.getenv('MYSQL_PASSWORD', '')
|
||||
|
||||
# Bounce handling configuration
|
||||
ENABLE_SNS_WEBHOOKS = os.getenv('ENABLE_SNS_WEBHOOKS', 'false').lower() == 'true'
|
||||
ENABLE_BOUNCE_HANDLING = os.getenv('ENABLE_BOUNCE_HANDLING', 'false').lower() == 'true'
|
||||
ENABLE_EMAIL_BOUNCE_PROCESSING = os.getenv('ENABLE_EMAIL_BOUNCE_PROCESSING', 'false').lower() == 'true'
|
||||
|
||||
# Enable bounce handling if either method is enabled
|
||||
ENABLE_BOUNCE_HANDLING = ENABLE_BOUNCE_HANDLING or ENABLE_SNS_WEBHOOKS or ENABLE_EMAIL_BOUNCE_PROCESSING
|
||||
|
||||
# Password hashing
|
||||
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
|
||||
|
||||
# FastAPI app
|
||||
app = FastAPI(
|
||||
title="Mailing List Manager API",
|
||||
@@ -28,12 +62,15 @@ app = FastAPI(
|
||||
)
|
||||
|
||||
# Add CORS middleware
|
||||
# Get allowed origins from environment or use secure defaults
|
||||
ALLOWED_ORIGINS = os.getenv('ALLOWED_ORIGINS', 'http://localhost:3000,http://127.0.0.1:3000').split(',')
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"], # In production, specify your frontend domain
|
||||
allow_origins=ALLOWED_ORIGINS, # Specific origins only
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
allow_methods=["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"], # Specific methods
|
||||
allow_headers=["Authorization", "Content-Type", "Accept", "Origin", "X-Requested-With"], # Specific headers
|
||||
)
|
||||
|
||||
security = HTTPBearer()
|
||||
@@ -58,14 +95,144 @@ def get_db():
|
||||
if connection and connection.is_connected():
|
||||
connection.close()
|
||||
|
||||
# Authentication
|
||||
# Role-based access control
|
||||
class UserRole(str, Enum):
|
||||
ADMINISTRATOR = "administrator"
|
||||
OPERATOR = "operator"
|
||||
READ_ONLY = "read-only"
|
||||
|
||||
class CurrentUser(BaseModel):
|
||||
user_id: int
|
||||
username: str
|
||||
role: UserRole
|
||||
active: bool
|
||||
|
||||
# Authentication and authorization
|
||||
def verify_password(plain_password: str, hashed_password: str) -> bool:
|
||||
"""Verify a password against its hash"""
|
||||
return pwd_context.verify(plain_password, hashed_password)
|
||||
|
||||
def get_password_hash(password: str) -> str:
|
||||
"""Hash a password"""
|
||||
return pwd_context.hash(password)
|
||||
|
||||
def create_access_token(data: dict, expires_delta: Optional[timedelta] = None):
|
||||
"""Create a JWT access token"""
|
||||
to_encode = data.copy()
|
||||
if expires_delta:
|
||||
expire = datetime.utcnow() + expires_delta
|
||||
else:
|
||||
expire = datetime.utcnow() + timedelta(minutes=15)
|
||||
to_encode.update({"exp": expire})
|
||||
encoded_jwt = jwt.encode(to_encode, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
|
||||
return encoded_jwt
|
||||
|
||||
def create_session_id() -> str:
|
||||
"""Create a secure session ID"""
|
||||
return secrets.token_urlsafe(32)
|
||||
|
||||
async def get_current_user(credentials: HTTPAuthorizationCredentials = Depends(security)) -> CurrentUser:
|
||||
"""Get current authenticated user from JWT token"""
|
||||
credentials_exception = HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Could not validate credentials",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
|
||||
try:
|
||||
# Check if it's the old API token (for backward compatibility during transition)
|
||||
if credentials.credentials == API_TOKEN:
|
||||
# Return a fake admin user for legacy API token
|
||||
return CurrentUser(
|
||||
user_id=0,
|
||||
username="legacy_admin",
|
||||
role=UserRole.ADMINISTRATOR,
|
||||
active=True
|
||||
)
|
||||
|
||||
# Try to decode JWT token
|
||||
payload = jwt.decode(credentials.credentials, JWT_SECRET_KEY, algorithms=[JWT_ALGORITHM])
|
||||
username: str = payload.get("sub")
|
||||
if username is None:
|
||||
raise credentials_exception
|
||||
except JWTError:
|
||||
raise credentials_exception
|
||||
|
||||
# Get user from database
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
cursor.execute("SELECT * FROM users WHERE username = %s AND active = 1", (username,))
|
||||
user = cursor.fetchone()
|
||||
cursor.close()
|
||||
|
||||
if user is None:
|
||||
raise credentials_exception
|
||||
|
||||
return CurrentUser(
|
||||
user_id=user["user_id"],
|
||||
username=user["username"],
|
||||
role=UserRole(user["role"]),
|
||||
active=user["active"]
|
||||
)
|
||||
|
||||
def require_role(required_roles: List[UserRole]):
|
||||
"""Decorator factory for role-based access control"""
|
||||
def role_checker(current_user: CurrentUser = Depends(get_current_user)) -> CurrentUser:
|
||||
if current_user.role not in required_roles:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail=f"Insufficient permissions. Required roles: {[role.value for role in required_roles]}"
|
||||
)
|
||||
return current_user
|
||||
return role_checker
|
||||
|
||||
# Convenience functions for common role requirements
|
||||
def require_admin() -> CurrentUser:
|
||||
return Depends(require_role([UserRole.ADMINISTRATOR]))
|
||||
|
||||
def require_write_access() -> CurrentUser:
|
||||
return Depends(require_role([UserRole.ADMINISTRATOR, UserRole.OPERATOR]))
|
||||
|
||||
def require_read_access() -> CurrentUser:
|
||||
return Depends(require_role([UserRole.ADMINISTRATOR, UserRole.OPERATOR, UserRole.READ_ONLY]))
|
||||
|
||||
# Legacy authentication (for backward compatibility)
|
||||
def verify_token(credentials: HTTPAuthorizationCredentials = Depends(security)):
|
||||
"""Verify API token"""
|
||||
"""Legacy API token verification - deprecated, use get_current_user instead"""
|
||||
if credentials.credentials != API_TOKEN:
|
||||
raise HTTPException(status_code=401, detail="Invalid authentication token")
|
||||
return credentials.credentials
|
||||
|
||||
# Pydantic models
|
||||
# Pydantic models for authentication
|
||||
class LoginRequest(BaseModel):
|
||||
username: str
|
||||
password: str
|
||||
|
||||
class TokenResponse(BaseModel):
|
||||
access_token: str
|
||||
token_type: str = "bearer"
|
||||
expires_in: int
|
||||
user: dict
|
||||
|
||||
class CreateUserRequest(BaseModel):
|
||||
username: str
|
||||
password: str
|
||||
role: UserRole
|
||||
|
||||
class UpdateUserRequest(BaseModel):
|
||||
password: Optional[str] = None
|
||||
role: Optional[UserRole] = None
|
||||
active: Optional[bool] = None
|
||||
|
||||
class UserResponse(BaseModel):
|
||||
user_id: int
|
||||
username: str
|
||||
role: UserRole
|
||||
created_at: datetime
|
||||
last_login: Optional[datetime] = None
|
||||
active: bool
|
||||
|
||||
# Pydantic models for mailing list functionality
|
||||
class MailingList(BaseModel):
|
||||
list_id: Optional[int] = None
|
||||
list_name: str
|
||||
@@ -83,6 +250,9 @@ class Member(BaseModel):
|
||||
name: str
|
||||
email: EmailStr
|
||||
active: bool = True
|
||||
bounce_count: Optional[int] = 0
|
||||
last_bounce_at: Optional[datetime] = None
|
||||
bounce_status: Optional[str] = 'clean'
|
||||
|
||||
class MemberUpdate(BaseModel):
|
||||
name: Optional[str] = None
|
||||
@@ -93,6 +263,191 @@ class Subscription(BaseModel):
|
||||
member_email: EmailStr
|
||||
active: bool = True
|
||||
|
||||
class BulkImportRequest(BaseModel):
|
||||
csv_data: str
|
||||
list_ids: List[int]
|
||||
|
||||
class BulkImportResult(BaseModel):
|
||||
total_rows: int
|
||||
processed_rows: int
|
||||
created_members: int
|
||||
updated_members: int
|
||||
subscriptions_added: int
|
||||
errors: List[str]
|
||||
|
||||
# Authentication routes
|
||||
@app.post("/auth/login", response_model=TokenResponse)
|
||||
async def login(login_request: LoginRequest, request: Request):
|
||||
"""Authenticate user and return JWT token"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
cursor.execute("SELECT * FROM users WHERE username = %s AND active = 1", (login_request.username,))
|
||||
user = cursor.fetchone()
|
||||
|
||||
if not user or not verify_password(login_request.password, user["password_hash"]):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Incorrect username or password"
|
||||
)
|
||||
|
||||
# Update last login
|
||||
cursor.execute("UPDATE users SET last_login = %s WHERE user_id = %s",
|
||||
(datetime.utcnow(), user["user_id"]))
|
||||
conn.commit()
|
||||
|
||||
# Create session record
|
||||
session_id = create_session_id()
|
||||
expires_at = datetime.utcnow() + timedelta(hours=SESSION_EXPIRE_HOURS)
|
||||
|
||||
client_ip = request.client.host if request.client else None
|
||||
user_agent = request.headers.get("user-agent", "")
|
||||
|
||||
cursor.execute("""
|
||||
INSERT INTO user_sessions (session_id, user_id, expires_at, ip_address, user_agent)
|
||||
VALUES (%s, %s, %s, %s, %s)
|
||||
""", (session_id, user["user_id"], expires_at, client_ip, user_agent))
|
||||
conn.commit()
|
||||
cursor.close()
|
||||
|
||||
# Create JWT token
|
||||
access_token_expires = timedelta(minutes=JWT_ACCESS_TOKEN_EXPIRE_MINUTES)
|
||||
access_token = create_access_token(
|
||||
data={"sub": user["username"]}, expires_delta=access_token_expires
|
||||
)
|
||||
|
||||
return TokenResponse(
|
||||
access_token=access_token,
|
||||
expires_in=JWT_ACCESS_TOKEN_EXPIRE_MINUTES * 60,
|
||||
user={
|
||||
"user_id": user["user_id"],
|
||||
"username": user["username"],
|
||||
"role": user["role"],
|
||||
"active": user["active"]
|
||||
}
|
||||
)
|
||||
|
||||
@app.post("/auth/logout")
|
||||
async def logout(current_user: CurrentUser = Depends(get_current_user)):
|
||||
"""Logout current user (invalidate sessions)"""
|
||||
if current_user.user_id == 0: # Legacy admin user
|
||||
return {"message": "Logged out successfully"}
|
||||
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("UPDATE user_sessions SET active = 0 WHERE user_id = %s", (current_user.user_id,))
|
||||
conn.commit()
|
||||
cursor.close()
|
||||
|
||||
return {"message": "Logged out successfully"}
|
||||
|
||||
@app.get("/users", response_model=List[UserResponse])
|
||||
async def get_users(current_user: CurrentUser = require_admin()):
|
||||
"""Get all users (admin only)"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
cursor.execute("SELECT user_id, username, role, created_at, last_login, active FROM users ORDER BY username")
|
||||
users = cursor.fetchall()
|
||||
cursor.close()
|
||||
return users
|
||||
|
||||
@app.post("/users", response_model=UserResponse, status_code=201)
|
||||
async def create_user(user_request: CreateUserRequest, current_user: CurrentUser = require_admin()):
|
||||
"""Create a new user (admin only)"""
|
||||
# Check if username already exists
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
cursor.execute("SELECT user_id FROM users WHERE username = %s", (user_request.username,))
|
||||
existing_user = cursor.fetchone()
|
||||
|
||||
if existing_user:
|
||||
raise HTTPException(status_code=400, detail="Username already exists")
|
||||
|
||||
# Hash password and create user
|
||||
password_hash = get_password_hash(user_request.password)
|
||||
cursor.execute("""
|
||||
INSERT INTO users (username, password_hash, role)
|
||||
VALUES (%s, %s, %s)
|
||||
""", (user_request.username, password_hash, user_request.role.value))
|
||||
|
||||
conn.commit()
|
||||
user_id = cursor.lastrowid
|
||||
|
||||
# Return created user
|
||||
cursor.execute("""
|
||||
SELECT user_id, username, role, created_at, last_login, active
|
||||
FROM users WHERE user_id = %s
|
||||
""", (user_id,))
|
||||
new_user = cursor.fetchone()
|
||||
cursor.close()
|
||||
|
||||
return new_user
|
||||
|
||||
@app.patch("/users/{user_id}", response_model=UserResponse)
|
||||
async def update_user(user_id: int, updates: UpdateUserRequest, current_user: CurrentUser = require_admin()):
|
||||
"""Update a user (admin only)"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
|
||||
# Build update query dynamically
|
||||
update_fields = []
|
||||
values = []
|
||||
|
||||
if updates.password is not None:
|
||||
update_fields.append("password_hash = %s")
|
||||
values.append(get_password_hash(updates.password))
|
||||
if updates.role is not None:
|
||||
update_fields.append("role = %s")
|
||||
values.append(updates.role.value)
|
||||
if updates.active is not None:
|
||||
update_fields.append("active = %s")
|
||||
values.append(updates.active)
|
||||
|
||||
if not update_fields:
|
||||
raise HTTPException(status_code=400, detail="No fields to update")
|
||||
|
||||
values.append(user_id)
|
||||
query = f"UPDATE users SET {', '.join(update_fields)} WHERE user_id = %s"
|
||||
|
||||
cursor.execute(query, values)
|
||||
conn.commit()
|
||||
|
||||
# Return updated user
|
||||
cursor.execute("""
|
||||
SELECT user_id, username, role, created_at, last_login, active
|
||||
FROM users WHERE user_id = %s
|
||||
""", (user_id,))
|
||||
updated_user = cursor.fetchone()
|
||||
cursor.close()
|
||||
|
||||
if not updated_user:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
return updated_user
|
||||
|
||||
@app.delete("/users/{user_id}", status_code=204)
|
||||
async def delete_user(user_id: int, current_user: CurrentUser = require_admin()):
|
||||
"""Delete a user (admin only)"""
|
||||
if user_id == current_user.user_id:
|
||||
raise HTTPException(status_code=400, detail="Cannot delete your own account")
|
||||
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("DELETE FROM users WHERE user_id = %s", (user_id,))
|
||||
conn.commit()
|
||||
|
||||
if cursor.rowcount == 0:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
cursor.close()
|
||||
|
||||
@app.get("/auth/me", response_model=dict)
|
||||
async def get_current_user_info(current_user: CurrentUser = Depends(get_current_user)):
|
||||
"""Get current user information"""
|
||||
return {
|
||||
"user_id": current_user.user_id,
|
||||
"username": current_user.username,
|
||||
"role": current_user.role.value,
|
||||
"active": current_user.active
|
||||
}
|
||||
|
||||
# Routes
|
||||
@app.get("/")
|
||||
async def root():
|
||||
@@ -116,9 +471,18 @@ async def health():
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=503, detail=f"Unhealthy: {str(e)}")
|
||||
|
||||
@app.get("/config")
|
||||
async def get_config():
|
||||
"""Get public configuration settings"""
|
||||
return {
|
||||
"bounce_handling_enabled": ENABLE_BOUNCE_HANDLING,
|
||||
"sns_webhooks_enabled": ENABLE_SNS_WEBHOOKS,
|
||||
"email_bounce_processing_enabled": ENABLE_EMAIL_BOUNCE_PROCESSING
|
||||
}
|
||||
|
||||
# Mailing Lists endpoints
|
||||
@app.get("/lists", response_model=List[MailingList])
|
||||
async def get_lists(token: str = Depends(verify_token)):
|
||||
async def get_lists(current_user: CurrentUser = require_read_access()):
|
||||
"""Get all mailing lists"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
@@ -128,7 +492,7 @@ async def get_lists(token: str = Depends(verify_token)):
|
||||
return lists
|
||||
|
||||
@app.get("/lists/{list_id}", response_model=MailingList)
|
||||
async def get_list(list_id: int, token: str = Depends(verify_token)):
|
||||
async def get_list(list_id: int, current_user: CurrentUser = require_read_access()):
|
||||
"""Get a specific mailing list"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
@@ -141,7 +505,7 @@ async def get_list(list_id: int, token: str = Depends(verify_token)):
|
||||
return mailing_list
|
||||
|
||||
@app.post("/lists", response_model=MailingList, status_code=201)
|
||||
async def create_list(mailing_list: MailingList, token: str = Depends(verify_token)):
|
||||
async def create_list(mailing_list: MailingList, current_user: CurrentUser = require_write_access()):
|
||||
"""Create a new mailing list"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor()
|
||||
@@ -158,7 +522,7 @@ async def create_list(mailing_list: MailingList, token: str = Depends(verify_tok
|
||||
raise HTTPException(status_code=400, detail=f"Failed to create list: {str(e)}")
|
||||
|
||||
@app.patch("/lists/{list_id}", response_model=MailingList)
|
||||
async def update_list(list_id: int, updates: MailingListUpdate, token: str = Depends(verify_token)):
|
||||
async def update_list(list_id: int, updates: MailingListUpdate, current_user: CurrentUser = require_write_access()):
|
||||
"""Update a mailing list"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
@@ -196,7 +560,7 @@ async def update_list(list_id: int, updates: MailingListUpdate, token: str = Dep
|
||||
return updated_list
|
||||
|
||||
@app.delete("/lists/{list_id}", status_code=204)
|
||||
async def delete_list(list_id: int, token: str = Depends(verify_token)):
|
||||
async def delete_list(list_id: int, current_user: CurrentUser = require_write_access()):
|
||||
"""Delete a mailing list"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor()
|
||||
@@ -209,7 +573,7 @@ async def delete_list(list_id: int, token: str = Depends(verify_token)):
|
||||
|
||||
# Members endpoints
|
||||
@app.get("/members", response_model=List[Member])
|
||||
async def get_members(token: str = Depends(verify_token)):
|
||||
async def get_members(current_user: CurrentUser = require_read_access()):
|
||||
"""Get all members"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
@@ -219,7 +583,7 @@ async def get_members(token: str = Depends(verify_token)):
|
||||
return members
|
||||
|
||||
@app.get("/members/{member_id}", response_model=Member)
|
||||
async def get_member(member_id: int, token: str = Depends(verify_token)):
|
||||
async def get_member(member_id: int, current_user: CurrentUser = require_read_access()):
|
||||
"""Get a specific member"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
@@ -232,7 +596,7 @@ async def get_member(member_id: int, token: str = Depends(verify_token)):
|
||||
return member
|
||||
|
||||
@app.post("/members", response_model=Member, status_code=201)
|
||||
async def create_member(member: Member, token: str = Depends(verify_token)):
|
||||
async def create_member(member: Member, current_user: CurrentUser = require_write_access()):
|
||||
"""Create a new member"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor()
|
||||
@@ -249,7 +613,7 @@ async def create_member(member: Member, token: str = Depends(verify_token)):
|
||||
raise HTTPException(status_code=400, detail=f"Failed to create member: {str(e)}")
|
||||
|
||||
@app.patch("/members/{member_id}", response_model=Member)
|
||||
async def update_member(member_id: int, updates: MemberUpdate, token: str = Depends(verify_token)):
|
||||
async def update_member(member_id: int, updates: MemberUpdate, current_user: CurrentUser = require_write_access()):
|
||||
"""Update a member"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
@@ -282,7 +646,7 @@ async def update_member(member_id: int, updates: MemberUpdate, token: str = Depe
|
||||
return updated_member
|
||||
|
||||
@app.delete("/members/{member_id}", status_code=204)
|
||||
async def delete_member(member_id: int, token: str = Depends(verify_token)):
|
||||
async def delete_member(member_id: int, current_user: CurrentUser = require_write_access()):
|
||||
"""Delete a member"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor()
|
||||
@@ -295,7 +659,7 @@ async def delete_member(member_id: int, token: str = Depends(verify_token)):
|
||||
|
||||
# Subscription endpoints
|
||||
@app.get("/lists/{list_id}/members", response_model=List[Member])
|
||||
async def get_list_members(list_id: int, token: str = Depends(verify_token)):
|
||||
async def get_list_members(list_id: int, current_user: CurrentUser = require_read_access()):
|
||||
"""Get all members of a specific list"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
@@ -311,7 +675,7 @@ async def get_list_members(list_id: int, token: str = Depends(verify_token)):
|
||||
return members
|
||||
|
||||
@app.post("/subscriptions", status_code=201)
|
||||
async def subscribe_member(subscription: Subscription, token: str = Depends(verify_token)):
|
||||
async def subscribe_member(subscription: Subscription, current_user: CurrentUser = require_write_access()):
|
||||
"""Subscribe a member to a list"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor()
|
||||
@@ -345,7 +709,7 @@ async def subscribe_member(subscription: Subscription, token: str = Depends(veri
|
||||
raise HTTPException(status_code=400, detail=f"Failed to create subscription: {str(e)}")
|
||||
|
||||
@app.delete("/subscriptions")
|
||||
async def unsubscribe_member(list_email: EmailStr, member_email: EmailStr, token: str = Depends(verify_token)):
|
||||
async def unsubscribe_member(list_email: EmailStr, member_email: EmailStr, current_user: CurrentUser = require_write_access()):
|
||||
"""Unsubscribe a member from a list"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor()
|
||||
@@ -365,6 +729,429 @@ async def unsubscribe_member(list_email: EmailStr, member_email: EmailStr, token
|
||||
cursor.close()
|
||||
return {"message": "Unsubscribed successfully"}
|
||||
|
||||
@app.post("/bulk-import", response_model=BulkImportResult)
|
||||
async def bulk_import_members(bulk_request: BulkImportRequest, current_user: CurrentUser = require_write_access()):
|
||||
"""Bulk import members from CSV data and subscribe them to specified lists"""
|
||||
|
||||
result = BulkImportResult(
|
||||
total_rows=0,
|
||||
processed_rows=0,
|
||||
created_members=0,
|
||||
updated_members=0,
|
||||
subscriptions_added=0,
|
||||
errors=[]
|
||||
)
|
||||
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
|
||||
try:
|
||||
# Verify all list_ids exist
|
||||
if bulk_request.list_ids:
|
||||
placeholders = ','.join(['%s'] * len(bulk_request.list_ids))
|
||||
cursor.execute(f"SELECT list_id FROM lists WHERE list_id IN ({placeholders})", bulk_request.list_ids)
|
||||
existing_lists = [row['list_id'] for row in cursor.fetchall()]
|
||||
|
||||
invalid_lists = set(bulk_request.list_ids) - set(existing_lists)
|
||||
if invalid_lists:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid list IDs: {list(invalid_lists)}")
|
||||
|
||||
# Parse CSV data
|
||||
csv_reader = csv.DictReader(io.StringIO(bulk_request.csv_data))
|
||||
|
||||
# Validate CSV headers - we need at least Name and Email
|
||||
if not csv_reader.fieldnames or 'Name' not in csv_reader.fieldnames or 'Email' not in csv_reader.fieldnames:
|
||||
raise HTTPException(status_code=400, detail="CSV must contain 'Name' and 'Email' columns")
|
||||
|
||||
for row_num, row in enumerate(csv_reader, start=1):
|
||||
result.total_rows += 1
|
||||
|
||||
try:
|
||||
name = row.get('Name', '').strip()
|
||||
email = row.get('Email', '').strip()
|
||||
|
||||
# Skip rows without email (name is optional)
|
||||
if not email:
|
||||
result.errors.append(f"Row {row_num}: Missing email address")
|
||||
continue
|
||||
|
||||
# Basic email validation
|
||||
if '@' not in email or '.' not in email.split('@')[1]:
|
||||
result.errors.append(f"Row {row_num}: Invalid email format: {email}")
|
||||
continue
|
||||
|
||||
# Use email as name if no name provided
|
||||
if not name:
|
||||
name = email.split('@')[0] # Use the part before @ as a default name
|
||||
|
||||
# Check if member exists
|
||||
cursor.execute("SELECT member_id FROM members WHERE email = %s", (email,))
|
||||
existing_member = cursor.fetchone()
|
||||
|
||||
if existing_member:
|
||||
# Update existing member's name if we have a better name (not auto-generated from email)
|
||||
should_update_name = (
|
||||
row.get('Name', '').strip() and # We have a name in the CSV
|
||||
name != email.split('@')[0] # It's not the auto-generated name
|
||||
)
|
||||
if should_update_name:
|
||||
cursor.execute("UPDATE members SET name = %s WHERE member_id = %s", (name, existing_member['member_id']))
|
||||
if cursor.rowcount > 0:
|
||||
result.updated_members += 1
|
||||
member_id = existing_member['member_id']
|
||||
else:
|
||||
# Create new member
|
||||
cursor.execute(
|
||||
"INSERT INTO members (name, email, active) VALUES (%s, %s, %s)",
|
||||
(name, email, True)
|
||||
)
|
||||
member_id = cursor.lastrowid
|
||||
result.created_members += 1
|
||||
|
||||
# Subscribe to selected lists
|
||||
for list_id in bulk_request.list_ids:
|
||||
try:
|
||||
cursor.execute(
|
||||
"INSERT INTO list_members (list_id, member_id, active) VALUES (%s, %s, %s)",
|
||||
(list_id, member_id, True)
|
||||
)
|
||||
result.subscriptions_added += 1
|
||||
except Error as e:
|
||||
if "Duplicate entry" in str(e):
|
||||
# Member already subscribed to this list - not an error
|
||||
pass
|
||||
else:
|
||||
result.errors.append(f"Row {row_num}: Failed to subscribe to list {list_id}: {str(e)}")
|
||||
|
||||
result.processed_rows += 1
|
||||
|
||||
except Exception as e:
|
||||
result.errors.append(f"Row {row_num}: {str(e)}")
|
||||
continue
|
||||
|
||||
# Commit all changes
|
||||
conn.commit()
|
||||
cursor.close()
|
||||
|
||||
return result
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
cursor.close()
|
||||
raise HTTPException(status_code=500, detail=f"Bulk import failed: {str(e)}")
|
||||
|
||||
# SNS Webhook for Bounce Handling (conditionally enabled)
|
||||
if ENABLE_SNS_WEBHOOKS:
|
||||
async def verify_sns_signature(request: Request) -> dict:
|
||||
"""Verify SNS message signature"""
|
||||
try:
|
||||
body = await request.body()
|
||||
print(f"SNS webhook received body: {body}")
|
||||
print(f"SNS webhook body length: {len(body)}")
|
||||
print(f"SNS webhook headers: {dict(request.headers)}")
|
||||
|
||||
if not body:
|
||||
print("ERROR: Empty body received")
|
||||
raise HTTPException(status_code=400, detail="Empty request body")
|
||||
|
||||
message = json.loads(body.decode('utf-8'))
|
||||
print(f"SNS webhook parsed message type: {message.get('Type')}")
|
||||
|
||||
# For SubscriptionConfirmation and UnsubscribeConfirmation, we don't validate signature
|
||||
# AWS will send a URL to confirm
|
||||
if message.get('Type') in ['SubscriptionConfirmation', 'UnsubscribeConfirmation']:
|
||||
return message
|
||||
|
||||
# Get certificate URL and download certificate
|
||||
cert_url = message.get('SigningCertURL')
|
||||
if not cert_url:
|
||||
raise HTTPException(status_code=400, detail="Missing SigningCertURL")
|
||||
|
||||
# Verify certificate URL is from AWS
|
||||
parsed_url = urlparse(cert_url)
|
||||
if not parsed_url.hostname.endswith('.amazonaws.com'):
|
||||
raise HTTPException(status_code=400, detail="Invalid certificate URL")
|
||||
|
||||
# Download certificate
|
||||
async with httpx.AsyncClient() as client:
|
||||
cert_response = await client.get(cert_url)
|
||||
cert_response.raise_for_status()
|
||||
cert_pem = cert_response.content
|
||||
|
||||
# Load certificate and extract public key
|
||||
cert = x509.load_pem_x509_certificate(cert_pem, default_backend())
|
||||
public_key = cert.public_key()
|
||||
|
||||
# Build string to sign
|
||||
if message.get('Type') == 'Notification':
|
||||
string_to_sign = (
|
||||
f"Message\n{message['Message']}\n"
|
||||
f"MessageId\n{message['MessageId']}\n"
|
||||
)
|
||||
if 'Subject' in message:
|
||||
string_to_sign += f"Subject\n{message['Subject']}\n"
|
||||
string_to_sign += (
|
||||
f"Timestamp\n{message['Timestamp']}\n"
|
||||
f"TopicArn\n{message['TopicArn']}\n"
|
||||
f"Type\n{message['Type']}\n"
|
||||
)
|
||||
else:
|
||||
string_to_sign = (
|
||||
f"Message\n{message['Message']}\n"
|
||||
f"MessageId\n{message['MessageId']}\n"
|
||||
f"SubscribeURL\n{message['SubscribeURL']}\n"
|
||||
f"Timestamp\n{message['Timestamp']}\n"
|
||||
f"Token\n{message['Token']}\n"
|
||||
f"TopicArn\n{message['TopicArn']}\n"
|
||||
f"Type\n{message['Type']}\n"
|
||||
)
|
||||
|
||||
# Verify signature
|
||||
signature = base64.b64decode(message['Signature'])
|
||||
try:
|
||||
public_key.verify(
|
||||
signature,
|
||||
string_to_sign.encode('utf-8'),
|
||||
padding.PKCS1v15(),
|
||||
hashes.SHA1()
|
||||
)
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid signature: {str(e)}")
|
||||
|
||||
return message
|
||||
|
||||
except json.JSONDecodeError:
|
||||
raise HTTPException(status_code=400, detail="Invalid JSON")
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=400, detail=f"Signature verification failed: {str(e)}")
|
||||
|
||||
async def process_bounce(bounce_data: dict):
|
||||
"""Process bounce notification and update database"""
|
||||
try:
|
||||
bounce_type = bounce_data.get('bounceType') # Permanent, Transient, Undetermined
|
||||
bounce_subtype = bounce_data.get('bounceSubType', '')
|
||||
timestamp_str = bounce_data.get('timestamp')
|
||||
feedback_id = bounce_data.get('feedbackId', '')
|
||||
|
||||
# Convert ISO 8601 timestamp to MySQL datetime format
|
||||
# SES sends: '2025-10-13T16:22:40.359Z'
|
||||
# MySQL needs: '2025-10-13 16:22:40'
|
||||
from datetime import datetime as dt
|
||||
timestamp = dt.fromisoformat(timestamp_str.replace('Z', '+00:00')).strftime('%Y-%m-%d %H:%M:%S')
|
||||
|
||||
bounced_recipients = bounce_data.get('bouncedRecipients', [])
|
||||
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
|
||||
for recipient in bounced_recipients:
|
||||
email = recipient.get('emailAddress')
|
||||
diagnostic_code = recipient.get('diagnosticCode', '')
|
||||
|
||||
if not email:
|
||||
continue
|
||||
|
||||
# Find member by email
|
||||
cursor.execute("SELECT member_id FROM members WHERE email = %s", (email,))
|
||||
member = cursor.fetchone()
|
||||
member_id = member['member_id'] if member else None
|
||||
|
||||
# Log the bounce
|
||||
cursor.execute("""
|
||||
INSERT INTO bounce_logs
|
||||
(member_id, email, bounce_type, bounce_subtype, diagnostic_code, timestamp, feedback_id)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s)
|
||||
""", (member_id, email, bounce_type, bounce_subtype, diagnostic_code, timestamp, feedback_id))
|
||||
|
||||
# Update member bounce status
|
||||
if member_id:
|
||||
# Determine bounce status
|
||||
if bounce_type == 'Permanent':
|
||||
new_status = 'hard_bounce'
|
||||
# Deactivate member with hard bounce
|
||||
cursor.execute("""
|
||||
UPDATE members
|
||||
SET bounce_count = bounce_count + 1,
|
||||
last_bounce_at = %s,
|
||||
bounce_status = %s,
|
||||
active = 0
|
||||
WHERE member_id = %s
|
||||
""", (timestamp, new_status, member_id))
|
||||
elif bounce_type == 'Transient':
|
||||
# Check current bounce count
|
||||
cursor.execute("SELECT bounce_count, bounce_status FROM members WHERE member_id = %s", (member_id,))
|
||||
current = cursor.fetchone()
|
||||
|
||||
# If already hard bounced, don't change status
|
||||
if current and current['bounce_status'] != 'hard_bounce':
|
||||
new_count = current['bounce_count'] + 1
|
||||
# After 3 soft bounces, mark as soft_bounce status
|
||||
new_status = 'soft_bounce' if new_count >= 3 else 'clean'
|
||||
|
||||
cursor.execute("""
|
||||
UPDATE members
|
||||
SET bounce_count = %s,
|
||||
last_bounce_at = %s,
|
||||
bounce_status = %s
|
||||
WHERE member_id = %s
|
||||
""", (new_count, timestamp, new_status, member_id))
|
||||
else: # Undetermined
|
||||
cursor.execute("""
|
||||
UPDATE members
|
||||
SET bounce_count = bounce_count + 1,
|
||||
last_bounce_at = %s
|
||||
WHERE member_id = %s
|
||||
""", (timestamp, member_id))
|
||||
|
||||
conn.commit()
|
||||
cursor.close()
|
||||
|
||||
except Exception as e:
|
||||
print(f"✗ Error processing bounce: {str(e)}")
|
||||
print(f"Error type: {type(e).__name__}")
|
||||
print(f"Bounce data: {bounce_data}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
raise
|
||||
|
||||
@app.post("/webhooks/sns", response_class=PlainTextResponse)
|
||||
async def sns_webhook(request: Request):
|
||||
"""Handle SNS notifications for bounces and complaints"""
|
||||
try:
|
||||
print(f"=== SNS Webhook Request ===")
|
||||
print(f"Headers: {dict(request.headers)}")
|
||||
print(f"Content-Type: {request.headers.get('content-type')}")
|
||||
print(f"User-Agent: {request.headers.get('user-agent')}")
|
||||
|
||||
# Verify SNS signature
|
||||
message = await verify_sns_signature(request)
|
||||
|
||||
print(f"Message Type: {message.get('Type')}")
|
||||
print(f"Message Keys: {list(message.keys())}")
|
||||
|
||||
message_type = message.get('Type')
|
||||
|
||||
# Handle subscription confirmation
|
||||
if message_type == 'SubscriptionConfirmation':
|
||||
subscribe_url = message.get('SubscribeURL')
|
||||
print(f"Subscription confirmation received, URL: {subscribe_url}")
|
||||
if subscribe_url:
|
||||
# Confirm subscription
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get(subscribe_url)
|
||||
print(f"Subscription confirmation response: {response.status_code}")
|
||||
return "Subscription confirmed"
|
||||
|
||||
# Handle notification
|
||||
elif message_type == 'Notification':
|
||||
# Parse the message
|
||||
inner_message = message.get('Message', '{}')
|
||||
print(f"Inner message (first 500 chars): {inner_message[:500]}")
|
||||
notification = json.loads(inner_message)
|
||||
|
||||
# SES can send either 'notificationType' or 'eventType' depending on configuration
|
||||
notification_type = notification.get('notificationType') or notification.get('eventType')
|
||||
print(f"Notification type: {notification_type}")
|
||||
|
||||
if notification_type == 'Bounce':
|
||||
bounce = notification.get('bounce', {})
|
||||
print(f"\n✓ Processing Bounce")
|
||||
print(f" Bounce Type: {bounce.get('bounceType')}")
|
||||
print(f" Recipients: {[r.get('emailAddress') for r in bounce.get('bouncedRecipients', [])]}")
|
||||
await process_bounce(bounce)
|
||||
print(f" ✓ Bounce processed successfully")
|
||||
return "Bounce processed"
|
||||
|
||||
elif notification_type == 'Complaint':
|
||||
# We could also track complaints similarly to bounces
|
||||
print(f"\n✓ Complaint received")
|
||||
return "Complaint received"
|
||||
|
||||
print(f"=== End SNS Webhook Request ===")
|
||||
return "OK"
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"SNS webhook error: {str(e)}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
else:
|
||||
# Provide stub functions when SNS webhooks are disabled
|
||||
print("SNS webhooks disabled - bounce handling via email only")
|
||||
|
||||
# Bounce management endpoints (conditionally enabled)
|
||||
if ENABLE_BOUNCE_HANDLING:
|
||||
class BounceLog(BaseModel):
|
||||
bounce_id: int
|
||||
email: str
|
||||
bounce_type: str
|
||||
bounce_subtype: Optional[str] = None
|
||||
diagnostic_code: Optional[str] = None
|
||||
timestamp: datetime
|
||||
feedback_id: Optional[str] = None
|
||||
created_at: datetime
|
||||
|
||||
class MemberWithBounces(BaseModel):
|
||||
member_id: int
|
||||
name: str
|
||||
email: str
|
||||
active: bool
|
||||
bounce_count: int
|
||||
last_bounce_at: Optional[datetime] = None
|
||||
bounce_status: str
|
||||
|
||||
@app.get("/members/{member_id}/bounces", response_model=List[BounceLog])
|
||||
async def get_member_bounces(member_id: int, current_user: CurrentUser = require_read_access()):
|
||||
"""Get bounce history for a member"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor(dictionary=True)
|
||||
cursor.execute("""
|
||||
SELECT bounce_id, email, bounce_type, bounce_subtype, diagnostic_code,
|
||||
timestamp, feedback_id, created_at
|
||||
FROM bounce_logs
|
||||
WHERE member_id = %s
|
||||
ORDER BY timestamp DESC
|
||||
""", (member_id,))
|
||||
bounces = cursor.fetchall()
|
||||
cursor.close()
|
||||
return bounces
|
||||
|
||||
@app.patch("/members/{member_id}/bounce-status")
|
||||
async def reset_bounce_status(member_id: int, current_user: CurrentUser = require_write_access()):
|
||||
"""Reset bounce status for a member (e.g., after email address is corrected)"""
|
||||
with get_db() as conn:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("""
|
||||
UPDATE members
|
||||
SET bounce_count = 0,
|
||||
last_bounce_at = NULL,
|
||||
bounce_status = 'clean'
|
||||
WHERE member_id = %s
|
||||
""", (member_id,))
|
||||
conn.commit()
|
||||
|
||||
if cursor.rowcount == 0:
|
||||
raise HTTPException(status_code=404, detail="Member not found")
|
||||
|
||||
cursor.close()
|
||||
return {"message": "Bounce status reset successfully"}
|
||||
|
||||
else:
|
||||
# When bounce handling is disabled, provide stub endpoints that return appropriate responses
|
||||
@app.get("/members/{member_id}/bounces")
|
||||
async def get_member_bounces_disabled(member_id: int, current_user: CurrentUser = require_read_access()):
|
||||
"""Bounce history disabled - returns empty list"""
|
||||
return []
|
||||
|
||||
@app.patch("/members/{member_id}/bounce-status")
|
||||
async def reset_bounce_status_disabled(member_id: int, current_user: CurrentUser = require_write_access()):
|
||||
"""Bounce status reset disabled"""
|
||||
raise HTTPException(status_code=501, detail="Bounce handling is disabled")
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8000)
|
||||
|
||||
@@ -5,3 +5,7 @@ pydantic==2.5.0
|
||||
pydantic-settings==2.1.0
|
||||
python-multipart==0.0.6
|
||||
email-validator==2.1.0
|
||||
bcrypt==4.0.1
|
||||
python-jose[cryptography]==3.3.0
|
||||
passlib[bcrypt]==1.7.4
|
||||
httpx==0.25.2
|
||||
|
||||
@@ -2,6 +2,26 @@
|
||||
|
||||
This mailing list system uses MySQL with **Postfix's native MySQL support** for real-time dynamic list management. Postfix queries the database directly for each email - no scripts or reloads needed.
|
||||
|
||||
## Management Options
|
||||
|
||||
**1. Web Interface (Recommended for Non-Technical Users)**
|
||||
- Access: http://localhost:3000
|
||||
- Visual interface with tables and forms
|
||||
- Toggle-based subscription management
|
||||
- No SQL knowledge required
|
||||
|
||||
**2. REST API (Recommended for Automation)**
|
||||
- Access: http://localhost:8000/docs
|
||||
- Full CRUD operations via HTTP
|
||||
- Token authentication
|
||||
- Perfect for scripts and integrations
|
||||
|
||||
**3. Direct MySQL (Recommended for Advanced Users)**
|
||||
- Full SQL access for complex queries
|
||||
- Bulk operations and reporting
|
||||
- Database administration tasks
|
||||
- Described in detail below
|
||||
|
||||
## Database Schema
|
||||
|
||||
Three-table design with many-to-many relationships:
|
||||
@@ -36,7 +56,38 @@ Three-table design with many-to-many relationships:
|
||||
|
||||
## Managing Lists and Members
|
||||
|
||||
### Via MySQL Client
|
||||
### Via Web Interface (Easiest)
|
||||
|
||||
1. Open http://localhost:3000 in your browser
|
||||
2. Enter your API_TOKEN (from .env file)
|
||||
3. Use the tabs to:
|
||||
- **Lists Tab**: View, create, edit, delete mailing lists
|
||||
- **Members Tab**: View, add, edit, remove members
|
||||
- **Subscriptions**: Click "Subscriptions" button on any member to toggle their list memberships
|
||||
|
||||
### Via REST API (For Automation)
|
||||
|
||||
See `api/README.md` for complete API documentation, or visit http://localhost:8000/docs for interactive docs.
|
||||
|
||||
Quick examples:
|
||||
```bash
|
||||
# Get all lists
|
||||
curl -H "Authorization: Bearer $API_TOKEN" http://localhost:8000/lists
|
||||
|
||||
# Create member
|
||||
curl -X POST http://localhost:8000/members \
|
||||
-H "Authorization: Bearer $API_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"name":"John Doe","email":"john@example.com","active":true}'
|
||||
|
||||
# Subscribe to list
|
||||
curl -X POST http://localhost:8000/subscriptions \
|
||||
-H "Authorization: Bearer $API_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"list_email":"community@lists.sasalliance.org","member_email":"john@example.com"}'
|
||||
```
|
||||
|
||||
### Via MySQL Client (Advanced)
|
||||
|
||||
Connect to the database:
|
||||
|
||||
|
||||
@@ -1,5 +1,37 @@
|
||||
-- Mail List Manager Database Schema
|
||||
|
||||
-- Table: users
|
||||
-- Stores user authentication and authorization information
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
user_id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
username VARCHAR(50) NOT NULL UNIQUE,
|
||||
password_hash VARCHAR(255) NOT NULL, -- bcrypt hash
|
||||
role ENUM('administrator', 'operator', 'read-only') NOT NULL DEFAULT 'read-only',
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
last_login TIMESTAMP NULL,
|
||||
active BOOLEAN DEFAULT TRUE,
|
||||
INDEX idx_username (username),
|
||||
INDEX idx_role (role),
|
||||
INDEX idx_active (active)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
|
||||
|
||||
-- Table: user_sessions
|
||||
-- Stores active user sessions for authentication
|
||||
CREATE TABLE IF NOT EXISTS user_sessions (
|
||||
session_id VARCHAR(64) PRIMARY KEY,
|
||||
user_id INT NOT NULL,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
expires_at TIMESTAMP NOT NULL,
|
||||
ip_address VARCHAR(45), -- Supports both IPv4 and IPv6
|
||||
user_agent TEXT,
|
||||
active BOOLEAN DEFAULT TRUE,
|
||||
FOREIGN KEY (user_id) REFERENCES users(user_id) ON DELETE CASCADE,
|
||||
INDEX idx_user_id (user_id),
|
||||
INDEX idx_expires_at (expires_at),
|
||||
INDEX idx_active (active)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
|
||||
|
||||
-- Table: lists
|
||||
-- Stores mailing list information
|
||||
CREATE TABLE IF NOT EXISTS lists (
|
||||
@@ -23,8 +55,12 @@ CREATE TABLE IF NOT EXISTS members (
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
active BOOLEAN DEFAULT TRUE,
|
||||
bounce_count INT DEFAULT 0,
|
||||
last_bounce_at TIMESTAMP NULL,
|
||||
bounce_status ENUM('clean', 'soft_bounce', 'hard_bounce') DEFAULT 'clean',
|
||||
INDEX idx_email (email),
|
||||
INDEX idx_active (active)
|
||||
INDEX idx_active (active),
|
||||
INDEX idx_bounce_status (bounce_status)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
|
||||
|
||||
-- Table: list_members
|
||||
@@ -43,7 +79,33 @@ CREATE TABLE IF NOT EXISTS list_members (
|
||||
INDEX idx_active (active)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
|
||||
|
||||
-- Table: bounce_logs
|
||||
-- Stores bounce notifications from SES SNS
|
||||
CREATE TABLE IF NOT EXISTS bounce_logs (
|
||||
bounce_id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
member_id INT,
|
||||
email VARCHAR(255) NOT NULL,
|
||||
bounce_type ENUM('Permanent', 'Transient', 'Undetermined') NOT NULL,
|
||||
bounce_subtype VARCHAR(50),
|
||||
diagnostic_code TEXT,
|
||||
timestamp TIMESTAMP NOT NULL,
|
||||
sns_message_id VARCHAR(255),
|
||||
feedback_id VARCHAR(255),
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
FOREIGN KEY (member_id) REFERENCES members(member_id) ON DELETE SET NULL,
|
||||
INDEX idx_member_id (member_id),
|
||||
INDEX idx_email (email),
|
||||
INDEX idx_timestamp (timestamp),
|
||||
INDEX idx_bounce_type (bounce_type)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
|
||||
|
||||
-- Insert sample data
|
||||
|
||||
-- Create default admin user (password: 'password')
|
||||
-- Hash generated with passlib CryptContext using bcrypt
|
||||
INSERT INTO users (username, password_hash, role) VALUES
|
||||
('admin', '$2b$12$6Lsp/i.clxyEE9O/4D9RLOfyFlxfjvJBhJ8VXT/t4H/QvrnKrE/YK', 'administrator');
|
||||
|
||||
INSERT INTO lists (list_name, list_email, description) VALUES
|
||||
('Community', 'community@lists.sasalliance.org', 'General community announcements'),
|
||||
('Board', 'board@lists.sasalliance.org', 'Board members only'),
|
||||
|
||||
59
nginx-production-example.conf
Normal file
59
nginx-production-example.conf
Normal file
@@ -0,0 +1,59 @@
|
||||
# Example nginx configuration for production with SNS webhook support
|
||||
server {
|
||||
listen 80;
|
||||
server_name yourdomain.com;
|
||||
|
||||
# Redirect HTTP to HTTPS in production
|
||||
return 301 https://$server_name$request_uri;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name yourdomain.com;
|
||||
|
||||
# SSL configuration (add your certificates)
|
||||
# ssl_certificate /path/to/your/cert.pem;
|
||||
# ssl_certificate_key /path/to/your/key.pem;
|
||||
|
||||
# Frontend (static files)
|
||||
location / {
|
||||
proxy_pass http://maillist-web:80;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
|
||||
# API endpoints (including SNS webhook)
|
||||
location /api/ {
|
||||
# Remove /api prefix when forwarding to backend
|
||||
rewrite ^/api/(.*) /$1 break;
|
||||
|
||||
proxy_pass http://maillist-api:8000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Important for SNS webhooks - increase timeouts and body size
|
||||
proxy_connect_timeout 60s;
|
||||
proxy_send_timeout 60s;
|
||||
proxy_read_timeout 60s;
|
||||
client_max_body_size 10M;
|
||||
}
|
||||
|
||||
# Direct SNS webhook endpoint (alternative path)
|
||||
location /webhooks/sns {
|
||||
proxy_pass http://maillist-api:8000/webhooks/sns;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# SNS specific settings
|
||||
proxy_connect_timeout 30s;
|
||||
proxy_send_timeout 30s;
|
||||
proxy_read_timeout 30s;
|
||||
client_max_body_size 1M;
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,8 @@ RUN apt-get update && \
|
||||
mailutils \
|
||||
gettext-base \
|
||||
netcat-openbsd \
|
||||
python3 \
|
||||
python3-pymysql \
|
||||
&& apt-get clean && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy configs
|
||||
@@ -16,11 +18,16 @@ COPY main.cf.template /etc/postfix/main.cf.template
|
||||
COPY sasl_passwd.template /etc/postfix/sasl_passwd.template
|
||||
COPY mysql_virtual_alias_maps.cf /etc/postfix/mysql_virtual_alias_maps.cf.template
|
||||
COPY sender_access /etc/postfix/sender_access
|
||||
COPY smtp_generic /etc/postfix/smtp_generic
|
||||
COPY aliases /etc/aliases
|
||||
COPY process-bounce.py /usr/local/bin/process-bounce.py
|
||||
COPY entrypoint.sh /entrypoint.sh
|
||||
RUN chmod +x /entrypoint.sh
|
||||
RUN chmod +x /entrypoint.sh /usr/local/bin/process-bounce.py
|
||||
|
||||
# Generate Postfix maps for sender access
|
||||
RUN postmap /etc/postfix/sender_access
|
||||
# Generate Postfix maps for sender access, sender canonical, and aliases
|
||||
RUN postmap /etc/postfix/sender_access && \
|
||||
postmap /etc/postfix/smtp_generic && \
|
||||
newaliases
|
||||
|
||||
# Expose SMTP
|
||||
EXPOSE 25
|
||||
|
||||
17
postfix/aliases
Normal file
17
postfix/aliases
Normal file
@@ -0,0 +1,17 @@
|
||||
# Postfix aliases for bounce handling
|
||||
#
|
||||
# This file defines how bounces are processed
|
||||
# The bounces address pipes messages to our bounce processing script
|
||||
|
||||
# Bounce processing - pipe to Python script
|
||||
bounces: "|/usr/bin/python3 /usr/local/bin/process-bounce.py"
|
||||
|
||||
# Standard aliases
|
||||
postmaster: root
|
||||
mailer-daemon: postmaster
|
||||
abuse: postmaster
|
||||
spam: postmaster
|
||||
root: postmaster
|
||||
|
||||
# Default fallback
|
||||
MAILER-DAEMON: bounces
|
||||
@@ -32,5 +32,28 @@ chmod 644 /etc/postfix/sender_access /etc/postfix/sender_access.db
|
||||
# Set permissions on MySQL config
|
||||
chmod 644 /etc/postfix/mysql_virtual_alias_maps.cf
|
||||
|
||||
# Configure bounce processing based on environment variable
|
||||
if [ "${ENABLE_EMAIL_BOUNCE_PROCESSING:-false}" = "true" ]; then
|
||||
echo "Email-based bounce processing enabled"
|
||||
# Regenerate aliases database to enable bounce processing
|
||||
newaliases
|
||||
chmod 644 /etc/aliases /etc/aliases.db
|
||||
|
||||
# Generate smtp generic maps to ensure bounces come to our bounce address
|
||||
postmap /etc/postfix/smtp_generic
|
||||
chmod 644 /etc/postfix/smtp_generic /etc/postfix/smtp_generic.db
|
||||
echo "Configured return path rewriting to bounces@lists.sasalliance.org"
|
||||
else
|
||||
echo "Email-based bounce processing disabled"
|
||||
# Create minimal aliases without bounce processing
|
||||
echo "postmaster: root" > /etc/aliases
|
||||
echo "root: postmaster" >> /etc/aliases
|
||||
newaliases
|
||||
|
||||
# Disable smtp generic maps
|
||||
echo "# Email bounce processing disabled" > /etc/postfix/smtp_generic
|
||||
postmap /etc/postfix/smtp_generic
|
||||
fi
|
||||
|
||||
# Start Postfix in foreground
|
||||
exec postfix start-fg
|
||||
|
||||
@@ -33,3 +33,23 @@ smtpd_recipient_restrictions =
|
||||
# Other recommended settings
|
||||
alias_maps = hash:/etc/aliases
|
||||
alias_database = hash:/etc/aliases
|
||||
|
||||
# Bounce handling configuration for email-based processing
|
||||
# Configure bounce notification recipients
|
||||
bounce_notice_recipient = bounces@lists.sasalliance.org
|
||||
2bounce_notice_recipient = bounces@lists.sasalliance.org
|
||||
delay_notice_recipient =
|
||||
error_notice_recipient = bounces@lists.sasalliance.org
|
||||
|
||||
# Bounce settings
|
||||
bounce_size_limit = 50000
|
||||
bounce_queue_lifetime = 5d
|
||||
maximal_bounce_delay = 1d
|
||||
|
||||
# Return path configuration - CRITICAL for bounce handling
|
||||
# This ensures that when we relay emails via mailing lists through SES,
|
||||
# bounces come back to our bounce processing address
|
||||
#
|
||||
# Use smtp_generic_maps instead of sender_canonical_maps because we only want
|
||||
# to rewrite the return path for outbound SMTP (via SES), not for local delivery
|
||||
smtp_generic_maps = hash:/etc/postfix/smtp_generic
|
||||
|
||||
348
postfix/process-bounce.py
Normal file
348
postfix/process-bounce.py
Normal file
@@ -0,0 +1,348 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Email-based bounce processing script for Postfix
|
||||
Parses bounce emails and updates the database with bounce information
|
||||
|
||||
This script is called by Postfix when bounce emails are received.
|
||||
It reads the email from stdin, parses it for bounce information,
|
||||
and updates the database accordingly.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import re
|
||||
import email
|
||||
from email.message import Message
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import List, Tuple, Optional, Dict
|
||||
import pymysql
|
||||
|
||||
# Configure logging
|
||||
log_file = '/var/log/bounce-processor.log'
|
||||
log_handlers = [logging.StreamHandler()]
|
||||
|
||||
# Add file handler if we can write to the log directory
|
||||
try:
|
||||
log_handlers.append(logging.FileHandler(log_file))
|
||||
except (PermissionError, FileNotFoundError):
|
||||
pass # Just use stdout if we can't write to log file
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
|
||||
handlers=log_handlers
|
||||
)
|
||||
logger = logging.getLogger('bounce-processor')
|
||||
|
||||
# Database configuration from environment
|
||||
MYSQL_HOST = os.getenv('MYSQL_HOST', 'mysql')
|
||||
MYSQL_PORT = int(os.getenv('MYSQL_PORT', 3306))
|
||||
MYSQL_DATABASE = os.getenv('MYSQL_DATABASE', 'maillist')
|
||||
MYSQL_USER = os.getenv('MYSQL_USER', 'maillist')
|
||||
MYSQL_PASSWORD = os.getenv('MYSQL_PASSWORD', '')
|
||||
|
||||
class BounceProcessor:
|
||||
"""Processes bounce emails and updates the database"""
|
||||
|
||||
def __init__(self):
|
||||
self.bounce_patterns = self._compile_bounce_patterns()
|
||||
|
||||
def _compile_bounce_patterns(self) -> List[re.Pattern]:
|
||||
"""Compile regex patterns for detecting bounce information"""
|
||||
patterns = [
|
||||
# Standard bounce formats
|
||||
re.compile(r'550.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
|
||||
re.compile(r'554.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
|
||||
re.compile(r'553.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
|
||||
re.compile(r'552.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
|
||||
re.compile(r'551.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
|
||||
|
||||
# Delivery Status Notification (DSN) format
|
||||
re.compile(r'Final-Recipient:.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
|
||||
re.compile(r'Original-Recipient:.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
|
||||
|
||||
# Common bounce message patterns
|
||||
re.compile(r'user.*?([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?unknown', re.IGNORECASE),
|
||||
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?does not exist', re.IGNORECASE),
|
||||
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?not found', re.IGNORECASE),
|
||||
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?mailbox.*?unavailable', re.IGNORECASE),
|
||||
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*?recipient.*?rejected', re.IGNORECASE),
|
||||
|
||||
# Generic email extraction (fallback)
|
||||
re.compile(r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', re.IGNORECASE),
|
||||
]
|
||||
return patterns
|
||||
|
||||
def parse_bounce_email(self, email_content: str) -> List[Dict]:
|
||||
"""Parse bounce email and extract bounce information"""
|
||||
try:
|
||||
# Parse the email
|
||||
msg = email.message_from_string(email_content)
|
||||
bounces = []
|
||||
|
||||
# Get the email body
|
||||
body = self._get_email_body(msg)
|
||||
if not body:
|
||||
logger.warning("No email body found")
|
||||
return bounces
|
||||
|
||||
logger.info(f"Processing email body (first 500 chars): {body[:500]}")
|
||||
|
||||
# Extract bounced addresses
|
||||
bounced_addresses = self._extract_bounced_addresses(body)
|
||||
|
||||
# Determine bounce type and create bounce records
|
||||
for address in bounced_addresses:
|
||||
bounce_info = self._analyze_bounce(body, address)
|
||||
if bounce_info:
|
||||
bounces.append(bounce_info)
|
||||
|
||||
return bounces
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error parsing bounce email: {str(e)}")
|
||||
return []
|
||||
|
||||
def _get_email_body(self, msg: Message) -> Optional[str]:
|
||||
"""Extract the email body from the message"""
|
||||
body = ""
|
||||
|
||||
if msg.is_multipart():
|
||||
for part in msg.walk():
|
||||
content_type = part.get_content_type()
|
||||
if content_type in ['text/plain', 'text/html']:
|
||||
payload = part.get_payload(decode=True)
|
||||
if payload:
|
||||
body += payload.decode('utf-8', errors='ignore') + "\n"
|
||||
else:
|
||||
payload = msg.get_payload(decode=True)
|
||||
if payload:
|
||||
body = payload.decode('utf-8', errors='ignore')
|
||||
|
||||
return body.strip() if body else None
|
||||
|
||||
def _extract_bounced_addresses(self, body: str) -> List[str]:
|
||||
"""Extract bounced email addresses from the bounce message"""
|
||||
addresses = []
|
||||
|
||||
for pattern in self.bounce_patterns:
|
||||
matches = pattern.findall(body)
|
||||
for match in matches:
|
||||
email_addr = match.strip().lower()
|
||||
if self._is_valid_email(email_addr) and email_addr not in addresses:
|
||||
# Skip our own addresses
|
||||
if not email_addr.endswith('@lists.sasalliance.org'):
|
||||
addresses.append(email_addr)
|
||||
|
||||
logger.info(f"Extracted addresses: {addresses}")
|
||||
return addresses
|
||||
|
||||
def _is_valid_email(self, email_addr: str) -> bool:
|
||||
"""Validate email address format"""
|
||||
email_pattern = re.compile(r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$')
|
||||
return bool(email_pattern.match(email_addr))
|
||||
|
||||
def _analyze_bounce(self, body: str, email_addr: str) -> Optional[Dict]:
|
||||
"""Analyze bounce message to determine bounce type and details"""
|
||||
bounce_info = {
|
||||
'email': email_addr,
|
||||
'bounce_type': 'Undetermined',
|
||||
'bounce_subtype': '',
|
||||
'diagnostic_code': '',
|
||||
'timestamp': datetime.now().strftime('%Y-%m-%d %H:%M:%S')
|
||||
}
|
||||
|
||||
# Analyze bounce type based on SMTP codes and message content
|
||||
if re.search(r'5[0-9]{2}', body): # 5xx codes are permanent failures
|
||||
bounce_info['bounce_type'] = 'Permanent'
|
||||
|
||||
if re.search(r'550|551|553', body):
|
||||
bounce_info['bounce_subtype'] = 'General'
|
||||
elif re.search(r'552', body):
|
||||
bounce_info['bounce_subtype'] = 'MailboxFull'
|
||||
elif re.search(r'554', body):
|
||||
bounce_info['bounce_subtype'] = 'MessageTooLarge'
|
||||
|
||||
elif re.search(r'4[0-9]{2}', body): # 4xx codes are temporary failures
|
||||
bounce_info['bounce_type'] = 'Transient'
|
||||
|
||||
if re.search(r'450|451|452', body):
|
||||
bounce_info['bounce_subtype'] = 'General'
|
||||
elif re.search(r'452', body):
|
||||
bounce_info['bounce_subtype'] = 'MailboxFull'
|
||||
|
||||
# Extract diagnostic code
|
||||
smtp_code_match = re.search(r'([45][0-9]{2}.*?)[\r\n]', body)
|
||||
if smtp_code_match:
|
||||
bounce_info['diagnostic_code'] = smtp_code_match.group(1).strip()[:500] # Limit length
|
||||
|
||||
return bounce_info
|
||||
|
||||
def update_database(self, bounces: List[Dict]) -> None:
|
||||
"""Update the database with bounce information"""
|
||||
if not bounces:
|
||||
logger.info("No bounces to process")
|
||||
return
|
||||
|
||||
try:
|
||||
connection = pymysql.connect(
|
||||
host=MYSQL_HOST,
|
||||
port=MYSQL_PORT,
|
||||
database=MYSQL_DATABASE,
|
||||
user=MYSQL_USER,
|
||||
password=MYSQL_PASSWORD,
|
||||
cursorclass=pymysql.cursors.DictCursor
|
||||
)
|
||||
|
||||
cursor = connection.cursor()
|
||||
|
||||
for bounce in bounces:
|
||||
try:
|
||||
email_addr = bounce['email']
|
||||
|
||||
# Find member by email
|
||||
cursor.execute("SELECT member_id FROM members WHERE email = %s", (email_addr,))
|
||||
member = cursor.fetchone()
|
||||
member_id = member['member_id'] if member else None
|
||||
|
||||
# Log the bounce
|
||||
cursor.execute("""
|
||||
INSERT INTO bounce_logs
|
||||
(member_id, email, bounce_type, bounce_subtype, diagnostic_code, timestamp, created_at)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, NOW())
|
||||
""", (member_id, email_addr, bounce['bounce_type'], bounce['bounce_subtype'],
|
||||
bounce['diagnostic_code'], bounce['timestamp']))
|
||||
|
||||
logger.info(f"Logged bounce for {email_addr}: {bounce['bounce_type']}")
|
||||
|
||||
# Update member bounce status if member exists
|
||||
if member_id:
|
||||
if bounce['bounce_type'] == 'Permanent':
|
||||
# Hard bounce - deactivate member
|
||||
cursor.execute("""
|
||||
UPDATE members
|
||||
SET bounce_count = bounce_count + 1,
|
||||
last_bounce_at = %s,
|
||||
bounce_status = 'hard_bounce',
|
||||
active = 0
|
||||
WHERE member_id = %s
|
||||
""", (bounce['timestamp'], member_id))
|
||||
logger.info(f"Deactivated member {email_addr} due to hard bounce")
|
||||
|
||||
elif bounce['bounce_type'] == 'Transient':
|
||||
# Soft bounce - increment counter and check threshold
|
||||
cursor.execute("SELECT bounce_count, bounce_status FROM members WHERE member_id = %s", (member_id,))
|
||||
current = cursor.fetchone()
|
||||
|
||||
if current and current['bounce_status'] != 'hard_bounce':
|
||||
new_count = current['bounce_count'] + 1
|
||||
new_status = 'soft_bounce' if new_count >= 3 else 'clean'
|
||||
|
||||
cursor.execute("""
|
||||
UPDATE members
|
||||
SET bounce_count = %s,
|
||||
last_bounce_at = %s,
|
||||
bounce_status = %s
|
||||
WHERE member_id = %s
|
||||
""", (new_count, bounce['timestamp'], new_status, member_id))
|
||||
|
||||
logger.info(f"Updated member {email_addr} bounce count to {new_count}, status: {new_status}")
|
||||
else:
|
||||
# Undetermined - just increment counter
|
||||
cursor.execute("""
|
||||
UPDATE members
|
||||
SET bounce_count = bounce_count + 1,
|
||||
last_bounce_at = %s
|
||||
WHERE member_id = %s
|
||||
""", (bounce['timestamp'], member_id))
|
||||
|
||||
logger.info(f"Updated member {email_addr} bounce count (undetermined)")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Database error processing bounce for {bounce['email']}: {str(e)}")
|
||||
continue
|
||||
|
||||
connection.commit()
|
||||
cursor.close()
|
||||
connection.close()
|
||||
|
||||
logger.info(f"Successfully processed {len(bounces)} bounces")
|
||||
|
||||
except pymysql.Error as e:
|
||||
logger.error(f"Database connection error: {str(e)}")
|
||||
except Exception as e:
|
||||
logger.error(f"Unexpected error updating database: {str(e)}")
|
||||
|
||||
def main():
|
||||
"""Main function - reads email from stdin and processes bounces"""
|
||||
try:
|
||||
logger.info("Starting bounce processing")
|
||||
|
||||
# Check if we're in test mode
|
||||
test_mode = len(sys.argv) > 1 and sys.argv[1] == '--test'
|
||||
|
||||
if test_mode:
|
||||
# Test mode - use sample bounce email
|
||||
email_content = """From: Mail Delivery Subsystem <MAILER-DAEMON@example.com>
|
||||
To: bounces@lists.sasalliance.org
|
||||
Subject: Delivery Status Notification (Failure)
|
||||
Content-Type: multipart/report; report-type=delivery-status; boundary="boundary123"
|
||||
|
||||
--boundary123
|
||||
Content-Type: text/plain
|
||||
|
||||
This is a test bounce message.
|
||||
|
||||
The following address(es) failed:
|
||||
testuser@example.com
|
||||
SMTP error from remote mail server after RCPT TO:<testuser@example.com>:
|
||||
550 5.1.1 User unknown
|
||||
|
||||
--boundary123
|
||||
Content-Type: message/delivery-status
|
||||
|
||||
Reporting-MTA: dns; mail.example.com
|
||||
Received-From-MTA: dns; localhost
|
||||
|
||||
Final-Recipient: rfc822; testuser@example.com
|
||||
Action: failed
|
||||
Status: 5.1.1
|
||||
Diagnostic-Code: smtp; 550 5.1.1 User unknown
|
||||
|
||||
--boundary123--
|
||||
"""
|
||||
logger.info("Running in test mode with sample bounce email")
|
||||
else:
|
||||
# Read email from stdin
|
||||
email_content = sys.stdin.read()
|
||||
|
||||
if not email_content.strip():
|
||||
logger.warning("No email content received")
|
||||
return
|
||||
|
||||
logger.info(f"Received email content ({len(email_content)} bytes)")
|
||||
|
||||
# Process the bounce
|
||||
processor = BounceProcessor()
|
||||
bounces = processor.parse_bounce_email(email_content)
|
||||
|
||||
if test_mode:
|
||||
logger.info(f"Test mode - would process {len(bounces)} bounce(s):")
|
||||
for bounce in bounces:
|
||||
logger.info(f" {bounce}")
|
||||
else:
|
||||
if bounces:
|
||||
processor.update_database(bounces)
|
||||
logger.info(f"Processed {len(bounces)} bounce(s)")
|
||||
else:
|
||||
logger.info("No bounces detected in email")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error in main: {str(e)}")
|
||||
import traceback
|
||||
logger.error(f"Traceback: {traceback.format_exc()}")
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
23
postfix/smtp_generic
Normal file
23
postfix/smtp_generic
Normal file
@@ -0,0 +1,23 @@
|
||||
# Postfix SMTP Generic Maps
|
||||
# This file controls the return path (envelope sender) for outbound SMTP emails
|
||||
# Only applies to emails being relayed through external SMTP (SES in our case)
|
||||
#
|
||||
# Format: original_sender rewritten_sender
|
||||
#
|
||||
# For mailing list emails, we want bounces to go to our bounce processing address
|
||||
# This is critical for email-based bounce handling to work properly
|
||||
|
||||
# Rewrite envelope sender for all mailing list addresses to bounce address
|
||||
# When emails are forwarded through mailing lists via SES, bounces come back to us
|
||||
community@lists.sasalliance.org bounces@lists.sasalliance.org
|
||||
board@lists.sasalliance.org bounces@lists.sasalliance.org
|
||||
members@lists.sasalliance.org bounces@lists.sasalliance.org
|
||||
announcements@lists.sasalliance.org bounces@lists.sasalliance.org
|
||||
|
||||
# Generic pattern - any @lists.sasalliance.org sender gets rewritten to bounce address
|
||||
# This catches any new lists automatically
|
||||
@lists.sasalliance.org bounces@lists.sasalliance.org
|
||||
|
||||
# Also rewrite any envelope sender that's sending through our system
|
||||
# This ensures ALL outgoing mail via SES has our bounce address as return path
|
||||
@sasalliance.org bounces@lists.sasalliance.org
|
||||
@@ -1,11 +0,0 @@
|
||||
# Community mailing list - general announcements
|
||||
community@lists.sasalliance.org james@pattinson.org, james.pattinson@sasalliance.org
|
||||
|
||||
# Board members mailing list
|
||||
board@lists.sasalliance.org james.pattinson@sasalliance.org
|
||||
|
||||
# All members mailing list
|
||||
members@lists.sasalliance.org james@pattinson.org, james.pattinson@sasalliance.org
|
||||
|
||||
# Announcements mailing list
|
||||
announcements@lists.sasalliance.org james@pattinson.org, james.pattinson@sasalliance.org
|
||||
206
simulate_bounce.sh
Executable file
206
simulate_bounce.sh
Executable file
@@ -0,0 +1,206 @@
|
||||
#!/bin/bash
|
||||
# Simulate Bounce Events for Testing
|
||||
# This script inserts bounce data directly into the database to test the UI
|
||||
# without needing to set up AWS SNS
|
||||
|
||||
set -e
|
||||
|
||||
echo "=== Bounce Simulation Script ==="
|
||||
echo
|
||||
echo "This script will create test bounce events for existing members"
|
||||
echo
|
||||
|
||||
# Get database password from .env
|
||||
DB_PASSWORD=$(grep MYSQL_ROOT_PASSWORD /home/jamesp/docker/maillist/.env | cut -d'=' -f2)
|
||||
|
||||
# Check if containers are running
|
||||
if ! sudo docker-compose ps | grep -q "maillist-mysql.*Up"; then
|
||||
echo "❌ MySQL container is not running. Starting containers..."
|
||||
sudo docker-compose up -d
|
||||
echo "Waiting for MySQL to be ready..."
|
||||
sleep 5
|
||||
fi
|
||||
|
||||
echo "1. Fetching existing members..."
|
||||
echo
|
||||
|
||||
# Get list of members
|
||||
MEMBERS=$(sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -N -e "SELECT member_id, name, email FROM members LIMIT 5;")
|
||||
|
||||
if [ -z "$MEMBERS" ]; then
|
||||
echo "❌ No members found in database. Please add members first."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Available members:"
|
||||
echo "$MEMBERS" | while read -r line; do
|
||||
MEMBER_ID=$(echo "$line" | awk '{print $1}')
|
||||
NAME=$(echo "$line" | awk '{print $2, $3}')
|
||||
EMAIL=$(echo "$line" | awk '{$1=""; $2=""; print $0}' | xargs)
|
||||
echo " [$MEMBER_ID] $NAME - $EMAIL"
|
||||
done
|
||||
echo
|
||||
|
||||
# Prompt for member ID
|
||||
read -p "Enter member ID to simulate bounce for (or press Enter for first member): " MEMBER_ID
|
||||
|
||||
if [ -z "$MEMBER_ID" ]; then
|
||||
MEMBER_ID=$(echo "$MEMBERS" | head -1 | awk '{print $1}')
|
||||
echo "Using member ID: $MEMBER_ID"
|
||||
fi
|
||||
|
||||
# Get member details
|
||||
MEMBER_INFO=$(sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -N -e "SELECT member_id, name, email FROM members WHERE member_id = $MEMBER_ID;")
|
||||
|
||||
if [ -z "$MEMBER_INFO" ]; then
|
||||
echo "❌ Member ID $MEMBER_ID not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
MEMBER_EMAIL=$(echo "$MEMBER_INFO" | awk '{$1=""; $2=""; print $0}' | xargs)
|
||||
MEMBER_NAME=$(echo "$MEMBER_INFO" | awk '{print $2, $3}')
|
||||
|
||||
echo
|
||||
echo "Selected member: $MEMBER_NAME ($MEMBER_EMAIL)"
|
||||
echo
|
||||
|
||||
# Bounce type selection
|
||||
echo "Select bounce type to simulate:"
|
||||
echo " 1) Soft Bounce (Transient - e.g., mailbox full)"
|
||||
echo " 2) Hard Bounce (Permanent - e.g., invalid address)"
|
||||
echo " 3) Multiple Soft Bounces (3 bounces to trigger soft_bounce status)"
|
||||
echo " 4) Undetermined Bounce"
|
||||
echo
|
||||
read -p "Enter choice (1-4): " BOUNCE_CHOICE
|
||||
|
||||
case $BOUNCE_CHOICE in
|
||||
1)
|
||||
BOUNCE_TYPE="Transient"
|
||||
BOUNCE_SUBTYPE="MailboxFull"
|
||||
DIAGNOSTIC="smtp; 452 4.2.2 Mailbox full"
|
||||
COUNT=1
|
||||
;;
|
||||
2)
|
||||
BOUNCE_TYPE="Permanent"
|
||||
BOUNCE_SUBTYPE="General"
|
||||
DIAGNOSTIC="smtp; 550 5.1.1 User unknown"
|
||||
COUNT=1
|
||||
;;
|
||||
3)
|
||||
BOUNCE_TYPE="Transient"
|
||||
BOUNCE_SUBTYPE="General"
|
||||
DIAGNOSTIC="smtp; 451 4.4.1 Temporary failure"
|
||||
COUNT=3
|
||||
;;
|
||||
4)
|
||||
BOUNCE_TYPE="Undetermined"
|
||||
BOUNCE_SUBTYPE=""
|
||||
DIAGNOSTIC="Unknown error occurred"
|
||||
COUNT=1
|
||||
;;
|
||||
*)
|
||||
echo "Invalid choice"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
echo
|
||||
echo "Simulating $COUNT bounce event(s)..."
|
||||
echo
|
||||
|
||||
# Insert bounce events
|
||||
for i in $(seq 1 $COUNT); do
|
||||
TIMESTAMP=$(date -u -d "-$((i * 24)) hours" '+%Y-%m-%d %H:%M:%S')
|
||||
FEEDBACK_ID="test-feedback-$(date +%s)-$i"
|
||||
|
||||
echo "Creating bounce event $i/$COUNT (timestamp: $TIMESTAMP)..."
|
||||
|
||||
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
|
||||
-- Insert bounce log
|
||||
INSERT INTO bounce_logs (member_id, email, bounce_type, bounce_subtype, diagnostic_code, timestamp, feedback_id)
|
||||
VALUES ($MEMBER_ID, '$MEMBER_EMAIL', '$BOUNCE_TYPE', '$BOUNCE_SUBTYPE', '$DIAGNOSTIC', '$TIMESTAMP', '$FEEDBACK_ID');
|
||||
EOF
|
||||
|
||||
# Update member bounce status
|
||||
if [ "$BOUNCE_TYPE" = "Permanent" ]; then
|
||||
echo " → Hard bounce: Deactivating member..."
|
||||
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
|
||||
UPDATE members
|
||||
SET bounce_count = bounce_count + 1,
|
||||
last_bounce_at = '$TIMESTAMP',
|
||||
bounce_status = 'hard_bounce',
|
||||
active = 0
|
||||
WHERE member_id = $MEMBER_ID;
|
||||
EOF
|
||||
elif [ "$BOUNCE_TYPE" = "Transient" ]; then
|
||||
# Check current bounce count
|
||||
CURRENT_COUNT=$(sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -N -e "SELECT bounce_count FROM members WHERE member_id = $MEMBER_ID;")
|
||||
NEW_COUNT=$((CURRENT_COUNT + 1))
|
||||
|
||||
if [ $NEW_COUNT -ge 3 ]; then
|
||||
echo " → Soft bounce threshold reached: Marking as soft_bounce..."
|
||||
BOUNCE_STATUS="soft_bounce"
|
||||
else
|
||||
echo " → Soft bounce: Incrementing counter ($NEW_COUNT)..."
|
||||
BOUNCE_STATUS="clean"
|
||||
fi
|
||||
|
||||
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
|
||||
UPDATE members
|
||||
SET bounce_count = $NEW_COUNT,
|
||||
last_bounce_at = '$TIMESTAMP',
|
||||
bounce_status = '$BOUNCE_STATUS'
|
||||
WHERE member_id = $MEMBER_ID;
|
||||
EOF
|
||||
else
|
||||
echo " → Undetermined bounce: Incrementing counter..."
|
||||
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist <<EOF
|
||||
UPDATE members
|
||||
SET bounce_count = bounce_count + 1,
|
||||
last_bounce_at = '$TIMESTAMP'
|
||||
WHERE member_id = $MEMBER_ID;
|
||||
EOF
|
||||
fi
|
||||
done
|
||||
|
||||
echo
|
||||
echo "✅ Bounce simulation complete!"
|
||||
echo
|
||||
|
||||
# Show updated member status
|
||||
echo "Updated member status:"
|
||||
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -e "
|
||||
SELECT
|
||||
member_id,
|
||||
name,
|
||||
email,
|
||||
active,
|
||||
bounce_count,
|
||||
last_bounce_at,
|
||||
bounce_status
|
||||
FROM members
|
||||
WHERE member_id = $MEMBER_ID\G
|
||||
"
|
||||
|
||||
echo
|
||||
echo "Bounce history for this member:"
|
||||
sudo docker-compose exec -T mysql mysql -u root -p"$DB_PASSWORD" maillist -e "
|
||||
SELECT
|
||||
bounce_id,
|
||||
bounce_type,
|
||||
bounce_subtype,
|
||||
diagnostic_code,
|
||||
timestamp
|
||||
FROM bounce_logs
|
||||
WHERE member_id = $MEMBER_ID
|
||||
ORDER BY timestamp DESC;
|
||||
"
|
||||
|
||||
echo
|
||||
echo "🎉 You can now view this bounce in the web UI:"
|
||||
echo " 1. Open http://localhost:3000"
|
||||
echo " 2. Go to the Members tab"
|
||||
echo " 3. Look for $MEMBER_NAME"
|
||||
echo " 4. Click the 'Bounces' button to see the history"
|
||||
echo
|
||||
echo "To simulate more bounces, run this script again!"
|
||||
46
test-cors-understanding.html
Normal file
46
test-cors-understanding.html
Normal file
@@ -0,0 +1,46 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<title>CORS Understanding Test</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>CORS Test</h1>
|
||||
<div id="results"></div>
|
||||
|
||||
<script>
|
||||
const results = document.getElementById('results');
|
||||
|
||||
// Test 1: Same-origin request (should work without CORS headers)
|
||||
console.log('Current origin:', window.location.origin);
|
||||
|
||||
// Test what your API client actually does
|
||||
const protocol = window.location.protocol;
|
||||
const hostname = window.location.hostname;
|
||||
|
||||
let baseURL;
|
||||
if (hostname === 'localhost' || hostname === '127.0.0.1') {
|
||||
baseURL = `${protocol}//${hostname}:8000`;
|
||||
results.innerHTML += `<p><strong>Development Mode Detected</strong></p>`;
|
||||
results.innerHTML += `<p>Frontend: ${window.location.origin}</p>`;
|
||||
results.innerHTML += `<p>API: ${baseURL}</p>`;
|
||||
results.innerHTML += `<p>This is a <strong>CROSS-ORIGIN</strong> request - CORS applies!</p>`;
|
||||
} else {
|
||||
baseURL = `${protocol}//${hostname}/api`;
|
||||
results.innerHTML += `<p><strong>Production Mode Detected</strong></p>`;
|
||||
results.innerHTML += `<p>Frontend: ${window.location.origin}</p>`;
|
||||
results.innerHTML += `<p>API: ${baseURL}</p>`;
|
||||
results.innerHTML += `<p>This is a <strong>SAME-ORIGIN</strong> request - NO CORS needed!</p>`;
|
||||
}
|
||||
|
||||
// Test the actual request
|
||||
fetch(`${baseURL}/health`)
|
||||
.then(response => response.json())
|
||||
.then(data => {
|
||||
results.innerHTML += `<p style="color: green;">✅ API Request Successful: ${JSON.stringify(data)}</p>`;
|
||||
})
|
||||
.catch(error => {
|
||||
results.innerHTML += `<p style="color: red;">❌ API Request Failed: ${error.message}</p>`;
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
45
test_bounce_quick.sh
Normal file
45
test_bounce_quick.sh
Normal file
@@ -0,0 +1,45 @@
|
||||
#!/bin/bash
|
||||
# Quick SES Bounce Test - One-liner to test bounce handling
|
||||
|
||||
echo "🚀 Testing SES Bounce Handling"
|
||||
echo "================================"
|
||||
echo ""
|
||||
|
||||
# Check if API and Postfix are running
|
||||
if ! sudo docker compose ps | grep -q "maillist-api.*Up"; then
|
||||
echo "❌ API container not running. Start with: sudo docker compose up -d"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! sudo docker compose ps | grep -q "maillist-postfix.*Up"; then
|
||||
echo "❌ Postfix container not running. Start with: sudo docker compose up -d"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✅ Containers are running"
|
||||
echo ""
|
||||
|
||||
# Start watching logs in background
|
||||
echo "📋 Opening log viewer (press Ctrl+C to stop)..."
|
||||
echo ""
|
||||
sleep 2
|
||||
|
||||
# Show the command they can run
|
||||
echo "Run this command to send a test bounce:"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "echo 'Test bounce' | sudo docker compose exec -T postfix mail -s 'Bounce Test' bounce@simulator.amazonses.com"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
echo "Then wait 30-60 seconds and check:"
|
||||
echo ""
|
||||
echo "1. API logs (in this window)"
|
||||
echo "2. Database: sudo docker compose exec mysql mysql -u maillist -pmaillist maillist -e 'SELECT * FROM bounce_logs ORDER BY created_at DESC LIMIT 5;'"
|
||||
echo "3. Web UI: http://localhost:3000 → Members tab"
|
||||
echo ""
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "Watching API logs now..."
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
|
||||
# Follow logs
|
||||
sudo docker compose logs api -f --tail 20
|
||||
76
test_bounce_webhook.sh
Executable file
76
test_bounce_webhook.sh
Executable file
@@ -0,0 +1,76 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Test SNS Bounce Webhook
|
||||
# This script simulates an SNS bounce notification (without signature verification for testing)
|
||||
|
||||
API_URL="http://localhost:8000"
|
||||
|
||||
echo "Testing SNS Bounce Webhook..."
|
||||
echo
|
||||
|
||||
# Test 1: Health check
|
||||
echo "1. Testing API health..."
|
||||
curl -s "$API_URL/health" | jq .
|
||||
echo
|
||||
|
||||
# Test 2: Create a test subscription confirmation (the webhook will auto-confirm)
|
||||
echo "2. Testing subscription confirmation handling..."
|
||||
curl -s -X POST "$API_URL/webhooks/sns" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"Type": "SubscriptionConfirmation",
|
||||
"MessageId": "test-message-id",
|
||||
"Token": "test-token",
|
||||
"TopicArn": "arn:aws:sns:eu-west-2:123456789:test-topic",
|
||||
"Message": "You have chosen to subscribe to the topic",
|
||||
"SubscribeURL": "https://example.com/subscribe",
|
||||
"Timestamp": "2025-01-01T12:00:00.000Z"
|
||||
}' || echo "Note: Signature verification will fail for test messages (expected)"
|
||||
echo
|
||||
echo
|
||||
|
||||
# Test 3: Simulated bounce notification (will fail signature verification)
|
||||
echo "3. Testing bounce notification structure..."
|
||||
echo "Note: In production, AWS SNS will send properly signed messages."
|
||||
echo "This test demonstrates the expected structure."
|
||||
echo
|
||||
|
||||
cat << 'EOF' > /tmp/test_bounce.json
|
||||
{
|
||||
"Type": "Notification",
|
||||
"MessageId": "test-notification-id",
|
||||
"TopicArn": "arn:aws:sns:eu-west-2:123456789:test-topic",
|
||||
"Subject": "Amazon SES Email Event Notification",
|
||||
"Message": "{\"notificationType\":\"Bounce\",\"bounce\":{\"bounceType\":\"Permanent\",\"bounceSubType\":\"General\",\"bouncedRecipients\":[{\"emailAddress\":\"bounce@simulator.amazonses.com\",\"diagnosticCode\":\"smtp; 550 5.1.1 user unknown\"}],\"timestamp\":\"2025-01-01T12:00:00.000Z\",\"feedbackId\":\"test-feedback-id\"}}",
|
||||
"Timestamp": "2025-01-01T12:00:00.000Z",
|
||||
"SignatureVersion": "1",
|
||||
"Signature": "test-signature",
|
||||
"SigningCertURL": "https://sns.eu-west-2.amazonaws.com/test.pem"
|
||||
}
|
||||
EOF
|
||||
|
||||
cat /tmp/test_bounce.json | jq .
|
||||
echo
|
||||
echo "Expected behavior: Signature verification will fail without real AWS credentials"
|
||||
echo "In production, AWS SNS will send properly signed messages that will be verified"
|
||||
echo
|
||||
|
||||
# Test 4: Check if database schema has bounce tables
|
||||
echo "4. Checking database schema for bounce tables..."
|
||||
sudo docker-compose exec -T mysql mysql -u maillist -pmaillist maillist -e "SHOW TABLES LIKE '%bounce%';" 2>/dev/null
|
||||
echo
|
||||
|
||||
# Test 5: Check members table for bounce columns
|
||||
echo "5. Checking members table for bounce columns..."
|
||||
sudo docker-compose exec -T mysql mysql -u maillist -pmaillist maillist -e "DESCRIBE members;" 2>/dev/null | grep -i bounce
|
||||
echo
|
||||
|
||||
echo "✓ Setup complete!"
|
||||
echo
|
||||
echo "To test with real AWS SNS:"
|
||||
echo "1. Set up SNS topic in AWS Console"
|
||||
echo "2. Subscribe webhook: https://your-domain.com:8000/webhooks/sns"
|
||||
echo "3. Configure SES to send bounce notifications to the SNS topic"
|
||||
echo "4. Send test email to bounce@simulator.amazonses.com"
|
||||
echo
|
||||
echo "See BOUNCE_HANDLING_SETUP.md for detailed setup instructions"
|
||||
@@ -38,23 +38,29 @@ web/
|
||||
|
||||
## Usage
|
||||
|
||||
### Development
|
||||
1. Ensure the API is running on port 8000
|
||||
2. Open `index.html` in a web browser
|
||||
3. Enter your API token to authenticate
|
||||
4. Start managing your mailing lists!
|
||||
|
||||
### Production (Docker)
|
||||
The web frontend is served using Nginx and included in the Docker Compose setup.
|
||||
### Accessing the Interface
|
||||
|
||||
**Via Docker (Recommended):**
|
||||
```bash
|
||||
# Build and start all services
|
||||
docker-compose up --build
|
||||
# Ensure all services are running
|
||||
sudo docker-compose up -d
|
||||
|
||||
# Access the web interface
|
||||
open http://localhost:3000
|
||||
```
|
||||
|
||||
**For Development:**
|
||||
You can also open `index.html` directly in a browser, but the API must be running on port 8000.
|
||||
|
||||
### Getting Started
|
||||
|
||||
1. **Authenticate**: Enter your API_TOKEN (from `.env` file) when prompted
|
||||
2. **Manage Lists**: Click the "Lists" tab to view and manage mailing lists
|
||||
3. **Manage Members**: Click the "Members" tab to add and edit member information
|
||||
4. **Manage Subscriptions**: Click "Subscriptions" button on any member to toggle their list memberships
|
||||
|
||||
The interface saves your token in browser storage, so you won't need to re-enter it on subsequent visits.
|
||||
|
||||
## Features Overview
|
||||
|
||||
### Authentication
|
||||
@@ -65,24 +71,39 @@ open http://localhost:3000
|
||||
### Subscription Management (New & Improved!)
|
||||
|
||||
#### Member-Centric Subscription Management
|
||||
The subscription management has been completely overhauled for a much better user experience:
|
||||
The subscription management has been completely redesigned for the best user experience:
|
||||
|
||||
1. **Access Member Subscriptions**: In the Members tab, click the "Subscriptions" button next to any member
|
||||
2. **Visual Toggle Interface**: See all available mailing lists with modern toggle switches
|
||||
3. **Intuitive Controls**:
|
||||
- Green toggle = Member is subscribed
|
||||
- Gray toggle = Member is not subscribed
|
||||
- Click anywhere on a list item to toggle subscription
|
||||
4. **Batch Operations**: Make multiple changes and save them all at once
|
||||
5. **Real-time Feedback**: The save button shows how many changes you've made
|
||||
**How to Use:**
|
||||
1. Navigate to the **Members** tab
|
||||
2. Find the member you want to manage
|
||||
3. Click the **"Subscriptions"** button next to their name
|
||||
4. A modal appears showing all available mailing lists with toggle switches
|
||||
|
||||
#### Benefits Over Previous System
|
||||
- ✅ **Much faster** - No need to add subscriptions one by one
|
||||
**Visual Interface:**
|
||||
- **Green toggle** = Member is subscribed to this list
|
||||
- **Gray toggle** = Member is not subscribed to this list
|
||||
- **Click anywhere** on a list item to toggle subscription on/off
|
||||
|
||||
**Batch Operations:**
|
||||
- Make multiple changes (subscribe to some lists, unsubscribe from others)
|
||||
- The "Save Changes" button shows how many changes you've made (e.g., "Save 3 Changes")
|
||||
- All changes are saved together when you click "Save Changes"
|
||||
- Click "Cancel" to discard all changes
|
||||
|
||||
**Benefits:**
|
||||
- ✅ **Fast** - Toggle multiple subscriptions at once
|
||||
- ✅ **Visual** - See all subscriptions at a glance with color coding
|
||||
- ✅ **Intuitive** - Toggle switches work like modern mobile apps
|
||||
- ✅ **Batch operations** - Change multiple subscriptions simultaneously
|
||||
- ✅ **Less error-prone** - Clear visual feedback prevents mistakes
|
||||
- ✅ **Change tracking** - Only saves actual changes, not unchanged items
|
||||
- ✅ **Intuitive** - Works like modern mobile app switches
|
||||
- ✅ **Smart** - Only saves actual changes, not unchanged items
|
||||
- ✅ **Clear** - Shows exactly how many changes you're about to save
|
||||
|
||||
#### Alternative: Legacy Bulk Subscription
|
||||
For power users who prefer the old approach:
|
||||
1. Go to the **Subscriptions** tab
|
||||
2. Select list and member from dropdowns
|
||||
3. Click "Add Subscription"
|
||||
|
||||
This is still available but the member-centric approach is much more efficient for managing multiple subscriptions.
|
||||
|
||||
### Mailing Lists
|
||||
- View all mailing lists in a clean table
|
||||
|
||||
460
web/index.html
460
web/index.html
@@ -9,26 +9,92 @@
|
||||
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css">
|
||||
</head>
|
||||
<body>
|
||||
<!-- Header -->
|
||||
<header class="header">
|
||||
<div class="container">
|
||||
<div class="header-content">
|
||||
<h1 class="logo">
|
||||
<!-- Login Page -->
|
||||
<div class="login-page" id="loginPage">
|
||||
<div class="login-container">
|
||||
<div class="login-header">
|
||||
<div class="login-logo">
|
||||
<i class="fas fa-envelope"></i>
|
||||
Mailing List Manager
|
||||
</h1>
|
||||
<div class="auth-section">
|
||||
<div class="auth-controls" id="authControls">
|
||||
<input type="password" id="apiToken" placeholder="Enter API Token" class="token-input">
|
||||
<button class="btn btn-primary" id="loginBtn">Login</button>
|
||||
</div>
|
||||
<h1>Mailing List Manager</h1>
|
||||
<p>Sign in to manage your mailing lists</p>
|
||||
</div>
|
||||
<form class="login-form" id="loginForm">
|
||||
<!-- Login Error Message -->
|
||||
<div class="login-error" id="loginError" style="display: none;">
|
||||
<i class="fas fa-exclamation-circle"></i>
|
||||
<span id="loginErrorMessage">Invalid username or password</span>
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="username">
|
||||
<i class="fas fa-user"></i>
|
||||
Username
|
||||
</label>
|
||||
<input type="text" id="username" placeholder="Enter your username" required autocomplete="username">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="password">
|
||||
<i class="fas fa-lock"></i>
|
||||
Password
|
||||
</label>
|
||||
<input type="password" id="password" placeholder="Enter your password" required autocomplete="current-password">
|
||||
</div>
|
||||
<button type="submit" class="btn btn-primary btn-block" id="loginBtn">
|
||||
<span>Sign In</span>
|
||||
<i class="fas fa-arrow-right"></i>
|
||||
</button>
|
||||
</form>
|
||||
<div class="login-footer">
|
||||
<p class="text-muted text-sm">
|
||||
<i class="fas fa-shield-alt"></i>
|
||||
Secure authentication required
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Menu Bar Header (shown after login) -->
|
||||
<header class="menu-bar" id="mainHeader" style="display: none;">
|
||||
<div class="menu-bar-content">
|
||||
<div class="app-title">
|
||||
<i class="fas fa-envelope"></i>
|
||||
<span>Mailing List Manager</span>
|
||||
</div>
|
||||
<div class="menu-spacer"></div>
|
||||
<div class="user-dropdown" id="userDropdown">
|
||||
<button class="user-dropdown-trigger" id="userDropdownTrigger">
|
||||
<div class="user-avatar">
|
||||
<i class="fas fa-user"></i>
|
||||
</div>
|
||||
<div class="user-info" id="userInfo" style="display: none;">
|
||||
<span class="status-indicator">
|
||||
<i class="fas fa-check-circle"></i>
|
||||
Connected
|
||||
</span>
|
||||
<button class="btn btn-secondary" id="logoutBtn">Logout</button>
|
||||
<div class="user-details">
|
||||
<span class="user-name" id="currentUsername">User</span>
|
||||
<span class="user-role" id="currentUserRole">role</span>
|
||||
</div>
|
||||
<i class="fas fa-chevron-down dropdown-arrow"></i>
|
||||
</button>
|
||||
<div class="user-dropdown-menu" id="userDropdownMenu">
|
||||
<div class="dropdown-header">
|
||||
<div class="dropdown-user-info">
|
||||
<div class="dropdown-avatar">
|
||||
<i class="fas fa-user"></i>
|
||||
</div>
|
||||
<div class="dropdown-details">
|
||||
<div class="dropdown-name" id="dropdownUsername">User</div>
|
||||
<div class="dropdown-role" id="dropdownUserRole">role</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="dropdown-divider"></div>
|
||||
<button class="dropdown-item" id="userManagementBtn" style="display: none;">
|
||||
<i class="fas fa-user-shield"></i>
|
||||
<span>User Management</span>
|
||||
</button>
|
||||
<div class="dropdown-divider" id="userManagementDivider" style="display: none;"></div>
|
||||
<button class="dropdown-item" id="logoutBtn">
|
||||
<i class="fas fa-sign-out-alt"></i>
|
||||
<span>Sign Out</span>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -38,7 +104,7 @@
|
||||
<main class="main-content" id="mainContent" style="display: none;">
|
||||
<div class="container">
|
||||
<!-- Navigation Tabs -->
|
||||
<nav class="tab-nav">
|
||||
<nav class="tab-nav" id="tabNav">
|
||||
<button class="tab-btn active" data-tab="lists">
|
||||
<i class="fas fa-list"></i>
|
||||
Mailing Lists
|
||||
@@ -89,17 +155,37 @@
|
||||
<!-- Members Tab -->
|
||||
<div class="tab-content" id="members-tab">
|
||||
<div class="section-header">
|
||||
<div class="header-content">
|
||||
<h2>Members</h2>
|
||||
<div class="header-help">
|
||||
<i class="fas fa-info-circle"></i>
|
||||
<span>Click the "Lists" button next to any member to manage their subscriptions</span>
|
||||
<h2>Members</h2>
|
||||
<div class="button-group">
|
||||
<button class="btn btn-primary" id="addMemberBtn">
|
||||
<i class="fas fa-plus"></i>
|
||||
Add Member
|
||||
</button>
|
||||
<button class="btn btn-secondary" id="showBulkImportBtn">
|
||||
<i class="fas fa-upload"></i>
|
||||
Bulk Import CSV
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Member Search -->
|
||||
<div class="search-section">
|
||||
<div class="search-container">
|
||||
<div class="search-input-wrapper">
|
||||
<i class="fas fa-search search-icon"></i>
|
||||
<input type="text"
|
||||
id="memberSearchInput"
|
||||
class="search-input"
|
||||
placeholder="Search members by name or email..."
|
||||
autocomplete="off">
|
||||
<button class="search-clear" id="memberSearchClear" style="display: none;">
|
||||
<i class="fas fa-times"></i>
|
||||
</button>
|
||||
</div>
|
||||
<div class="search-results-info" id="memberSearchInfo" style="display: none;">
|
||||
<span id="memberSearchCount">0</span> members found
|
||||
</div>
|
||||
</div>
|
||||
<button class="btn btn-primary" id="addMemberBtn">
|
||||
<i class="fas fa-plus"></i>
|
||||
Add Member
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div class="data-table">
|
||||
@@ -120,6 +206,35 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Users Tab (Admin Only) -->
|
||||
<div class="tab-content" id="users-tab">
|
||||
<div class="section-header">
|
||||
<h2>User Management</h2>
|
||||
<button class="btn btn-primary" id="addUserBtn">
|
||||
<i class="fas fa-plus"></i>
|
||||
Add User
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div class="data-table">
|
||||
<table class="table" id="usersTable">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Username</th>
|
||||
<th>Role</th>
|
||||
<th>Created</th>
|
||||
<th>Last Login</th>
|
||||
<th>Status</th>
|
||||
<th>Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="usersTableBody">
|
||||
<!-- Dynamic content -->
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
</div>
|
||||
</main>
|
||||
@@ -273,6 +388,295 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Bulk Import Modal -->
|
||||
<div class="modal modal-large" id="bulkImportModal">
|
||||
<div class="modal-content">
|
||||
<div class="modal-header">
|
||||
<h3>Bulk Import Members from CSV</h3>
|
||||
<button class="modal-close" id="bulkImportModalClose">
|
||||
<i class="fas fa-times"></i>
|
||||
</button>
|
||||
</div>
|
||||
<div class="modal-body">
|
||||
<!-- Step 1: File Upload -->
|
||||
<div class="import-step" id="importStep1">
|
||||
<h4><i class="fas fa-upload"></i> Step 1: Upload CSV File</h4>
|
||||
<p class="text-muted">Upload a CSV file containing member information. The file must have 'Name' and 'Email' columns.</p>
|
||||
|
||||
<div class="file-upload-area" id="fileUploadArea">
|
||||
<div class="file-upload-content">
|
||||
<i class="fas fa-cloud-upload-alt"></i>
|
||||
<p>Click to select a CSV file or drag and drop</p>
|
||||
<input type="file" id="csvFileInput" accept=".csv" style="display: none;">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="file-info" id="fileInfo" style="display: none;">
|
||||
<div class="file-details">
|
||||
<i class="fas fa-file-csv"></i>
|
||||
<span id="fileName">file.csv</span>
|
||||
<span id="fileSize" class="text-muted">(0 KB)</span>
|
||||
</div>
|
||||
<button type="button" class="btn btn-sm btn-secondary" id="removeFileBtn">
|
||||
<i class="fas fa-times"></i> Remove
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div class="csv-format-help">
|
||||
<h5><i class="fas fa-info-circle"></i> CSV Format</h5>
|
||||
<p>Your CSV file should look like this:</p>
|
||||
<div class="code-example">
|
||||
<code>Name,Email,Delivery,Member since<br>
|
||||
Ahmed Ajzan,ahmedajzan@doctors.org.uk,,2025-04-06T18:44:26.819454<br>
|
||||
Alan Bailey,abgower@icloud.com,,2025-04-06T18:44:26.824446</code>
|
||||
</div>
|
||||
<p class="text-muted text-sm">Only the 'Name' and 'Email' columns are required. Additional columns will be ignored.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Step 2: Preview -->
|
||||
<div class="import-step" id="importStep2" style="display: none;">
|
||||
<h4><i class="fas fa-eye"></i> Step 2: Preview Data</h4>
|
||||
<p class="text-muted">Review the data that will be imported:</p>
|
||||
|
||||
<div class="preview-stats" id="previewStats">
|
||||
<div class="stat-item">
|
||||
<span class="stat-value" id="totalRowsCount">0</span>
|
||||
<span class="stat-label">Total Rows</span>
|
||||
</div>
|
||||
<div class="stat-item">
|
||||
<span class="stat-value" id="validRowsCount">0</span>
|
||||
<span class="stat-label">Valid Rows</span>
|
||||
</div>
|
||||
<div class="stat-item">
|
||||
<span class="stat-value" id="errorRowsCount">0</span>
|
||||
<span class="stat-label">Errors</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="preview-table-container" id="previewTableContainer">
|
||||
<table class="table table-sm" id="previewTable">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Row</th>
|
||||
<th>Name</th>
|
||||
<th>Email</th>
|
||||
<th>Status</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="previewTableBody">
|
||||
<!-- Dynamic content -->
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<div class="error-list" id="errorList" style="display: none;">
|
||||
<div class="error-header">
|
||||
<h5><i class="fas fa-exclamation-triangle"></i> Errors Found</h5>
|
||||
<button type="button" class="btn btn-sm btn-secondary" id="toggleErrorsBtn">
|
||||
<span id="errorToggleText">Show All</span>
|
||||
<i class="fas fa-chevron-down" id="errorToggleIcon"></i>
|
||||
</button>
|
||||
</div>
|
||||
<div class="error-summary" id="errorSummary">
|
||||
<!-- Summary will be shown here -->
|
||||
</div>
|
||||
<div class="error-details" id="errorDetails" style="display: none;">
|
||||
<div class="error-table-container">
|
||||
<table class="table table-sm" id="errorTable">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Row</th>
|
||||
<th>Issue</th>
|
||||
<th>Name</th>
|
||||
<th>Email</th>
|
||||
<th>Raw Data</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="errorTableBody">
|
||||
<!-- Dynamic content -->
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Step 3: List Selection -->
|
||||
<div class="import-step" id="importStep3" style="display: none;">
|
||||
<h4><i class="fas fa-list"></i> Step 3: Select Mailing Lists</h4>
|
||||
<p class="text-muted">Choose which mailing lists these members should be subscribed to:</p>
|
||||
|
||||
<div class="list-selection" id="listSelection">
|
||||
<!-- Dynamic content -->
|
||||
</div>
|
||||
|
||||
<div class="selection-info">
|
||||
<p class="text-muted text-sm">
|
||||
<i class="fas fa-info-circle"></i>
|
||||
If a member already exists, they will be added to the selected lists (existing subscriptions will be preserved).
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Step 4: Import Results -->
|
||||
<div class="import-step" id="importStep4" style="display: none;">
|
||||
<h4><i class="fas fa-check-circle"></i> Step 4: Import Complete</h4>
|
||||
|
||||
<div class="import-results" id="importResults">
|
||||
<div class="result-stats">
|
||||
<div class="stat-item">
|
||||
<span class="stat-value" id="processedCount">0</span>
|
||||
<span class="stat-label">Processed</span>
|
||||
</div>
|
||||
<div class="stat-item">
|
||||
<span class="stat-value" id="createdCount">0</span>
|
||||
<span class="stat-label">Created</span>
|
||||
</div>
|
||||
<div class="stat-item">
|
||||
<span class="stat-value" id="updatedCount">0</span>
|
||||
<span class="stat-label">Updated</span>
|
||||
</div>
|
||||
<div class="stat-item">
|
||||
<span class="stat-value" id="subscriptionsCount">0</span>
|
||||
<span class="stat-label">Subscriptions</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="result-errors" id="resultErrors" style="display: none;">
|
||||
<h5><i class="fas fa-exclamation-triangle"></i> Import Errors</h5>
|
||||
<ul id="resultErrorList">
|
||||
<!-- Dynamic content -->
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="modal-actions">
|
||||
<button type="button" class="btn btn-secondary" id="bulkImportBackBtn" style="display: none;">
|
||||
<i class="fas fa-arrow-left"></i> Back
|
||||
</button>
|
||||
<button type="button" class="btn btn-secondary" id="bulkImportCancelBtn">Cancel</button>
|
||||
<button type="button" class="btn btn-primary" id="bulkImportNextBtn" disabled>
|
||||
Next <i class="fas fa-arrow-right"></i>
|
||||
</button>
|
||||
<button type="button" class="btn btn-primary" id="bulkImportBtn" style="display: none;">
|
||||
<i class="fas fa-upload"></i> Import Members
|
||||
</button>
|
||||
<button type="button" class="btn btn-success" id="bulkImportDoneBtn" style="display: none;">
|
||||
<i class="fas fa-check"></i> Done
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Add/Edit User Modal -->
|
||||
<div class="modal" id="userModal">
|
||||
<div class="modal-content">
|
||||
<div class="modal-header">
|
||||
<h3 id="userModalTitle">Add User</h3>
|
||||
<button class="modal-close" id="userModalClose">
|
||||
<i class="fas fa-times"></i>
|
||||
</button>
|
||||
</div>
|
||||
<form id="userForm">
|
||||
<div class="form-group">
|
||||
<label for="userName">Username *</label>
|
||||
<input type="text" id="userName" name="userName" required>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="userPassword">Password *</label>
|
||||
<input type="password" id="userPassword" name="userPassword" required>
|
||||
<div class="form-help" id="passwordHelp">
|
||||
Leave blank to keep current password (when editing)
|
||||
</div>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="userRole">Role *</label>
|
||||
<select id="userRole" name="userRole" required>
|
||||
<option value="">Select a role...</option>
|
||||
<option value="administrator">Administrator - Full access including user management</option>
|
||||
<option value="operator">Operator - Read/write access to lists and members</option>
|
||||
<option value="read-only">Read-Only - View-only access</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label class="checkbox-label">
|
||||
<input type="checkbox" id="userActive" name="userActive" checked>
|
||||
<span class="checkmark"></span>
|
||||
Active
|
||||
</label>
|
||||
</div>
|
||||
<div class="modal-actions">
|
||||
<button type="button" class="btn btn-secondary" id="userCancelBtn">Cancel</button>
|
||||
<button type="submit" class="btn btn-primary" id="userSubmitBtn">Save</button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Bounce History Modal -->
|
||||
<div class="modal" id="bounceHistoryModal">
|
||||
<div class="modal-content">
|
||||
<div class="modal-header">
|
||||
<h3 id="bounceHistoryTitle">Bounce History</h3>
|
||||
<button class="modal-close" id="bounceHistoryModalClose">
|
||||
<i class="fas fa-times"></i>
|
||||
</button>
|
||||
</div>
|
||||
<div class="modal-body">
|
||||
<div class="member-info-banner" id="bounceHistoryMemberInfo">
|
||||
<div class="member-avatar">
|
||||
<i class="fas fa-user"></i>
|
||||
</div>
|
||||
<div class="member-details">
|
||||
<h4 id="bounceHistoryMemberName">Member Name</h4>
|
||||
<p id="bounceHistoryMemberEmail">member@example.com</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="bounce-summary">
|
||||
<div class="bounce-stat">
|
||||
<i class="fas fa-exclamation-triangle"></i>
|
||||
<div>
|
||||
<div class="bounce-stat-label">Total Bounces</div>
|
||||
<div class="bounce-stat-value" id="bounceTotalCount">0</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="bounce-stat">
|
||||
<i class="fas fa-clock"></i>
|
||||
<div>
|
||||
<div class="bounce-stat-label">Last Bounce</div>
|
||||
<div class="bounce-stat-value" id="bounceLastDate">Never</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="bounce-stat">
|
||||
<i class="fas fa-info-circle"></i>
|
||||
<div>
|
||||
<div class="bounce-stat-label">Status</div>
|
||||
<div class="bounce-stat-value" id="bounceStatusText">Clean</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="bounce-history-section">
|
||||
<h5>Bounce Events</h5>
|
||||
<div class="bounce-history-list" id="bounceHistoryList">
|
||||
<!-- Dynamic content will be inserted here -->
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="modal-actions">
|
||||
<button type="button" class="btn btn-secondary" id="bounceHistoryCloseBtn">Close</button>
|
||||
<button type="button" class="btn btn-warning" id="bounceHistoryResetBtn" data-requires-write>
|
||||
<i class="fas fa-redo"></i>
|
||||
Reset Bounce Status
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Confirmation Modal -->
|
||||
<div class="modal" id="confirmModal">
|
||||
<div class="modal-content">
|
||||
@@ -293,7 +697,7 @@
|
||||
</div>
|
||||
|
||||
<script src="static/js/api.js"></script>
|
||||
<script src="static/js/ui.js"></script>
|
||||
<script src="static/js/app.js"></script>
|
||||
<script src="static/js/ui.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because it is too large
Load Diff
@@ -25,8 +25,8 @@ class APIClient {
|
||||
return `${protocol}//${hostname}:8000`;
|
||||
}
|
||||
|
||||
// If running in production, assume API is on port 8000
|
||||
return `${protocol}//${hostname}:8000`;
|
||||
// If running in production behind a reverse proxy, use /api path
|
||||
return `${protocol}//${hostname}/api`;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -50,9 +50,14 @@ class APIClient {
|
||||
*/
|
||||
async request(endpoint, options = {}) {
|
||||
const url = `${this.baseURL}${endpoint}`;
|
||||
|
||||
// Merge options, with custom headers taking precedence
|
||||
const config = {
|
||||
headers: { ...this.headers },
|
||||
...options
|
||||
...options,
|
||||
headers: {
|
||||
...this.headers,
|
||||
...(options.headers || {})
|
||||
}
|
||||
};
|
||||
|
||||
try {
|
||||
@@ -100,10 +105,48 @@ class APIClient {
|
||||
return this.request('/health');
|
||||
}
|
||||
|
||||
async getConfig() {
|
||||
return this.request('/config');
|
||||
}
|
||||
|
||||
async testAuth() {
|
||||
return this.request('/');
|
||||
}
|
||||
|
||||
async login(username, password) {
|
||||
// Don't include Authorization header for login
|
||||
const tempHeaders = { ...this.headers };
|
||||
delete tempHeaders['Authorization'];
|
||||
|
||||
const response = await this.request('/auth/login', {
|
||||
method: 'POST',
|
||||
headers: tempHeaders,
|
||||
body: JSON.stringify({ username, password })
|
||||
});
|
||||
|
||||
// Set the token from the response
|
||||
if (response.access_token) {
|
||||
this.setToken(response.access_token);
|
||||
}
|
||||
|
||||
return response;
|
||||
}
|
||||
|
||||
async logout() {
|
||||
try {
|
||||
await this.request('/auth/logout', {
|
||||
method: 'POST'
|
||||
});
|
||||
} finally {
|
||||
// Clear token even if logout fails
|
||||
this.clearToken();
|
||||
}
|
||||
}
|
||||
|
||||
async getCurrentUser() {
|
||||
return this.request('/auth/me');
|
||||
}
|
||||
|
||||
// Mailing Lists API
|
||||
async getLists() {
|
||||
return this.request('/lists');
|
||||
@@ -184,6 +227,55 @@ class APIClient {
|
||||
method: 'DELETE'
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Bulk import members from CSV data
|
||||
*/
|
||||
async bulkImportMembers(csvData, listIds) {
|
||||
return this.request('/bulk-import', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({
|
||||
csv_data: csvData,
|
||||
list_ids: listIds
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
// User Management API
|
||||
async getUsers() {
|
||||
return this.request('/users');
|
||||
}
|
||||
|
||||
async createUser(userData) {
|
||||
return this.request('/users', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify(userData)
|
||||
});
|
||||
}
|
||||
|
||||
async updateUser(userId, userData) {
|
||||
return this.request(`/users/${userId}`, {
|
||||
method: 'PATCH',
|
||||
body: JSON.stringify(userData)
|
||||
});
|
||||
}
|
||||
|
||||
async deleteUser(userId) {
|
||||
return this.request(`/users/${userId}`, {
|
||||
method: 'DELETE'
|
||||
});
|
||||
}
|
||||
|
||||
// Bounce Management API
|
||||
async getMemberBounces(memberId) {
|
||||
return this.request(`/members/${memberId}/bounces`);
|
||||
}
|
||||
|
||||
async resetBounceStatus(memberId) {
|
||||
return this.request(`/members/${memberId}/bounce-status`, {
|
||||
method: 'PATCH'
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -6,8 +6,10 @@
|
||||
class MailingListApp {
|
||||
constructor() {
|
||||
this.isAuthenticated = false;
|
||||
this.currentUser = null;
|
||||
this.lists = [];
|
||||
this.members = [];
|
||||
this.users = [];
|
||||
this.subscriptions = new Map(); // list_id -> members[]
|
||||
|
||||
this.initializeApp();
|
||||
@@ -20,9 +22,10 @@ class MailingListApp {
|
||||
this.setupEventListeners();
|
||||
|
||||
// Check for saved token
|
||||
const savedToken = localStorage.getItem('apiToken');
|
||||
const savedToken = localStorage.getItem('authToken');
|
||||
if (savedToken) {
|
||||
await this.login(savedToken, false);
|
||||
apiClient.setToken(savedToken);
|
||||
await this.checkCurrentUser();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -30,69 +33,215 @@ class MailingListApp {
|
||||
* Setup event listeners
|
||||
*/
|
||||
setupEventListeners() {
|
||||
// Login/logout
|
||||
document.getElementById('loginBtn').addEventListener('click', () => {
|
||||
// Login form submission
|
||||
document.getElementById('loginForm').addEventListener('submit', (e) => {
|
||||
e.preventDefault();
|
||||
this.handleLogin();
|
||||
});
|
||||
|
||||
// Clear error when user starts typing
|
||||
document.getElementById('username').addEventListener('input', () => {
|
||||
this.clearLoginError();
|
||||
});
|
||||
|
||||
document.getElementById('password').addEventListener('input', () => {
|
||||
this.clearLoginError();
|
||||
});
|
||||
|
||||
document.getElementById('logoutBtn').addEventListener('click', () => {
|
||||
this.logout();
|
||||
});
|
||||
|
||||
// Enter key in token input
|
||||
document.getElementById('apiToken').addEventListener('keypress', (e) => {
|
||||
if (e.key === 'Enter') {
|
||||
this.handleLogin();
|
||||
// User dropdown functionality
|
||||
const userDropdownTrigger = document.getElementById('userDropdownTrigger');
|
||||
const userDropdown = document.getElementById('userDropdown');
|
||||
|
||||
userDropdownTrigger.addEventListener('click', (e) => {
|
||||
e.preventDefault();
|
||||
e.stopPropagation();
|
||||
userDropdown.classList.toggle('active');
|
||||
userDropdownTrigger.classList.toggle('active');
|
||||
});
|
||||
|
||||
// Close dropdown when clicking outside
|
||||
document.addEventListener('click', (e) => {
|
||||
if (!userDropdown.contains(e.target)) {
|
||||
userDropdown.classList.remove('active');
|
||||
userDropdownTrigger.classList.remove('active');
|
||||
}
|
||||
});
|
||||
|
||||
// Close dropdown on escape key
|
||||
document.addEventListener('keydown', (e) => {
|
||||
if (e.key === 'Escape') {
|
||||
userDropdown.classList.remove('active');
|
||||
userDropdownTrigger.classList.remove('active');
|
||||
}
|
||||
});
|
||||
|
||||
// Bulk import button
|
||||
document.getElementById('showBulkImportBtn').addEventListener('click', () => {
|
||||
uiManager.showBulkImportModal();
|
||||
});
|
||||
|
||||
// Add user button (admin only)
|
||||
document.getElementById('addUserBtn').addEventListener('click', () => {
|
||||
uiManager.showUserModal();
|
||||
});
|
||||
|
||||
// User management dropdown item
|
||||
document.getElementById('userManagementBtn').addEventListener('click', () => {
|
||||
// Close the dropdown
|
||||
userDropdown.classList.remove('active');
|
||||
userDropdownTrigger.classList.remove('active');
|
||||
// Switch to users tab
|
||||
this.switchToUsersTab();
|
||||
});
|
||||
|
||||
// Member search functionality
|
||||
const memberSearchInput = document.getElementById('memberSearchInput');
|
||||
const memberSearchClear = document.getElementById('memberSearchClear');
|
||||
|
||||
memberSearchInput.addEventListener('input', (e) => {
|
||||
this.filterMembers(e.target.value);
|
||||
|
||||
// Show/hide clear button
|
||||
if (e.target.value.trim()) {
|
||||
memberSearchClear.style.display = 'flex';
|
||||
} else {
|
||||
memberSearchClear.style.display = 'none';
|
||||
}
|
||||
});
|
||||
|
||||
memberSearchClear.addEventListener('click', () => {
|
||||
memberSearchInput.value = '';
|
||||
memberSearchClear.style.display = 'none';
|
||||
this.filterMembers('');
|
||||
memberSearchInput.focus();
|
||||
});
|
||||
|
||||
// Clear search when switching tabs
|
||||
document.querySelectorAll('.tab-btn').forEach(btn => {
|
||||
btn.addEventListener('click', () => {
|
||||
if (btn.dataset.tab !== 'members') {
|
||||
memberSearchInput.value = '';
|
||||
memberSearchClear.style.display = 'none';
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if current user is still valid
|
||||
*/
|
||||
async checkCurrentUser() {
|
||||
try {
|
||||
const user = await apiClient.getCurrentUser();
|
||||
this.currentUser = user;
|
||||
this.isAuthenticated = true;
|
||||
this.showAuthenticatedUI();
|
||||
await this.loadData();
|
||||
} catch (error) {
|
||||
// Token is invalid, clear it
|
||||
this.logout();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle login button click
|
||||
*/
|
||||
async handleLogin() {
|
||||
const tokenInput = document.getElementById('apiToken');
|
||||
const token = tokenInput.value.trim();
|
||||
const usernameInput = document.getElementById('username');
|
||||
const passwordInput = document.getElementById('password');
|
||||
const username = usernameInput.value.trim();
|
||||
const password = passwordInput.value.trim();
|
||||
|
||||
if (!token) {
|
||||
uiManager.showNotification('Please enter an API token', 'error');
|
||||
// Clear previous error states
|
||||
this.clearLoginError();
|
||||
|
||||
if (!username || !password) {
|
||||
this.showLoginError('Please enter both username and password');
|
||||
return;
|
||||
}
|
||||
|
||||
await this.login(token, true);
|
||||
await this.login(username, password);
|
||||
}
|
||||
|
||||
/**
|
||||
* Show login error message
|
||||
*/
|
||||
showLoginError(message) {
|
||||
const errorDiv = document.getElementById('loginError');
|
||||
const errorMessage = document.getElementById('loginErrorMessage');
|
||||
const usernameInput = document.getElementById('username');
|
||||
const passwordInput = document.getElementById('password');
|
||||
|
||||
errorMessage.textContent = message;
|
||||
errorDiv.style.display = 'flex';
|
||||
|
||||
// Add error class to inputs
|
||||
usernameInput.classList.add('error');
|
||||
passwordInput.classList.add('error');
|
||||
|
||||
// Remove animation class and re-add to trigger animation
|
||||
errorDiv.style.animation = 'none';
|
||||
setTimeout(() => {
|
||||
errorDiv.style.animation = '';
|
||||
}, 10);
|
||||
}
|
||||
|
||||
/**
|
||||
* Clear login error message
|
||||
*/
|
||||
clearLoginError() {
|
||||
const errorDiv = document.getElementById('loginError');
|
||||
const usernameInput = document.getElementById('username');
|
||||
const passwordInput = document.getElementById('password');
|
||||
|
||||
errorDiv.style.display = 'none';
|
||||
usernameInput.classList.remove('error');
|
||||
passwordInput.classList.remove('error');
|
||||
}
|
||||
|
||||
/**
|
||||
* Authenticate with API
|
||||
*/
|
||||
async login(token, saveToken = true) {
|
||||
async login(username, password) {
|
||||
try {
|
||||
uiManager.setLoading(true);
|
||||
|
||||
// Set token and test authentication
|
||||
apiClient.setToken(token);
|
||||
await apiClient.testAuth();
|
||||
|
||||
// Authentication successful
|
||||
// Login and get token
|
||||
const response = await apiClient.login(username, password);
|
||||
this.currentUser = response.user;
|
||||
this.isAuthenticated = true;
|
||||
|
||||
if (saveToken) {
|
||||
localStorage.setItem('apiToken', token);
|
||||
}
|
||||
// Save token
|
||||
localStorage.setItem('authToken', response.access_token);
|
||||
|
||||
this.showAuthenticatedUI();
|
||||
await this.loadData();
|
||||
|
||||
uiManager.showNotification('Successfully connected to API', 'success');
|
||||
uiManager.showNotification(`Welcome back, ${this.currentUser.username}!`, 'success');
|
||||
} catch (error) {
|
||||
this.isAuthenticated = false;
|
||||
this.currentUser = null;
|
||||
apiClient.clearToken();
|
||||
localStorage.removeItem('authToken');
|
||||
|
||||
if (saveToken) {
|
||||
localStorage.removeItem('apiToken');
|
||||
// Show error on login form
|
||||
let errorMessage = 'Login failed';
|
||||
if (error.message) {
|
||||
if (error.message.includes('401') || error.message.includes('Unauthorized') ||
|
||||
error.message.includes('Invalid') || error.message.includes('credentials')) {
|
||||
errorMessage = 'Invalid username or password';
|
||||
} else if (error.message.includes('network') || error.message.includes('fetch')) {
|
||||
errorMessage = 'Unable to connect to server';
|
||||
} else {
|
||||
errorMessage = error.message;
|
||||
}
|
||||
}
|
||||
|
||||
uiManager.handleError(error, 'Authentication failed');
|
||||
this.showLoginError(errorMessage);
|
||||
} finally {
|
||||
uiManager.setLoading(false);
|
||||
}
|
||||
@@ -101,10 +250,17 @@ class MailingListApp {
|
||||
/**
|
||||
* Logout
|
||||
*/
|
||||
logout() {
|
||||
async logout() {
|
||||
try {
|
||||
await apiClient.logout();
|
||||
} catch (error) {
|
||||
// Ignore logout errors
|
||||
}
|
||||
|
||||
this.isAuthenticated = false;
|
||||
this.currentUser = null;
|
||||
apiClient.clearToken();
|
||||
localStorage.removeItem('apiToken');
|
||||
localStorage.removeItem('authToken');
|
||||
|
||||
this.showUnauthenticatedUI();
|
||||
uiManager.showNotification('Logged out successfully', 'info');
|
||||
@@ -114,23 +270,79 @@ class MailingListApp {
|
||||
* Show authenticated UI
|
||||
*/
|
||||
showAuthenticatedUI() {
|
||||
document.getElementById('authControls').style.display = 'none';
|
||||
document.getElementById('userInfo').style.display = 'flex';
|
||||
document.getElementById('loginPage').style.display = 'none';
|
||||
document.getElementById('mainHeader').style.display = 'block';
|
||||
document.getElementById('mainContent').style.display = 'block';
|
||||
|
||||
// Clear token input
|
||||
document.getElementById('apiToken').value = '';
|
||||
// Clear login inputs
|
||||
document.getElementById('username').value = '';
|
||||
document.getElementById('password').value = '';
|
||||
|
||||
// Update user info display
|
||||
if (this.currentUser) {
|
||||
document.getElementById('currentUsername').textContent = this.currentUser.username;
|
||||
document.getElementById('currentUserRole').textContent = this.currentUser.role;
|
||||
document.getElementById('dropdownUsername').textContent = this.currentUser.username;
|
||||
document.getElementById('dropdownUserRole').textContent = this.currentUser.role;
|
||||
|
||||
// Show/hide admin-only features
|
||||
const isAdmin = this.currentUser.role === 'administrator';
|
||||
document.getElementById('userManagementBtn').style.display = isAdmin ? 'block' : 'none';
|
||||
document.getElementById('userManagementDivider').style.display = isAdmin ? 'block' : 'none';
|
||||
|
||||
// Show/hide write access features
|
||||
const hasWriteAccess = this.currentUser.role === 'administrator' || this.currentUser.role === 'operator';
|
||||
this.updateUIForPermissions(hasWriteAccess);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Show unauthenticated UI
|
||||
*/
|
||||
showUnauthenticatedUI() {
|
||||
document.getElementById('authControls').style.display = 'flex';
|
||||
document.getElementById('userInfo').style.display = 'none';
|
||||
document.getElementById('loginPage').style.display = 'flex';
|
||||
document.getElementById('mainHeader').style.display = 'none';
|
||||
document.getElementById('mainContent').style.display = 'none';
|
||||
}
|
||||
|
||||
/**
|
||||
* Update UI elements based on user permissions
|
||||
*/
|
||||
updateUIForPermissions(hasWriteAccess) {
|
||||
// Disable/enable write action buttons
|
||||
const writeButtons = document.querySelectorAll('[data-requires-write]');
|
||||
writeButtons.forEach(button => {
|
||||
button.style.display = hasWriteAccess ? '' : 'none';
|
||||
});
|
||||
|
||||
// Update button attributes for later reference
|
||||
document.getElementById('addListBtn').setAttribute('data-requires-write', '');
|
||||
document.getElementById('addMemberBtn').setAttribute('data-requires-write', '');
|
||||
document.getElementById('showBulkImportBtn').setAttribute('data-requires-write', '');
|
||||
}
|
||||
|
||||
/**
|
||||
* Switch to users tab (triggered from user dropdown)
|
||||
*/
|
||||
switchToUsersTab() {
|
||||
// Switch to users tab programmatically
|
||||
const tabButtons = document.querySelectorAll('.tab-btn');
|
||||
const tabContents = document.querySelectorAll('.tab-content');
|
||||
|
||||
// Remove active class from all tabs and contents
|
||||
tabButtons.forEach(btn => btn.classList.remove('active'));
|
||||
tabContents.forEach(content => content.classList.remove('active'));
|
||||
|
||||
// Show users tab content
|
||||
const usersTab = document.getElementById('users-tab');
|
||||
if (usersTab) {
|
||||
usersTab.classList.add('active');
|
||||
}
|
||||
|
||||
// Load users data if needed
|
||||
this.loadUsers();
|
||||
}
|
||||
|
||||
/**
|
||||
* Load all data from API
|
||||
*/
|
||||
@@ -140,21 +352,37 @@ class MailingListApp {
|
||||
try {
|
||||
uiManager.setLoading(true);
|
||||
|
||||
// Load lists and members in parallel
|
||||
const [lists, members] = await Promise.all([
|
||||
// Load configuration, lists and members in parallel
|
||||
const [config, lists, members] = await Promise.all([
|
||||
apiClient.getConfig(),
|
||||
apiClient.getLists(),
|
||||
apiClient.getMembers()
|
||||
]);
|
||||
|
||||
this.config = config;
|
||||
|
||||
this.lists = lists;
|
||||
this.members = members;
|
||||
|
||||
// Load users if admin
|
||||
if (this.currentUser && this.currentUser.role === 'administrator') {
|
||||
try {
|
||||
this.users = await apiClient.getUsers();
|
||||
} catch (error) {
|
||||
console.warn('Failed to load users:', error);
|
||||
this.users = [];
|
||||
}
|
||||
}
|
||||
|
||||
// Load subscriptions for each list
|
||||
await this.loadSubscriptions();
|
||||
|
||||
// Render all views
|
||||
this.renderLists();
|
||||
this.renderMembers();
|
||||
if (this.currentUser && this.currentUser.role === 'administrator') {
|
||||
this.renderUsers();
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
uiManager.handleError(error, 'Failed to load data');
|
||||
@@ -205,6 +433,8 @@ class MailingListApp {
|
||||
return;
|
||||
}
|
||||
|
||||
const hasWriteAccess = this.currentUser && (this.currentUser.role === 'administrator' || this.currentUser.role === 'operator');
|
||||
|
||||
this.lists.forEach(list => {
|
||||
const row = document.createElement('tr');
|
||||
const memberCount = this.subscriptions.get(list.list_id)?.length || 0;
|
||||
@@ -237,24 +467,27 @@ class MailingListApp {
|
||||
const statusCell = row.cells[4];
|
||||
statusCell.appendChild(uiManager.createStatusBadge(list.active));
|
||||
|
||||
// Add action buttons
|
||||
// Add action buttons only for users with write access
|
||||
const actionsCell = row.cells[5].querySelector('.action-buttons');
|
||||
|
||||
const editBtn = uiManager.createActionButton('Edit', 'edit', 'btn-secondary', () => {
|
||||
uiManager.showListModal(list);
|
||||
});
|
||||
if (hasWriteAccess) {
|
||||
const editBtn = uiManager.createActionButton('Edit', 'edit', 'btn-secondary', () => {
|
||||
uiManager.showListModal(list);
|
||||
});
|
||||
|
||||
const deleteBtn = uiManager.createActionButton('Delete', 'trash', 'btn-danger', () => {
|
||||
uiManager.showConfirmation(
|
||||
`Are you sure you want to delete the mailing list "${list.list_name}"? This action cannot be undone.`,
|
||||
async () => {
|
||||
await this.deleteList(list.list_id);
|
||||
}
|
||||
);
|
||||
});
|
||||
|
||||
actionsCell.appendChild(editBtn);
|
||||
actionsCell.appendChild(deleteBtn);
|
||||
}
|
||||
|
||||
const deleteBtn = uiManager.createActionButton('Delete', 'trash', 'btn-danger', () => {
|
||||
uiManager.showConfirmation(
|
||||
`Are you sure you want to delete the mailing list "${list.list_name}"? This action cannot be undone.`,
|
||||
async () => {
|
||||
await this.deleteList(list.list_id);
|
||||
}
|
||||
);
|
||||
});
|
||||
|
||||
actionsCell.appendChild(editBtn);
|
||||
actionsCell.appendChild(deleteBtn);
|
||||
tbody.appendChild(row);
|
||||
});
|
||||
}
|
||||
@@ -282,6 +515,8 @@ class MailingListApp {
|
||||
return;
|
||||
}
|
||||
|
||||
const hasWriteAccess = this.currentUser && (this.currentUser.role === 'administrator' || this.currentUser.role === 'operator');
|
||||
|
||||
this.members.forEach(member => {
|
||||
const row = document.createElement('tr');
|
||||
|
||||
@@ -299,6 +534,7 @@ class MailingListApp {
|
||||
row.innerHTML = `
|
||||
<td>
|
||||
<div class="font-medium">${uiManager.escapeHtml(member.name)}</div>
|
||||
${this.config?.bounce_handling_enabled && member.bounce_count > 0 ? `<div class="text-xs text-muted" style="margin-top: 2px;"></div>` : ''}
|
||||
</td>
|
||||
<td>
|
||||
<a href="mailto:${member.email}" style="color: var(--primary-color)">
|
||||
@@ -319,6 +555,15 @@ class MailingListApp {
|
||||
</td>
|
||||
`;
|
||||
|
||||
// Add bounce badge if member has bounces (only if bounce handling is enabled)
|
||||
if (this.config?.bounce_handling_enabled && member.bounce_count > 0) {
|
||||
const bounceInfoDiv = row.cells[0].querySelector('.text-xs');
|
||||
const bounceBadge = uiManager.createBounceStatusBadge(member.bounce_status, member.bounce_count);
|
||||
if (bounceBadge) {
|
||||
bounceInfoDiv.appendChild(bounceBadge);
|
||||
}
|
||||
}
|
||||
|
||||
// Add status badge
|
||||
const statusCell = row.cells[3];
|
||||
statusCell.appendChild(uiManager.createStatusBadge(member.active));
|
||||
@@ -335,23 +580,120 @@ class MailingListApp {
|
||||
uiManager.showMemberSubscriptionsModal(member);
|
||||
});
|
||||
|
||||
// Create Bounces button (show if bounce handling is enabled and member has bounces or for admins/operators)
|
||||
if (this.config?.bounce_handling_enabled && (member.bounce_count > 0 || hasWriteAccess)) {
|
||||
const bouncesBtn = document.createElement('button');
|
||||
bouncesBtn.className = `btn btn-sm ${member.bounce_count > 0 ? 'btn-warning' : 'btn-secondary'}`;
|
||||
bouncesBtn.innerHTML = `<i class="fas fa-exclamation-triangle"></i> Bounces${member.bounce_count > 0 ? ` (${member.bounce_count})` : ''}`;
|
||||
bouncesBtn.title = 'View Bounce History';
|
||||
bouncesBtn.addEventListener('click', () => {
|
||||
uiManager.showBounceHistoryModal(member);
|
||||
});
|
||||
actionsCell.appendChild(bouncesBtn);
|
||||
}
|
||||
|
||||
// Only show edit/delete buttons for users with write access
|
||||
if (hasWriteAccess) {
|
||||
const editBtn = uiManager.createActionButton('Edit', 'edit', 'btn-secondary', () => {
|
||||
uiManager.showMemberModal(member);
|
||||
});
|
||||
|
||||
const deleteBtn = uiManager.createActionButton('Delete', 'trash', 'btn-danger', () => {
|
||||
uiManager.showConfirmation(
|
||||
`Are you sure you want to delete the member "${member.name}"? This will remove them from all mailing lists.`,
|
||||
async () => {
|
||||
await this.deleteMember(member.member_id);
|
||||
}
|
||||
);
|
||||
});
|
||||
|
||||
actionsCell.appendChild(subscriptionsBtn);
|
||||
actionsCell.appendChild(editBtn);
|
||||
actionsCell.appendChild(deleteBtn);
|
||||
} else {
|
||||
actionsCell.appendChild(subscriptionsBtn);
|
||||
}
|
||||
|
||||
tbody.appendChild(row);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Render users table (admin only)
|
||||
*/
|
||||
renderUsers() {
|
||||
const tbody = document.getElementById('usersTableBody');
|
||||
tbody.innerHTML = '';
|
||||
|
||||
if (this.users.length === 0) {
|
||||
tbody.innerHTML = `
|
||||
<tr>
|
||||
<td colspan="6" class="text-center text-muted">
|
||||
No users found. <a href="#" id="createFirstUser">Create your first user</a>
|
||||
</td>
|
||||
</tr>
|
||||
`;
|
||||
|
||||
document.getElementById('createFirstUser').addEventListener('click', (e) => {
|
||||
e.preventDefault();
|
||||
uiManager.showUserModal();
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
this.users.forEach(user => {
|
||||
const row = document.createElement('tr');
|
||||
|
||||
// Format dates
|
||||
const createdAt = new Date(user.created_at).toLocaleDateString();
|
||||
const lastLogin = user.last_login ? new Date(user.last_login).toLocaleDateString() : 'Never';
|
||||
|
||||
row.innerHTML = `
|
||||
<td>
|
||||
<div class="font-medium">${uiManager.escapeHtml(user.username)}</div>
|
||||
</td>
|
||||
<td>
|
||||
<span class="role-badge role-${user.role}">${user.role.replace('-', ' ')}</span>
|
||||
</td>
|
||||
<td>
|
||||
<div class="text-sm text-muted">${createdAt}</div>
|
||||
</td>
|
||||
<td>
|
||||
<div class="text-sm text-muted">${lastLogin}</div>
|
||||
</td>
|
||||
<td></td>
|
||||
<td>
|
||||
<div class="action-buttons"></div>
|
||||
</td>
|
||||
`;
|
||||
|
||||
// Add status badge
|
||||
const statusCell = row.cells[4];
|
||||
statusCell.appendChild(uiManager.createStatusBadge(user.active));
|
||||
|
||||
// Add action buttons
|
||||
const actionsCell = row.cells[5].querySelector('.action-buttons');
|
||||
|
||||
const editBtn = uiManager.createActionButton('Edit', 'edit', 'btn-secondary', () => {
|
||||
uiManager.showMemberModal(member);
|
||||
uiManager.showUserModal(user);
|
||||
});
|
||||
|
||||
const deleteBtn = uiManager.createActionButton('Delete', 'trash', 'btn-danger', () => {
|
||||
uiManager.showConfirmation(
|
||||
`Are you sure you want to delete the member "${member.name}"? This will remove them from all mailing lists.`,
|
||||
async () => {
|
||||
await this.deleteMember(member.member_id);
|
||||
}
|
||||
);
|
||||
});
|
||||
// Don't allow deletion of current user
|
||||
if (user.user_id !== this.currentUser.user_id) {
|
||||
const deleteBtn = uiManager.createActionButton('Delete', 'trash', 'btn-danger', () => {
|
||||
uiManager.showConfirmation(
|
||||
`Are you sure you want to delete the user "${user.username}"? This action cannot be undone.`,
|
||||
async () => {
|
||||
await this.deleteUser(user.user_id);
|
||||
}
|
||||
);
|
||||
});
|
||||
actionsCell.appendChild(editBtn);
|
||||
actionsCell.appendChild(deleteBtn);
|
||||
} else {
|
||||
actionsCell.appendChild(editBtn);
|
||||
}
|
||||
|
||||
// Append all buttons
|
||||
actionsCell.appendChild(subscriptionsBtn);
|
||||
actionsCell.appendChild(editBtn);
|
||||
actionsCell.appendChild(deleteBtn);
|
||||
tbody.appendChild(row);
|
||||
});
|
||||
}
|
||||
@@ -405,6 +747,165 @@ class MailingListApp {
|
||||
uiManager.setLoading(false);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a user
|
||||
*/
|
||||
async deleteUser(userId) {
|
||||
try {
|
||||
uiManager.setLoading(true);
|
||||
await apiClient.deleteUser(userId);
|
||||
uiManager.showNotification('User deleted successfully', 'success');
|
||||
await this.loadData();
|
||||
} catch (error) {
|
||||
uiManager.handleError(error, 'Failed to delete user');
|
||||
} finally {
|
||||
uiManager.setLoading(false);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Filter members by search term (name or email)
|
||||
*/
|
||||
filterMembers(searchTerm) {
|
||||
const searchInfo = document.getElementById('memberSearchInfo');
|
||||
const searchCount = document.getElementById('memberSearchCount');
|
||||
const tbody = document.getElementById('membersTableBody');
|
||||
|
||||
if (!this.members || !Array.isArray(this.members)) {
|
||||
return;
|
||||
}
|
||||
|
||||
// If no search term, show all members
|
||||
if (!searchTerm || searchTerm.trim() === '') {
|
||||
this.renderMembers();
|
||||
searchInfo.style.display = 'none';
|
||||
return;
|
||||
}
|
||||
|
||||
const normalizedSearch = searchTerm.trim().toLowerCase();
|
||||
|
||||
// Filter members by name or email
|
||||
const filteredMembers = this.members.filter(member => {
|
||||
const name = (member.name || '').toLowerCase();
|
||||
const email = (member.email || '').toLowerCase();
|
||||
return name.includes(normalizedSearch) || email.includes(normalizedSearch);
|
||||
});
|
||||
|
||||
// Update search results info
|
||||
searchCount.textContent = filteredMembers.length;
|
||||
searchInfo.style.display = 'block';
|
||||
|
||||
// Render filtered results
|
||||
this.renderFilteredMembers(filteredMembers);
|
||||
}
|
||||
|
||||
/**
|
||||
* Render filtered members (similar to renderMembers but with filtered data)
|
||||
*/
|
||||
renderFilteredMembers(filteredMembers) {
|
||||
const tbody = document.getElementById('membersTableBody');
|
||||
tbody.innerHTML = '';
|
||||
|
||||
if (filteredMembers.length === 0) {
|
||||
// Show no results message
|
||||
const row = tbody.insertRow();
|
||||
const cell = row.insertCell();
|
||||
cell.colSpan = 5;
|
||||
cell.className = 'no-results';
|
||||
cell.innerHTML = `
|
||||
<i class="fas fa-search"></i>
|
||||
<h3>No members found</h3>
|
||||
<p>Try adjusting your search terms or check the spelling.</p>
|
||||
`;
|
||||
return;
|
||||
}
|
||||
|
||||
filteredMembers.forEach(member => {
|
||||
const row = tbody.insertRow();
|
||||
|
||||
// Name
|
||||
const nameCell = row.insertCell();
|
||||
nameCell.textContent = member.name;
|
||||
|
||||
// Email
|
||||
const emailCell = row.insertCell();
|
||||
const emailLink = document.createElement('a');
|
||||
emailLink.href = `mailto:${member.email}`;
|
||||
emailLink.textContent = member.email;
|
||||
emailLink.style.color = 'var(--primary-color)';
|
||||
emailCell.appendChild(emailLink);
|
||||
|
||||
// Lists (show member's subscriptions)
|
||||
const listsCell = row.insertCell();
|
||||
const memberLists = [];
|
||||
this.subscriptions.forEach((members, listId) => {
|
||||
if (members.some(m => m.member_id === member.member_id)) {
|
||||
const list = this.lists.find(l => l.list_id === listId);
|
||||
if (list) {
|
||||
memberLists.push(list.list_name);
|
||||
}
|
||||
}
|
||||
});
|
||||
listsCell.textContent = memberLists.length > 0 ? memberLists.join(', ') : 'None';
|
||||
|
||||
// Status
|
||||
const statusCell = row.insertCell();
|
||||
const statusBadge = document.createElement('span');
|
||||
statusBadge.className = `status-badge ${member.active ? 'active' : 'inactive'}`;
|
||||
statusBadge.innerHTML = `<i class="fas fa-${member.active ? 'check' : 'times'}"></i> ${member.active ? 'Active' : 'Inactive'}`;
|
||||
|
||||
// Add bounce status if exists and bounce handling is enabled
|
||||
if (this.config?.bounce_handling_enabled && member.bounce_status && member.bounce_status !== 'clean') {
|
||||
const bounceIndicator = document.createElement('span');
|
||||
bounceIndicator.className = `bounce-badge bounce-${member.bounce_status === 'hard_bounce' ? 'hard' : 'soft'}`;
|
||||
bounceIndicator.innerHTML = `<i class="fas fa-exclamation-triangle"></i> Bounces`;
|
||||
bounceIndicator.title = `${member.bounce_count || 0} bounces - ${member.bounce_status}`;
|
||||
statusCell.appendChild(document.createElement('br'));
|
||||
statusCell.appendChild(bounceIndicator);
|
||||
}
|
||||
|
||||
statusCell.appendChild(statusBadge);
|
||||
|
||||
// Actions
|
||||
const actionsCell = row.insertCell();
|
||||
actionsCell.className = 'action-buttons';
|
||||
|
||||
// Bounces button (if bounce handling is enabled and member has bounce data)
|
||||
if (this.config?.bounce_handling_enabled && member.bounce_count > 0) {
|
||||
const bouncesBtn = uiManager.createActionButton('Bounces', 'exclamation-triangle', 'btn-warning', () => {
|
||||
uiManager.showBounceHistory(member);
|
||||
});
|
||||
actionsCell.appendChild(bouncesBtn);
|
||||
}
|
||||
|
||||
const subscriptionsBtn = uiManager.createActionButton('Lists', 'list', 'btn-secondary', () => {
|
||||
uiManager.showMemberSubscriptions(member);
|
||||
});
|
||||
actionsCell.appendChild(subscriptionsBtn);
|
||||
|
||||
const editBtn = uiManager.createActionButton('Edit', 'edit', 'btn-primary', () => {
|
||||
uiManager.showMemberModal(member);
|
||||
});
|
||||
editBtn.setAttribute('data-requires-write', '');
|
||||
actionsCell.appendChild(editBtn);
|
||||
|
||||
const deleteBtn = uiManager.createActionButton('Delete', 'trash', 'btn-danger', () => {
|
||||
uiManager.showConfirmation(
|
||||
`Are you sure you want to delete member "${member.name}"? This will also remove them from all mailing lists.`,
|
||||
async () => {
|
||||
await this.deleteMember(member.member_id);
|
||||
}
|
||||
);
|
||||
});
|
||||
deleteBtn.setAttribute('data-requires-write', '');
|
||||
actionsCell.appendChild(deleteBtn);
|
||||
});
|
||||
|
||||
// Update UI permissions for the filtered results
|
||||
const hasWriteAccess = this.currentUser && (this.currentUser.role === 'administrator' || this.currentUser.role === 'operator');
|
||||
this.updateUIForPermissions(hasWriteAccess);
|
||||
}
|
||||
}
|
||||
|
||||
// Initialize the application when DOM is loaded
|
||||
|
||||
@@ -50,10 +50,17 @@ class UIManager {
|
||||
this.showListModal();
|
||||
});
|
||||
|
||||
// Initialize bulk import listeners
|
||||
this.initializeBulkImportListeners();
|
||||
|
||||
document.getElementById('addMemberBtn').addEventListener('click', () => {
|
||||
this.showMemberModal();
|
||||
});
|
||||
|
||||
document.getElementById('addUserBtn').addEventListener('click', () => {
|
||||
this.showUserModal();
|
||||
});
|
||||
|
||||
|
||||
|
||||
// Member subscriptions modal
|
||||
@@ -69,6 +76,19 @@ class UIManager {
|
||||
this.handleMemberSubscriptionsSave();
|
||||
});
|
||||
|
||||
// Bounce history modal
|
||||
document.getElementById('bounceHistoryModalClose').addEventListener('click', () => {
|
||||
this.closeModal(document.getElementById('bounceHistoryModal'));
|
||||
});
|
||||
|
||||
document.getElementById('bounceHistoryCloseBtn').addEventListener('click', () => {
|
||||
this.closeModal(document.getElementById('bounceHistoryModal'));
|
||||
});
|
||||
|
||||
document.getElementById('bounceHistoryResetBtn').addEventListener('click', () => {
|
||||
this.handleBounceStatusReset();
|
||||
});
|
||||
|
||||
// Form submissions
|
||||
document.getElementById('listForm').addEventListener('submit', (e) => {
|
||||
e.preventDefault();
|
||||
@@ -85,6 +105,11 @@ class UIManager {
|
||||
this.handleSubscriptionFormSubmit();
|
||||
});
|
||||
|
||||
document.getElementById('userForm').addEventListener('submit', (e) => {
|
||||
e.preventDefault();
|
||||
this.handleUserFormSubmit();
|
||||
});
|
||||
|
||||
// Confirmation modal
|
||||
document.getElementById('confirmOkBtn').addEventListener('click', () => {
|
||||
if (this.confirmCallback) {
|
||||
@@ -98,6 +123,19 @@ class UIManager {
|
||||
this.confirmCallback = null;
|
||||
this.closeModal(document.getElementById('confirmModal'));
|
||||
});
|
||||
|
||||
// Bounce history modal
|
||||
document.getElementById('bounceHistoryModalClose').addEventListener('click', () => {
|
||||
this.closeModal(document.getElementById('bounceHistoryModal'));
|
||||
});
|
||||
|
||||
document.getElementById('bounceHistoryCloseBtn').addEventListener('click', () => {
|
||||
this.closeModal(document.getElementById('bounceHistoryModal'));
|
||||
});
|
||||
|
||||
document.getElementById('bounceHistoryResetBtn').addEventListener('click', () => {
|
||||
this.handleResetBounceStatus();
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -242,6 +280,42 @@ class UIManager {
|
||||
this.showModal(modal);
|
||||
}
|
||||
|
||||
/**
|
||||
* Show user modal (add/edit)
|
||||
*/
|
||||
showUserModal(userData = null) {
|
||||
const modal = document.getElementById('userModal');
|
||||
const title = document.getElementById('userModalTitle');
|
||||
const form = document.getElementById('userForm');
|
||||
const passwordHelp = document.getElementById('passwordHelp');
|
||||
const passwordField = document.getElementById('userPassword');
|
||||
|
||||
if (userData) {
|
||||
// Edit mode
|
||||
title.textContent = 'Edit User';
|
||||
document.getElementById('userName').value = userData.username;
|
||||
document.getElementById('userName').readOnly = true; // Can't change username
|
||||
passwordField.placeholder = 'Leave blank to keep current password';
|
||||
passwordField.required = false;
|
||||
passwordHelp.style.display = 'block';
|
||||
document.getElementById('userRole').value = userData.role;
|
||||
document.getElementById('userActive').checked = userData.active;
|
||||
this.currentEditingItem = userData;
|
||||
} else {
|
||||
// Add mode
|
||||
title.textContent = 'Add User';
|
||||
form.reset();
|
||||
document.getElementById('userName').readOnly = false;
|
||||
passwordField.placeholder = 'Password';
|
||||
passwordField.required = true;
|
||||
passwordHelp.style.display = 'none';
|
||||
document.getElementById('userActive').checked = true;
|
||||
this.currentEditingItem = null;
|
||||
}
|
||||
|
||||
this.showModal(modal);
|
||||
}
|
||||
|
||||
/**
|
||||
* Show subscription modal
|
||||
*/
|
||||
@@ -397,6 +471,50 @@ class UIManager {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle user form submission
|
||||
*/
|
||||
async handleUserFormSubmit() {
|
||||
const form = document.getElementById('userForm');
|
||||
const formData = new FormData(form);
|
||||
|
||||
const userData = {
|
||||
username: formData.get('userName'),
|
||||
role: formData.get('userRole'),
|
||||
active: formData.get('userActive') === 'on'
|
||||
};
|
||||
|
||||
// Only include password if it's provided (for updates, empty means no change)
|
||||
const password = formData.get('userPassword');
|
||||
if (password) {
|
||||
userData.password = password;
|
||||
}
|
||||
|
||||
try {
|
||||
this.setLoading(true);
|
||||
|
||||
if (this.currentEditingItem) {
|
||||
// Update existing user
|
||||
await apiClient.updateUser(this.currentEditingItem.user_id, userData);
|
||||
this.showNotification('User updated successfully', 'success');
|
||||
} else {
|
||||
// Create new user
|
||||
if (!userData.password) {
|
||||
throw new Error('Password is required for new users');
|
||||
}
|
||||
await apiClient.createUser(userData);
|
||||
this.showNotification('User created successfully', 'success');
|
||||
}
|
||||
|
||||
this.closeModal(document.getElementById('userModal'));
|
||||
await window.app.loadData();
|
||||
} catch (error) {
|
||||
this.handleError(error, 'Failed to save user');
|
||||
} finally {
|
||||
this.setLoading(false);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Show member subscriptions modal
|
||||
*/
|
||||
@@ -572,6 +690,117 @@ class UIManager {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Show bounce history modal for a member
|
||||
*/
|
||||
async showBounceHistoryModal(member) {
|
||||
const modal = document.getElementById('bounceHistoryModal');
|
||||
|
||||
// Update member info
|
||||
document.getElementById('bounceHistoryTitle').textContent = `Bounce History - ${member.name}`;
|
||||
document.getElementById('bounceHistoryMemberName').textContent = member.name;
|
||||
document.getElementById('bounceHistoryMemberEmail').textContent = member.email;
|
||||
|
||||
// Update summary stats
|
||||
document.getElementById('bounceTotalCount').textContent = member.bounce_count || 0;
|
||||
document.getElementById('bounceLastDate').textContent = this.formatDateTime(member.last_bounce_at);
|
||||
|
||||
const statusText = document.getElementById('bounceStatusText');
|
||||
statusText.className = 'bounce-stat-value';
|
||||
if (member.bounce_status === 'hard_bounce') {
|
||||
statusText.textContent = 'Hard Bounce';
|
||||
statusText.classList.add('text-danger');
|
||||
} else if (member.bounce_status === 'soft_bounce') {
|
||||
statusText.textContent = 'Soft Bounce';
|
||||
statusText.classList.add('text-warning');
|
||||
} else {
|
||||
statusText.textContent = 'Clean';
|
||||
statusText.classList.add('text-success');
|
||||
}
|
||||
|
||||
try {
|
||||
this.currentMemberForBounces = member;
|
||||
|
||||
// Load bounce history
|
||||
const bounces = await apiClient.getMemberBounces(member.member_id);
|
||||
this.renderBounceHistory(bounces);
|
||||
|
||||
this.showModal(modal);
|
||||
} catch (error) {
|
||||
this.handleError(error, 'Failed to load bounce history');
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Render bounce history list
|
||||
*/
|
||||
renderBounceHistory(bounces) {
|
||||
const container = document.getElementById('bounceHistoryList');
|
||||
container.innerHTML = '';
|
||||
|
||||
if (bounces.length === 0) {
|
||||
container.innerHTML = `
|
||||
<div class="empty-state">
|
||||
<i class="fas fa-check-circle"></i>
|
||||
<p>No bounces recorded for this member</p>
|
||||
</div>
|
||||
`;
|
||||
return;
|
||||
}
|
||||
|
||||
bounces.forEach(bounce => {
|
||||
const item = document.createElement('div');
|
||||
item.className = 'bounce-history-item';
|
||||
|
||||
let typeClass = 'bounce-type-';
|
||||
if (bounce.bounce_type === 'Permanent') {
|
||||
typeClass += 'permanent';
|
||||
} else if (bounce.bounce_type === 'Transient') {
|
||||
typeClass += 'transient';
|
||||
} else {
|
||||
typeClass += 'undetermined';
|
||||
}
|
||||
|
||||
item.innerHTML = `
|
||||
<div class="bounce-header">
|
||||
<span class="bounce-type ${typeClass}">
|
||||
<i class="fas fa-${bounce.bounce_type === 'Permanent' ? 'times-circle' : bounce.bounce_type === 'Transient' ? 'exclamation-circle' : 'question-circle'}"></i>
|
||||
${bounce.bounce_type}
|
||||
</span>
|
||||
<span class="bounce-date">${this.formatDateTime(bounce.timestamp)}</span>
|
||||
</div>
|
||||
${bounce.bounce_subtype ? `<div class="bounce-subtype">Subtype: ${this.escapeHtml(bounce.bounce_subtype)}</div>` : ''}
|
||||
${bounce.diagnostic_code ? `<div class="bounce-diagnostic">${this.escapeHtml(bounce.diagnostic_code)}</div>` : ''}
|
||||
`;
|
||||
|
||||
container.appendChild(item);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle bounce status reset
|
||||
*/
|
||||
async handleResetBounceStatus() {
|
||||
if (!this.currentMemberForBounces) return;
|
||||
|
||||
this.showConfirmation(
|
||||
`Are you sure you want to reset the bounce status for "${this.currentMemberForBounces.name}"? This will clear the bounce count and allow emails to be sent to this address again.`,
|
||||
async () => {
|
||||
try {
|
||||
this.setLoading(true);
|
||||
await apiClient.resetBounceStatus(this.currentMemberForBounces.member_id);
|
||||
this.showNotification('Bounce status reset successfully', 'success');
|
||||
this.closeModal(document.getElementById('bounceHistoryModal'));
|
||||
await window.app.loadData();
|
||||
} catch (error) {
|
||||
this.handleError(error, 'Failed to reset bounce status');
|
||||
} finally {
|
||||
this.setLoading(false);
|
||||
}
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle API errors
|
||||
*/
|
||||
@@ -623,6 +852,49 @@ class UIManager {
|
||||
return badge;
|
||||
}
|
||||
|
||||
/**
|
||||
* Create bounce status badge
|
||||
*/
|
||||
createBounceStatusBadge(bounceStatus, bounceCount) {
|
||||
const badge = document.createElement('span');
|
||||
|
||||
if (bounceStatus === 'hard_bounce') {
|
||||
badge.className = 'bounce-badge bounce-hard';
|
||||
badge.innerHTML = `<i class="fas fa-exclamation-triangle"></i> Hard Bounce`;
|
||||
badge.title = `${bounceCount} bounce(s) - Email permanently failed`;
|
||||
} else if (bounceStatus === 'soft_bounce') {
|
||||
badge.className = 'bounce-badge bounce-soft';
|
||||
badge.innerHTML = `<i class="fas fa-exclamation-circle"></i> Soft Bounce`;
|
||||
badge.title = `${bounceCount} bounce(s) - Temporary delivery issues`;
|
||||
} else if (bounceCount > 0) {
|
||||
badge.className = 'bounce-badge bounce-warning';
|
||||
badge.innerHTML = `<i class="fas fa-info-circle"></i> ${bounceCount} bounce(s)`;
|
||||
badge.title = `${bounceCount} bounce(s) recorded`;
|
||||
} else {
|
||||
return null; // No badge for clean status
|
||||
}
|
||||
|
||||
return badge;
|
||||
}
|
||||
|
||||
/**
|
||||
* Format date and time
|
||||
*/
|
||||
formatDateTime(dateString) {
|
||||
if (!dateString) return 'Never';
|
||||
const date = new Date(dateString);
|
||||
return date.toLocaleString();
|
||||
}
|
||||
|
||||
/**
|
||||
* Format date only
|
||||
*/
|
||||
formatDate(dateString) {
|
||||
if (!dateString) return 'Never';
|
||||
const date = new Date(dateString);
|
||||
return date.toLocaleDateString();
|
||||
}
|
||||
|
||||
/**
|
||||
* Format email as mailto link
|
||||
*/
|
||||
@@ -634,6 +906,546 @@ class UIManager {
|
||||
return link;
|
||||
}
|
||||
|
||||
/**
|
||||
* Show bulk import modal
|
||||
*/
|
||||
showBulkImportModal() {
|
||||
this.resetBulkImportModal();
|
||||
this.showModal(document.getElementById('bulkImportModal'));
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset bulk import modal state
|
||||
*/
|
||||
resetBulkImportModal() {
|
||||
// Reset to step 1
|
||||
document.querySelectorAll('.import-step').forEach(step => {
|
||||
step.style.display = 'none';
|
||||
});
|
||||
document.getElementById('importStep1').style.display = 'block';
|
||||
|
||||
// Reset button states
|
||||
document.getElementById('bulkImportBackBtn').style.display = 'none';
|
||||
document.getElementById('bulkImportNextBtn').style.display = 'inline-block';
|
||||
document.getElementById('bulkImportNextBtn').disabled = true;
|
||||
document.getElementById('bulkImportBtn').style.display = 'none';
|
||||
document.getElementById('bulkImportDoneBtn').style.display = 'none';
|
||||
|
||||
// Clear file input
|
||||
document.getElementById('csvFileInput').value = '';
|
||||
document.getElementById('fileInfo').style.display = 'none';
|
||||
|
||||
// Reset parsed data
|
||||
this.csvData = null;
|
||||
this.parsedRows = [];
|
||||
this.currentStep = 1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Initialize bulk import event listeners
|
||||
*/
|
||||
initializeBulkImportListeners() {
|
||||
const fileUploadArea = document.getElementById('fileUploadArea');
|
||||
const csvFileInput = document.getElementById('csvFileInput');
|
||||
const nextBtn = document.getElementById('bulkImportNextBtn');
|
||||
const backBtn = document.getElementById('bulkImportBackBtn');
|
||||
const importBtn = document.getElementById('bulkImportBtn');
|
||||
const doneBtn = document.getElementById('bulkImportDoneBtn');
|
||||
const removeFileBtn = document.getElementById('removeFileBtn');
|
||||
|
||||
// File upload area click
|
||||
fileUploadArea.addEventListener('click', () => {
|
||||
csvFileInput.click();
|
||||
});
|
||||
|
||||
// File input change
|
||||
csvFileInput.addEventListener('change', (e) => {
|
||||
this.handleFileSelection(e.target.files[0]);
|
||||
});
|
||||
|
||||
// Remove file button
|
||||
removeFileBtn.addEventListener('click', () => {
|
||||
this.removeSelectedFile();
|
||||
});
|
||||
|
||||
// Drag and drop
|
||||
fileUploadArea.addEventListener('dragover', (e) => {
|
||||
e.preventDefault();
|
||||
fileUploadArea.classList.add('dragover');
|
||||
});
|
||||
|
||||
fileUploadArea.addEventListener('dragleave', () => {
|
||||
fileUploadArea.classList.remove('dragover');
|
||||
});
|
||||
|
||||
fileUploadArea.addEventListener('drop', (e) => {
|
||||
e.preventDefault();
|
||||
fileUploadArea.classList.remove('dragover');
|
||||
const files = e.dataTransfer.files;
|
||||
if (files.length > 0) {
|
||||
this.handleFileSelection(files[0]);
|
||||
}
|
||||
});
|
||||
|
||||
// Navigation buttons
|
||||
nextBtn.addEventListener('click', () => {
|
||||
this.handleBulkImportNext();
|
||||
});
|
||||
|
||||
backBtn.addEventListener('click', () => {
|
||||
this.handleBulkImportBack();
|
||||
});
|
||||
|
||||
importBtn.addEventListener('click', () => {
|
||||
this.handleBulkImportSubmit();
|
||||
});
|
||||
|
||||
doneBtn.addEventListener('click', () => {
|
||||
this.closeModal(document.getElementById('bulkImportModal'));
|
||||
// Reload data to show imported members
|
||||
if (window.app) {
|
||||
window.app.loadData();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle file selection
|
||||
*/
|
||||
handleFileSelection(file) {
|
||||
if (!file) return;
|
||||
|
||||
if (!file.name.toLowerCase().endsWith('.csv')) {
|
||||
this.showNotification('Please select a CSV file', 'error');
|
||||
return;
|
||||
}
|
||||
|
||||
const reader = new FileReader();
|
||||
reader.onload = (e) => {
|
||||
this.csvData = e.target.result;
|
||||
this.displayFileInfo(file);
|
||||
this.parseCSVPreview();
|
||||
document.getElementById('bulkImportNextBtn').disabled = false;
|
||||
};
|
||||
reader.readAsText(file);
|
||||
}
|
||||
|
||||
/**
|
||||
* Display file information
|
||||
*/
|
||||
displayFileInfo(file) {
|
||||
document.getElementById('fileName').textContent = file.name;
|
||||
document.getElementById('fileSize').textContent = `(${this.formatFileSize(file.size)})`;
|
||||
document.getElementById('fileInfo').style.display = 'flex';
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove selected file
|
||||
*/
|
||||
removeSelectedFile() {
|
||||
document.getElementById('csvFileInput').value = '';
|
||||
document.getElementById('fileInfo').style.display = 'none';
|
||||
document.getElementById('bulkImportNextBtn').disabled = true;
|
||||
this.csvData = null;
|
||||
this.parsedRows = [];
|
||||
}
|
||||
|
||||
/**
|
||||
* Format file size in human readable format
|
||||
*/
|
||||
formatFileSize(bytes) {
|
||||
if (bytes === 0) return '0 B';
|
||||
const k = 1024;
|
||||
const sizes = ['B', 'KB', 'MB', 'GB'];
|
||||
const i = Math.floor(Math.log(bytes) / Math.log(k));
|
||||
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + ' ' + sizes[i];
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse CSV for preview
|
||||
*/
|
||||
parseCSVPreview() {
|
||||
if (!this.csvData) return;
|
||||
|
||||
const lines = this.csvData.trim().split('\n');
|
||||
if (lines.length < 2) {
|
||||
this.showNotification('CSV file must contain at least a header and one data row', 'error');
|
||||
return;
|
||||
}
|
||||
|
||||
// Parse header
|
||||
const headers = this.parseCSVLine(lines[0]);
|
||||
const nameIndex = headers.findIndex(h => h.toLowerCase().includes('name'));
|
||||
const emailIndex = headers.findIndex(h => h.toLowerCase().includes('email'));
|
||||
|
||||
if (nameIndex === -1 || emailIndex === -1) {
|
||||
this.showNotification('CSV must contain Name and Email columns', 'error');
|
||||
return;
|
||||
}
|
||||
|
||||
this.parsedRows = [];
|
||||
this.csvHeaders = headers;
|
||||
let validRows = 0;
|
||||
let errorRows = 0;
|
||||
|
||||
// Parse data rows
|
||||
for (let i = 1; i < lines.length; i++) {
|
||||
const rawLine = lines[i].trim();
|
||||
if (!rawLine) continue; // Skip empty lines
|
||||
|
||||
const values = this.parseCSVLine(rawLine);
|
||||
const name = (values[nameIndex] || '').trim();
|
||||
const email = (values[emailIndex] || '').trim();
|
||||
|
||||
// Determine error type and message
|
||||
let error = null;
|
||||
let isValid = true;
|
||||
|
||||
if (!email) {
|
||||
error = 'Missing email address';
|
||||
isValid = false;
|
||||
} else if (!this.isValidEmail(email)) {
|
||||
error = 'Invalid email format';
|
||||
isValid = false;
|
||||
}
|
||||
// Note: Missing name is OK - we'll use the email as display name if needed
|
||||
|
||||
const row = {
|
||||
index: i,
|
||||
name,
|
||||
email,
|
||||
valid: isValid,
|
||||
error,
|
||||
rawLine,
|
||||
parsedValues: values,
|
||||
headers: headers
|
||||
};
|
||||
|
||||
this.parsedRows.push(row);
|
||||
if (isValid) validRows++;
|
||||
else errorRows++;
|
||||
}
|
||||
|
||||
// Update preview stats
|
||||
document.getElementById('totalRowsCount').textContent = this.parsedRows.length;
|
||||
document.getElementById('validRowsCount').textContent = validRows;
|
||||
document.getElementById('errorRowsCount').textContent = errorRows;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a CSV line handling quoted values
|
||||
*/
|
||||
parseCSVLine(line) {
|
||||
const result = [];
|
||||
let current = '';
|
||||
let inQuotes = false;
|
||||
|
||||
for (let i = 0; i < line.length; i++) {
|
||||
const char = line[i];
|
||||
|
||||
if (char === '"') {
|
||||
inQuotes = !inQuotes;
|
||||
} else if (char === ',' && !inQuotes) {
|
||||
result.push(current.trim());
|
||||
current = '';
|
||||
} else {
|
||||
current += char;
|
||||
}
|
||||
}
|
||||
|
||||
result.push(current.trim());
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Simple email validation
|
||||
*/
|
||||
isValidEmail(email) {
|
||||
const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
|
||||
return emailRegex.test(email);
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle next button click
|
||||
*/
|
||||
handleBulkImportNext() {
|
||||
if (this.currentStep === 1) {
|
||||
this.showStep2Preview();
|
||||
} else if (this.currentStep === 2) {
|
||||
this.showStep3ListSelection();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle back button click
|
||||
*/
|
||||
handleBulkImportBack() {
|
||||
if (this.currentStep === 2) {
|
||||
this.showStep1Upload();
|
||||
} else if (this.currentStep === 3) {
|
||||
this.showStep2Preview();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Show step 1 (upload)
|
||||
*/
|
||||
showStep1Upload() {
|
||||
document.querySelectorAll('.import-step').forEach(step => {
|
||||
step.style.display = 'none';
|
||||
});
|
||||
document.getElementById('importStep1').style.display = 'block';
|
||||
|
||||
document.getElementById('bulkImportBackBtn').style.display = 'none';
|
||||
document.getElementById('bulkImportNextBtn').style.display = 'inline-block';
|
||||
document.getElementById('bulkImportBtn').style.display = 'none';
|
||||
|
||||
this.currentStep = 1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Show step 2 (preview)
|
||||
*/
|
||||
showStep2Preview() {
|
||||
document.querySelectorAll('.import-step').forEach(step => {
|
||||
step.style.display = 'none';
|
||||
});
|
||||
document.getElementById('importStep2').style.display = 'block';
|
||||
|
||||
document.getElementById('bulkImportBackBtn').style.display = 'inline-block';
|
||||
document.getElementById('bulkImportNextBtn').style.display = 'inline-block';
|
||||
document.getElementById('bulkImportBtn').style.display = 'none';
|
||||
|
||||
this.renderPreviewTable();
|
||||
this.currentStep = 2;
|
||||
}
|
||||
|
||||
/**
|
||||
* Show step 3 (list selection)
|
||||
*/
|
||||
showStep3ListSelection() {
|
||||
document.querySelectorAll('.import-step').forEach(step => {
|
||||
step.style.display = 'none';
|
||||
});
|
||||
document.getElementById('importStep3').style.display = 'block';
|
||||
|
||||
document.getElementById('bulkImportBackBtn').style.display = 'inline-block';
|
||||
document.getElementById('bulkImportNextBtn').style.display = 'none';
|
||||
document.getElementById('bulkImportBtn').style.display = 'inline-block';
|
||||
|
||||
this.renderListSelection();
|
||||
this.currentStep = 3;
|
||||
}
|
||||
|
||||
/**
|
||||
* Render preview table
|
||||
*/
|
||||
renderPreviewTable() {
|
||||
const tbody = document.getElementById('previewTableBody');
|
||||
tbody.innerHTML = '';
|
||||
|
||||
// Show first 10 rows for preview
|
||||
const previewRows = this.parsedRows.slice(0, 10);
|
||||
|
||||
previewRows.forEach(row => {
|
||||
const tr = document.createElement('tr');
|
||||
tr.innerHTML = `
|
||||
<td>${row.index}</td>
|
||||
<td>${this.escapeHtml(row.name)}</td>
|
||||
<td>${this.escapeHtml(row.email)}</td>
|
||||
<td>
|
||||
<span class="row-status ${row.valid ? 'valid' : 'error'}">
|
||||
${row.valid ? 'Valid' : 'Error'}
|
||||
</span>
|
||||
</td>
|
||||
`;
|
||||
tbody.appendChild(tr);
|
||||
});
|
||||
|
||||
// Show errors if any
|
||||
const errors = this.parsedRows.filter(row => !row.valid);
|
||||
if (errors.length > 0) {
|
||||
document.getElementById('errorList').style.display = 'block';
|
||||
this.renderErrorDisplay(errors);
|
||||
} else {
|
||||
document.getElementById('errorList').style.display = 'none';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Render error display with expandable details
|
||||
*/
|
||||
renderErrorDisplay(errors) {
|
||||
// Update error summary
|
||||
const errorSummary = document.getElementById('errorSummary');
|
||||
const errorTypes = {};
|
||||
|
||||
errors.forEach(error => {
|
||||
const type = error.error;
|
||||
errorTypes[type] = (errorTypes[type] || 0) + 1;
|
||||
});
|
||||
|
||||
const summaryText = Object.entries(errorTypes)
|
||||
.map(([type, count]) => `${count} ${type.toLowerCase()}`)
|
||||
.join(', ');
|
||||
|
||||
errorSummary.innerHTML = `
|
||||
<p><strong>${errors.length}</strong> rows have issues: ${summaryText}</p>
|
||||
<p class="text-muted text-sm">These rows will be skipped during import. Click "Show All" to see details.</p>
|
||||
`;
|
||||
|
||||
// Render error table
|
||||
const errorTableBody = document.getElementById('errorTableBody');
|
||||
errorTableBody.innerHTML = '';
|
||||
|
||||
errors.forEach(row => {
|
||||
const tr = document.createElement('tr');
|
||||
tr.innerHTML = `
|
||||
<td class="font-medium">${row.index + 1}</td>
|
||||
<td class="font-medium">${this.escapeHtml(row.error)}</td>
|
||||
<td>${this.escapeHtml(row.name || '-')}</td>
|
||||
<td>${this.escapeHtml(row.email || '-')}</td>
|
||||
<td class="raw-data-cell" title="${this.escapeHtml(row.rawLine)}">${this.escapeHtml(row.rawLine)}</td>
|
||||
`;
|
||||
errorTableBody.appendChild(tr);
|
||||
});
|
||||
|
||||
// Setup toggle functionality
|
||||
this.setupErrorToggle();
|
||||
}
|
||||
|
||||
/**
|
||||
* Setup error toggle functionality
|
||||
*/
|
||||
setupErrorToggle() {
|
||||
const toggleBtn = document.getElementById('toggleErrorsBtn');
|
||||
const errorDetails = document.getElementById('errorDetails');
|
||||
const toggleText = document.getElementById('errorToggleText');
|
||||
const toggleIcon = document.getElementById('errorToggleIcon');
|
||||
|
||||
// Remove existing listener to prevent duplicates
|
||||
const newToggleBtn = toggleBtn.cloneNode(true);
|
||||
toggleBtn.parentNode.replaceChild(newToggleBtn, toggleBtn);
|
||||
|
||||
newToggleBtn.addEventListener('click', () => {
|
||||
const isHidden = errorDetails.style.display === 'none';
|
||||
|
||||
if (isHidden) {
|
||||
errorDetails.style.display = 'block';
|
||||
toggleText.textContent = 'Hide Details';
|
||||
toggleIcon.className = 'fas fa-chevron-up';
|
||||
} else {
|
||||
errorDetails.style.display = 'none';
|
||||
toggleText.textContent = 'Show All';
|
||||
toggleIcon.className = 'fas fa-chevron-down';
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Render list selection checkboxes
|
||||
*/
|
||||
renderListSelection() {
|
||||
const container = document.getElementById('listSelection');
|
||||
container.innerHTML = '';
|
||||
|
||||
if (!window.app || !window.app.lists) {
|
||||
container.innerHTML = '<p class="text-muted">No mailing lists available</p>';
|
||||
return;
|
||||
}
|
||||
|
||||
window.app.lists.forEach(list => {
|
||||
const div = document.createElement('div');
|
||||
div.className = 'list-checkbox';
|
||||
div.innerHTML = `
|
||||
<input type="checkbox" id="list_${list.list_id}" value="${list.list_id}">
|
||||
<div class="list-checkbox-label">
|
||||
<span class="list-checkbox-name">${this.escapeHtml(list.list_name)}</span>
|
||||
<span class="list-checkbox-email">${this.escapeHtml(list.list_email)}</span>
|
||||
</div>
|
||||
`;
|
||||
container.appendChild(div);
|
||||
|
||||
// Make the whole div clickable
|
||||
div.addEventListener('click', (e) => {
|
||||
if (e.target.type !== 'checkbox') {
|
||||
const checkbox = div.querySelector('input[type="checkbox"]');
|
||||
checkbox.checked = !checkbox.checked;
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle bulk import submission
|
||||
*/
|
||||
async handleBulkImportSubmit() {
|
||||
const selectedLists = Array.from(document.querySelectorAll('#listSelection input[type="checkbox"]:checked'))
|
||||
.map(cb => parseInt(cb.value));
|
||||
|
||||
if (selectedLists.length === 0) {
|
||||
this.showNotification('Please select at least one mailing list', 'error');
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
this.setLoading(true);
|
||||
|
||||
const result = await apiClient.bulkImportMembers(this.csvData, selectedLists);
|
||||
|
||||
this.showStep4Results(result);
|
||||
|
||||
} catch (error) {
|
||||
this.handleError(error, 'Bulk import failed');
|
||||
} finally {
|
||||
this.setLoading(false);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Show step 4 (results)
|
||||
*/
|
||||
showStep4Results(result) {
|
||||
document.querySelectorAll('.import-step').forEach(step => {
|
||||
step.style.display = 'none';
|
||||
});
|
||||
document.getElementById('importStep4').style.display = 'block';
|
||||
|
||||
document.getElementById('bulkImportBackBtn').style.display = 'none';
|
||||
document.getElementById('bulkImportNextBtn').style.display = 'none';
|
||||
document.getElementById('bulkImportBtn').style.display = 'none';
|
||||
document.getElementById('bulkImportDoneBtn').style.display = 'inline-block';
|
||||
|
||||
// Update result stats
|
||||
document.getElementById('processedCount').textContent = result.processed_rows;
|
||||
document.getElementById('createdCount').textContent = result.created_members;
|
||||
document.getElementById('updatedCount').textContent = result.updated_members;
|
||||
document.getElementById('subscriptionsCount').textContent = result.subscriptions_added;
|
||||
|
||||
// Show errors if any
|
||||
if (result.errors && result.errors.length > 0) {
|
||||
document.getElementById('resultErrors').style.display = 'block';
|
||||
const errorList = document.getElementById('resultErrorList');
|
||||
errorList.innerHTML = '';
|
||||
|
||||
result.errors.forEach(error => {
|
||||
const li = document.createElement('li');
|
||||
li.textContent = error;
|
||||
errorList.appendChild(li);
|
||||
});
|
||||
} else {
|
||||
document.getElementById('resultErrors').style.display = 'none';
|
||||
}
|
||||
|
||||
this.currentStep = 4;
|
||||
|
||||
// Show success notification
|
||||
this.showNotification(
|
||||
`Successfully imported ${result.processed_rows} members with ${result.subscriptions_added} subscriptions`,
|
||||
'success'
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Escape HTML to prevent XSS
|
||||
*/
|
||||
|
||||
Reference in New Issue
Block a user