387 lines
12 KiB
Markdown
387 lines
12 KiB
Markdown
# Mail List Manager
|
|
|
|
A complete containerized mailing list management system with web interface, REST API, MySQL database, and Postfix mail server configured as an SMTP relay through Amazon SES.
|
|
|
|
## Architecture
|
|
|
|
**Multi-Service Architecture:**
|
|
- **Web Frontend** - Modern responsive interface for managing lists and members (Port 3000)
|
|
- **REST API** - FastAPI backend with token authentication (Port 8000)
|
|
- **MySQL Database** - Stores lists, members, and subscriptions with real-time integration
|
|
- **Postfix Mail Server** - SMTP server with native MySQL integration for dynamic list expansion (Port 25)
|
|
- **Amazon SES** - Outbound mail relay for reliable delivery
|
|
|
|
**Key Features:**
|
|
- Real-time list updates (no Postfix reload needed)
|
|
- Native Postfix MySQL queries for instant list expansion
|
|
- Token-based API authentication
|
|
- Sender whitelisting for authorized board members
|
|
- Private Docker network for security
|
|
- Complete web-based management interface
|
|
|
|
## Quick Start
|
|
|
|
1. Copy the environment template:
|
|
```bash
|
|
cp .env.example .env
|
|
```
|
|
|
|
2. Edit `.env` with your credentials:
|
|
```bash
|
|
# SES Credentials (Required)
|
|
SES_USER=your_ses_access_key
|
|
SES_PASS=your_ses_secret_key
|
|
|
|
# MySQL Credentials (Required)
|
|
MYSQL_ROOT_PASSWORD=secure_root_password
|
|
MYSQL_PASSWORD=secure_maillist_password
|
|
|
|
# API Authentication (Required)
|
|
API_TOKEN=your_secure_api_token
|
|
|
|
# Optional: SMTP configuration (defaults to EU West 2)
|
|
SMTP_HOST=email-smtp.eu-west-2.amazonaws.com
|
|
SMTP_PORT=587
|
|
```
|
|
|
|
3. Build and start all services:
|
|
```bash
|
|
sudo docker-compose up --build
|
|
```
|
|
|
|
4. Access the web interface:
|
|
```
|
|
http://localhost:3000
|
|
```
|
|
|
|
Enter your API_TOKEN to authenticate and start managing lists!
|
|
|
|
5. Alternative: Use the REST API directly:
|
|
```bash
|
|
# Health check
|
|
curl http://localhost:8000/health
|
|
|
|
# Get all lists (requires authentication)
|
|
curl -H "Authorization: Bearer your_api_token" http://localhost:8000/lists
|
|
```
|
|
|
|
## Services
|
|
|
|
### Web Interface (Port 3000)
|
|
- Modern, responsive interface for managing mailing lists and members
|
|
- Token-based authentication
|
|
- Real-time updates and feedback
|
|
- See `web/README.md` for details
|
|
|
|
### REST API (Port 8000)
|
|
- Complete CRUD operations for lists, members, and subscriptions
|
|
- Bearer token authentication
|
|
- Interactive documentation at http://localhost:8000/docs
|
|
- See `api/README.md` for API reference
|
|
|
|
### MySQL Database (Internal Only)
|
|
- Stores all mailing list and member data
|
|
- Automatically initialized with schema
|
|
- Native Postfix integration for real-time queries
|
|
- See `database/README.md` for schema and management
|
|
|
|
### Postfix Mail Server (Port 25)
|
|
- Accepts mail for lists.sasalliance.org
|
|
- Queries MySQL for list expansion
|
|
- Relays through Amazon SES
|
|
- Sender whitelist: @sasalliance.org domain
|
|
- Changes take effect immediately (no restart needed)
|
|
|
|
## Configuration
|
|
|
|
### Managing Lists and Members
|
|
|
|
**Via Web Interface (Recommended):**
|
|
1. Open http://localhost:3000
|
|
2. Enter your API token
|
|
3. Use the intuitive interface to:
|
|
- Create/edit/delete mailing lists
|
|
- Add/edit/remove members
|
|
- Manage subscriptions with toggle switches
|
|
- View member counts and list details
|
|
|
|
**Via REST API:**
|
|
```bash
|
|
# Create a new list
|
|
curl -X POST http://localhost:8000/lists \
|
|
-H "Authorization: Bearer your_token" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"list_name": "Developers",
|
|
"list_email": "dev@lists.sasalliance.org",
|
|
"description": "Developer discussions",
|
|
"active": true
|
|
}'
|
|
|
|
# Add a new member
|
|
curl -X POST http://localhost:8000/members \
|
|
-H "Authorization: Bearer your_token" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"name": "John Doe",
|
|
"email": "john@example.com",
|
|
"active": true
|
|
}'
|
|
|
|
# Subscribe member to list
|
|
curl -X POST http://localhost:8000/subscriptions \
|
|
-H "Authorization: Bearer your_token" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"list_email": "dev@lists.sasalliance.org",
|
|
"member_email": "john@example.com"
|
|
}'
|
|
```
|
|
|
|
**Via Direct Database Access:**
|
|
```bash
|
|
# Connect to MySQL
|
|
docker-compose exec mysql mysql -u maillist -p maillist
|
|
|
|
# Run SQL queries (see database/README.md for examples)
|
|
```
|
|
|
|
### Mail Server Configuration
|
|
|
|
### Mail Server Configuration
|
|
|
|
The Postfix mail server is configured for:
|
|
- **Hostname**: `lists.sasalliance.org` (accepts mail for this domain)
|
|
- **Origin Domain**: `sasalliance.org`
|
|
- **SES Region**: EU West 2 (configurable via `SMTP_HOST`)
|
|
- **Sender Whitelist**: @sasalliance.org (only authorized board members can send to lists)
|
|
- **Dynamic Lists**: Postfix queries MySQL in real-time for list members
|
|
|
|
### Testing Mail Delivery
|
|
|
|
```bash
|
|
# From inside Postfix container
|
|
docker-compose exec postfix bash
|
|
echo "Test message" | mail -s "Test Subject" community@lists.sasalliance.org
|
|
|
|
# Check delivery logs
|
|
docker-compose logs -f postfix | grep -E "(sent|bounced|deferred)"
|
|
|
|
# Verify MySQL query works
|
|
docker-compose exec postfix postmap -q "community@lists.sasalliance.org" \
|
|
mysql:/etc/postfix/mysql_virtual_alias_maps.cf
|
|
```
|
|
|
|
### Bounce Handling (Optional)
|
|
|
|
**Email bounce handling is optional and disabled by default.**
|
|
|
|
**Two Configuration Options:**
|
|
|
|
1. **SNS Webhooks** (Requires SES Production Access):
|
|
```bash
|
|
# In .env file
|
|
ENABLE_SNS_WEBHOOKS=true
|
|
ENABLE_BOUNCE_HANDLING=true
|
|
```
|
|
- Real-time bounce notifications via AWS SNS
|
|
- Automatic member deactivation for hard bounces
|
|
- Bounce history tracking and management
|
|
- Requires valid HTTPS domain and SES production access
|
|
- See `BOUNCE_HANDLING_SETUP.md` for complete setup
|
|
|
|
2. **Email-Based Handling** (Default for SES Sandbox):
|
|
```bash
|
|
# In .env file (or leave these commented out)
|
|
ENABLE_SNS_WEBHOOKS=false
|
|
ENABLE_BOUNCE_HANDLING=false
|
|
```
|
|
- Manual bounce management via email notifications
|
|
- No automatic processing - requires manual member cleanup
|
|
- Works with SES sandbox accounts
|
|
- Bounce-related UI elements are hidden
|
|
|
|
**When bounce handling is disabled:**
|
|
- `/webhooks/sns` endpoint is not registered
|
|
- Bounce history endpoints return empty results
|
|
- Web UI hides bounce badges and bounce management buttons
|
|
- No automatic member deactivation occurs
|
|
|
|
## Security
|
|
|
|
- **Environment Variables**: All credentials stored in `.env` (git-ignored)
|
|
- **Private Network**: MySQL communicates with Postfix/API on internal Docker network only
|
|
- **Token Authentication**: API requires Bearer token for all write operations
|
|
- **Sender Whitelist**: Only @sasalliance.org addresses can send to mailing lists
|
|
- **TLS Encryption**: Enforced for SES relay connections
|
|
- **File Permissions**: SASL password files have restricted permissions (600)
|
|
- **No External MySQL**: Database is not exposed to host (no port mapping)
|
|
|
|
## Development
|
|
|
|
### Project Structure
|
|
```
|
|
├── docker-compose.yaml # Multi-service orchestration
|
|
├── .env # Environment configuration (not in git)
|
|
├── web/ # Web frontend (Nginx + HTML/CSS/JS)
|
|
│ ├── index.html
|
|
│ ├── static/
|
|
│ │ ├── css/style.css
|
|
│ │ └── js/
|
|
│ │ ├── api.js
|
|
│ │ ├── ui.js
|
|
│ │ └── app.js
|
|
│ ├── nginx.conf
|
|
│ └── Dockerfile
|
|
├── api/ # REST API (FastAPI)
|
|
│ ├── main.py
|
|
│ ├── requirements.txt
|
|
│ └── Dockerfile
|
|
├── database/ # MySQL schema and docs
|
|
│ ├── schema.sql
|
|
│ └── README.md
|
|
├── postfix/ # Mail server configuration
|
|
│ ├── Dockerfile
|
|
│ ├── entrypoint.sh
|
|
│ ├── main.cf.template
|
|
│ ├── mysql_virtual_alias_maps.cf
|
|
│ ├── sasl_passwd.template
|
|
│ └── sender_access
|
|
└── .github/
|
|
└── copilot-instructions.md # AI agent guidance
|
|
```
|
|
|
|
### Environment Variables
|
|
- `SES_USER`: AWS SES access key ID
|
|
- `SES_PASS`: AWS SES secret access key
|
|
- `SMTP_HOST`: SMTP server hostname (default: email-smtp.eu-west-2.amazonaws.com)
|
|
- `SMTP_PORT`: SMTP server port (default: 587)
|
|
- `MYSQL_ROOT_PASSWORD`: MySQL root password
|
|
- `MYSQL_DATABASE`: Database name (default: maillist)
|
|
- `MYSQL_USER`: MySQL user (default: maillist)
|
|
- `MYSQL_PASSWORD`: MySQL user password
|
|
- `API_TOKEN`: Bearer token for API authentication
|
|
|
|
### Service Dependencies
|
|
```
|
|
mysql (healthcheck) → postfix, api → web
|
|
```
|
|
- MySQL must be healthy before Postfix/API start
|
|
- Web frontend depends on API being available
|
|
- All services communicate on private Docker network
|
|
|
|
### Debugging
|
|
|
|
Monitor services:
|
|
```bash
|
|
# View all service logs
|
|
docker-compose logs -f
|
|
|
|
# View specific service
|
|
docker-compose logs -f postfix
|
|
docker-compose logs -f api
|
|
docker-compose logs -f web
|
|
docker-compose logs -f mysql
|
|
|
|
# Filter for mail delivery status
|
|
docker-compose logs postfix | grep -E "(sent|bounced|deferred)"
|
|
|
|
# Check Postfix queue
|
|
docker-compose exec postfix postqueue -p
|
|
|
|
# Test MySQL connectivity from Postfix
|
|
docker-compose exec postfix postmap -q "community@lists.sasalliance.org" \
|
|
mysql:/etc/postfix/mysql_virtual_alias_maps.cf
|
|
|
|
# Check API health
|
|
curl http://localhost:8000/health
|
|
|
|
# Access MySQL directly
|
|
docker-compose exec mysql mysql -u maillist -p maillist
|
|
```
|
|
|
|
### Development Workflow
|
|
|
|
1. **Make changes** to code or configuration
|
|
2. **Rebuild affected service**:
|
|
```bash
|
|
sudo docker-compose up --build -d web # Just web frontend
|
|
sudo docker-compose up --build -d api # Just API
|
|
sudo docker-compose up --build -d postfix # Just mail server
|
|
```
|
|
3. **Check logs** for errors:
|
|
```bash
|
|
docker-compose logs -f [service-name]
|
|
```
|
|
4. **Test changes** via web interface or API
|
|
|
|
### Common Tasks
|
|
|
|
**Reset database completely:**
|
|
```bash
|
|
docker-compose down -v # Removes volumes
|
|
docker-compose up -d # Reinitializes from schema.sql
|
|
```
|
|
|
|
**Update API token:**
|
|
1. Edit `.env` and change `API_TOKEN`
|
|
2. Restart API: `docker-compose restart api`
|
|
3. Use new token in web interface
|
|
|
|
**Add authorized sender domain:**
|
|
1. Edit `postfix/sender_access`
|
|
2. Add line: `newdomain.com OK`
|
|
3. Rebuild: `docker-compose up --build -d postfix`
|
|
|
|
## How It Works
|
|
|
|
1. **User manages lists** via web interface (http://localhost:3000)
|
|
2. **Web frontend** makes authenticated API calls to FastAPI backend
|
|
3. **API updates MySQL** database with new lists/members/subscriptions
|
|
4. **Email arrives** at Postfix for `someone@lists.sasalliance.org`
|
|
5. **Postfix queries MySQL** in real-time using native MySQL support
|
|
6. **MySQL returns** comma-separated list of active member emails
|
|
7. **Postfix expands** the alias and delivers to all members via SES
|
|
8. **Changes take effect immediately** - no restart or reload needed!
|
|
|
|
## Features
|
|
|
|
- ✅ **Real-time Updates** - Changes to lists/members take effect immediately
|
|
- ✅ **Native MySQL Integration** - Postfix queries database directly (no scripts)
|
|
- ✅ **Web Interface** - Modern, responsive UI for easy management
|
|
- ✅ **REST API** - Complete programmatic access with token auth
|
|
- ✅ **Sender Whitelist** - Only authorized domains can send to lists
|
|
- ✅ **SES Integration** - Reliable email delivery through Amazon SES
|
|
- ✅ **Bounce Handling** - Automatic tracking and management of email bounces via SNS
|
|
- ✅ **Secure** - Private Docker network, token auth, environment-based credentials
|
|
- ✅ **Flexible** - Manage via web, API, or direct database access
|
|
- ✅ **Scalable** - Database-driven architecture supports many lists and members
|
|
|
|
## Documentation
|
|
|
|
- **Web Interface**: See `web/README.md` for frontend features and usage
|
|
- **REST API**: See `api/README.md` for complete API reference
|
|
- **Database**: See `database/README.md` for schema and SQL examples
|
|
- **Bounce Handling**: See `BOUNCE_HANDLING_SETUP.md` for SNS configuration
|
|
- **AI Agents**: See `.github/copilot-instructions.md` for development guidance
|
|
|
|
## Roadmap
|
|
|
|
- [x] Web frontend for mailing list management
|
|
- [x] SQL database backend for member storage
|
|
- [x] Dynamic configuration with native Postfix MySQL
|
|
- [x] Multi-service Docker Compose architecture
|
|
- [x] REST API with authentication
|
|
- [x] Sender whitelist for authorized domains
|
|
- [x] Bounce handling with SES SNS integration
|
|
- [ ] Email verification workflow for new members
|
|
- [ ] Subscription confirmation (double opt-in)
|
|
- [ ] List archive functionality
|
|
- [ ] Unsubscribe links in emails
|
|
- [ ] Rate limiting and anti-spam measures
|
|
- [ ] HTTPS support for web interface
|
|
- [ ] Admin roles and permissions
|
|
|
|
## License
|
|
|
|
MIT License - see LICENSE file for details. |