RBAC and Doc updates

2025-10-13 14:05:01 +00:00
parent f721be7280
commit d4b88e0952
13 changed files with 1400 additions and 208 deletions
--- a/.github/copilot-instructions.md
+++ b/.github/copilot-instructions.md
@@ -1,97 +1,169 @@
-# Copilot Instructions: Mail List Manager (Postfix + SES + Web Interface)
+# Copilot Instructions: Mail List Manager (Postfix + SES + MySQL + API + Web)

 ## Architecture Overview

-This is a containerized mailing list management system built around Postfix as an SMTP relay through Amazon SES. **Currently in Phase 1** with static configuration; **planned expansion** includes web frontend and SQL database for dynamic member management.
+This is a complete containerized mailing list management system with four services:

-**Current Components (Phase 1):**
- `docker-compose.yaml`: Single-service orchestration with SES credentials
- `postfix/`: Complete Postfix container configuration  
- Static virtual aliases system for mailing list distribution
+1. **Web Frontend** (Port 3000) - Nginx serving modern HTML/CSS/JS interface
+2. **REST API** (Port 8000) - FastAPI backend with token authentication  
+3. **MySQL Database** (Internal) - Stores lists, members, and subscriptions
+4. **Postfix Mail Server** (Port 25) - SMTP server with native MySQL integration

-**Planned Architecture (Phase 2+):**
- Web frontend for list management (view/add/remove members)
- SQL database backend for member storage
- Dynamic Postfix configuration generation
- Multi-service Docker Compose setup
+**Key Architecture Principles:**
+- Real-time operation: Changes take effect immediately without Postfix reload
+- Native integration: Postfix queries MySQL directly (no Python scripts)
+- Security-first: Private Docker network, token auth, sender whitelist
+- User-friendly: Web interface for easy management by non-technical users

 ## Configuration Pattern

-**Current (Static):** Environment variable substitution for secure credential management:
+**Dynamic Database-Driven Architecture:**

-1. **Credentials Flow**: SES credentials passed via environment → `sasl_passwd.template` → runtime generation in `entrypoint.sh`
-2. **Config Files**: Static configs (`main.cf`, `virtual_aliases.cf`) + dynamic SASL auth file
-3. **Postfix Maps**: Hash databases generated at build time (virtual aliases) and runtime (SASL)
+1. **Data Flow**: Web UI → REST API → MySQL ← Postfix (real-time queries)
+2. **Credentials**: All sensitive data in `.env` file (SES, MySQL, API_TOKEN)
+3. **Mail Processing**: 
+   - Email arrives for `list@lists.sasalliance.org`
+   - Postfix queries MySQL via `mysql_virtual_alias_maps.cf`
+   - MySQL returns comma-separated member emails
+   - Postfix expands alias and delivers via SES
+4. **Security**: Private Docker network (maillist-internal), sender whitelist (@sasalliance.org)

-**Future (Dynamic):** Database-driven configuration:
- Member lists stored in SQL database, with members able to join multiple lists
- Web interface for CRUD operations on members
- `virtual_aliases.cf` generated from database at runtime
- Postfix reload triggered by configuration changes
+**No Static Configuration:**
+- Virtual aliases are stored in MySQL database, not config files
+- Changes take effect immediately without container restarts
+- Postfix caches query results for performance

 ## Key Files and Their Roles

- `main.cf`: Core Postfix config - relay through SES, domain settings, security
- `sasl_passwd.template`: Template for SES authentication (uses `${SES_USER}:${SES_PASS}`)
- `virtual_aliases.cf`: Static email forwarding rules (one mailing list currently)
- `entrypoint.sh`: Runtime credential processing and Postfix startup
+### Docker & Environment
+- `docker-compose.yaml`: Multi-service orchestration (mysql, postfix, api, web)
+- `.env`: All credentials and configuration (SES, MySQL, API_TOKEN)
+
+### Web Frontend (`web/`)
+- `index.html`: Single-page application interface
+- `static/js/api.js`: API client with authentication
+- `static/js/ui.js`: UI helpers and modal management
+- `static/js/app.js`: Main application controller
+- `static/css/style.css`: Complete styling system
+- `nginx.conf`: Nginx configuration for static file serving
+
+### REST API (`api/`)
+- `main.py`: FastAPI application with all CRUD endpoints
+- `requirements.txt`: Python dependencies (fastapi, mysql-connector-python, etc.)
+- `Dockerfile`: Python 3.11 container build
+
+### Database (`database/`)
+- `schema.sql`: Database initialization with tables and sample data
+- Tables: `lists`, `members`, `list_members` (many-to-many junction)
+
+### Postfix (`postfix/`)
+- `main.cf.template`: Core Postfix config with MySQL virtual alias maps
+- `mysql_virtual_alias_maps.cf`: Native Postfix MySQL query configuration
+- `sender_access`: Sender whitelist (sasalliance.org domain allowed)
+- `sasl_passwd.template`: SES authentication template
+- `entrypoint.sh`: Container startup with MySQL health check and config generation

 ## Development Workflows

 **Building and Running:**
+
+Always use sudo for Docker commands. Don't bother opening the simple browser UI for Docker as it is not very useful.
+
 ```bash
-docker-compose up --build    # Build and start mail server
-docker-compose logs -f       # Monitor mail delivery logs
+sudo docker-compose up --build    # Build and start all services
+sudo docker-compose logs -f       # Monitor all service logs
+sudo docker-compose logs -f api   # Monitor specific service
 ```

-**Adding Mailing Lists (Current):**
-1. Edit `virtual_aliases.cf` with new list → recipients mapping
-2. Rebuild container (postmap runs at build time)
-3. No restart needed for existing virtual alias changes
+**Managing Lists and Members:**

-**Future Web Interface Workflow:**
-1. Access web frontend at configured port
-2. Use CRUD interface to manage mailing lists and members
-3. Changes automatically update database and regenerate Postfix configs
+**Via Web Interface (Recommended):**
+1. Open http://localhost:3000
+2. Enter API_TOKEN from .env file
+3. Use intuitive interface to manage lists, members, and subscriptions
+4. Click "Subscriptions" button on any member for toggle-based subscription management
+
+**Via REST API:**
+```bash
+# Get all lists
+curl -H "Authorization: Bearer $API_TOKEN" http://localhost:8000/lists
+
+# Create new list
+curl -X POST http://localhost:8000/lists \
+  -H "Authorization: Bearer $API_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"list_name":"Test","list_email":"test@lists.sasalliance.org","active":true}'
+
+# Subscribe member to list
+curl -X POST http://localhost:8000/subscriptions \
+  -H "Authorization: Bearer $API_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"list_email":"test@lists.sasalliance.org","member_email":"user@example.com"}'
+```
+
+**Via Direct Database Access:**
+```bash
+docker-compose exec mysql mysql -u maillist -p maillist
+# See database/README.md for SQL examples
+```

 **Testing Mail Delivery:**
 ```bash
 # From inside container
+docker-compose exec postfix bash
 echo "Test message" | mail -s "Subject" community@lists.sasalliance.org

 # Check logs for SES relay status
 docker-compose logs postfix | grep -E "(sent|bounced|deferred)"
+
+# Verify MySQL query works
+docker-compose exec postfix postmap -q "community@lists.sasalliance.org" \
+  mysql:/etc/postfix/mysql_virtual_alias_maps.cf
 ```

 ## Security Considerations

- SES credentials exposed in docker-compose.yaml (consider using Docker secrets)
+- All credentials stored in `.env` file (git-ignored)
+- MySQL on private Docker network (no host port mapping)
+- API requires Bearer token authentication
+- Sender whitelist restricts who can send to lists (@sasalliance.org only)
 - SASL password file permissions set to 600 in entrypoint
 - TLS encryption enforced for SES relay (`smtp_tls_security_level = encrypt`)
- Only localhost and configured hostname accepted for local delivery
+- Web interface requires token for all operations

 ## Configuration Conventions

 - **Hostname Pattern**: `lists.sasalliance.org` for mailing lists, origin domain `sasalliance.org`
- **Virtual Aliases**: Simple format `listname@lists.domain → recipient1, recipient2`
+- **Virtual Aliases**: Dynamically queried from MySQL database in real-time
 - **SES Region**: EU West 2 (`email-smtp.eu-west-2.amazonaws.com`)
 - **Port Mapping**: Standard SMTP port 25 exposed to host
+- **Database Tables**: `lists`, `members`, `list_members` (many-to-many design)
+- **API Port**: 8000 for REST API
+- **Web Port**: 3000 for web interface

 ## Common Modifications

-**Adding Recipients to Existing List (Current):**
-Edit `virtual_aliases.cf`, add comma-separated emails, rebuild container.
+**Adding Recipients to a List:**
+Use web interface or API - changes are immediate!

-**New Mailing List (Current):**
-Add line to `virtual_aliases.cf`: `newlist@lists.sasalliance.org recipients...`
+**Creating New Mailing List:**
+1. Via Web: Click "Add List" button, fill in form
+2. Via API: POST to /lists endpoint with list details
+3. Via MySQL: INSERT into lists table
+
+**Adding New Member:**
+1. Via Web: Click "Add Member" button in Members tab
+2. Via API: POST to /members endpoint
+3. Via MySQL: INSERT into members table
+
+**Managing Subscriptions:**
+1. Via Web: Click "Subscriptions" button on member, toggle lists
+2. Via API: POST/DELETE to /subscriptions endpoint
+3. Via MySQL: INSERT/DELETE in list_members table

 **Credential Updates:**
-Update `SES_USER`/`SES_PASS` in docker-compose.yaml, restart container.
-
-## Migration Considerations
-
-When implementing the web frontend and database backend:
- Preserve existing `community@lists.sasalliance.org` list during migration
- Consider migration script to import current virtual aliases into database
- Plan for zero-downtime deployment with database-driven config generation
- Web interface should validate email addresses and handle duplicate members
+1. Edit `.env` file with new credentials
+2. Restart affected services:
+   - SES: `docker-compose restart postfix`
+   - MySQL: `docker-compose restart mysql`
+   - API: `docker-compose restart api`