Doc update and SQL init

2025-10-25 15:10:54 +00:00
parent 7643c179fb
commit 77b5080bbd
10 changed files with 417 additions and 48 deletions
--- a/.copilot-instructions.md
+++ b/.copilot-instructions.md
@@ -2,7 +2,17 @@

 General Information

-As you perform more tests, document in this file the steps you are taking to avoid repetition. 
+As you perform more tests, document in this file the steps you are taking to avoid repetition.
+
+## System Overview
+
+The NextGen PPR system includes:
+- **FastAPI Backend** with comprehensive REST API
+- **MySQL Database** with full PPR tracking and audit trails
+- **Web Interfaces** for public submission, admin management, and reporting
+- **Email Notifications** for PPR submissions and cancellations
+- **Real-time Updates** via WebSocket for live tower operations
+- **Test Data Generation** utilities for development and testing

 ## API Base URL
 - Development: `http://localhost:8001/api/v1`
@@ -145,6 +155,58 @@ curl -X POST "http://localhost:8001/api/v1/pprs/" \

 **Endpoint:** `DELETE /api/v1/pprs/{ppr_id}` (soft delete - sets status to DELETED)

+## User Management Endpoints (Admin Only)
+
+### List Users
+
+**Endpoint:** `GET /api/v1/auth/users`
+
+### Create User
+
+**Endpoint:** `POST /api/v1/auth/users`
+
+**Request Body:**
+```json
+{
+  "username": "newuser",
+  "password": "securepassword",
+  "role": "OPERATOR"
+}
+```
+
+**Available Roles:**
+- `ADMINISTRATOR`: Full access including user management
+- `OPERATOR`: PPR management access
+- `READ_ONLY`: View-only access
+
+### Get User Details
+
+**Endpoint:** `GET /api/v1/auth/users/{user_id}`
+
+### Update User
+
+**Endpoint:** `PUT /api/v1/auth/users/{user_id}`
+
+### Delete User
+
+**Endpoint:** `DELETE /api/v1/auth/users/{user_id}`
+
+## Reference Data Endpoints
+
+### Lookup Airport
+
+**Endpoint:** `GET /api/v1/airport/lookup/{icao_code}`
+
+Returns airport details for the given ICAO code.
+
+### Lookup Aircraft
+
+**Endpoint:** `GET /api/v1/aircraft/lookup/{registration}`
+
+Returns aircraft details for the given registration.
+
+## Reference Data Endpoints
+
 ## Public Endpoints (No Authentication Required)

 ### Get Today's Arrivals
@@ -159,6 +221,40 @@ Returns all PPRs with status "NEW" and ETA for today.

 Returns all PPRs with status "LANDED" and ETD for today.

+### Submit Public PPR
+
+**Endpoint:** `POST /api/v1/public/pprs/`
+
+**Headers:**
+- `Content-Type: application/json`
+
+**Request Body:** Same as authenticated PPR creation (see above)
+
+**Notes:** 
+- No authentication required
+- Sends confirmation email if email provided
+- Returns PPR with public edit token
+
+### Get PPR for Public Editing
+
+**Endpoint:** `GET /api/v1/public/edit/{token}`
+
+Returns PPR details for public editing using the token from email.
+
+### Update PPR Publicly
+
+**Endpoint:** `PATCH /api/v1/public/edit/{token}`
+
+**Request Body:** Same as authenticated PPR update
+
+**Notes:** Only allowed if PPR status is not processed (not LANDED, DEPARTED, CANCELED, or DELETED)
+
+### Cancel PPR Publicly
+
+**Endpoint:** `DELETE /api/v1/public/cancel/{token}`
+
+Cancels PPR using public token and sends cancellation email.
+
 ## Data Models

 ### PPR Status Enum
@@ -256,10 +352,14 @@ The system automatically logs:

 ## Web Interfaces

-### Public Arrivals/Departures Board
+### Public PPR Forms
 - **URL:** http://localhost:8082
- **Features:** Real-time arrivals and departures display
- **Authentication:** None required
+- **Features:** 
+  - PPR submission with intelligent aircraft/airport field lookups
+  - Date/time pickers with 15-minute intervals
+  - Email notifications for submissions
+  - Public editing/cancellation via secure email tokens
+  - Real-time validation and feedback

 ### Admin Interface
 - **URL:** http://localhost:8082/admin.html
@@ -270,15 +370,80 @@ The system automatically logs:
  - Journal/audit trail viewing
  - Quick status updates (Confirm, Land, Depart, Cancel)
  - New PPR entry creation
+  - User management (administrators only)
+  - Real-time WebSocket updates
 - **Authentication:** Username/password prompt (uses API token)

+### Reports Interface
+- **URL:** http://localhost:8082/reports.html
+- **Features:**
+  - Comprehensive PPR reporting with date range filtering (defaults to current month)
+  - Search across aircraft registration, callsign, captain name, and airports
+  - Status-based filtering
+  - CSV and XLS export functionality
+  - Responsive table displaying all PPR fields
+  - Real-time data loading with authentication
+
 ## Development URLs

 - API Base: http://localhost:8001/api/v1
 - Public Web Interface: http://localhost:8082
 - Admin Interface: http://localhost:8082/admin.html
+- Reports Interface: http://localhost:8082/reports.html
 - API Documentation: http://localhost:8001/docs
 - Database: localhost:3307 (MySQL)
+- phpMyAdmin: http://localhost:8083
+
+## Email System
+
+The system includes transactional email support:
+
+### Configuration
+Email settings are configured via environment variables:
+- `SMTP_SERVER`: SMTP server hostname
+- `SMTP_PORT`: SMTP server port (default: 587)
+- `SMTP_USERNAME`: SMTP authentication username
+- `SMTP_PASSWORD`: SMTP authentication password
+- `FROM_EMAIL`: Sender email address
+- `BASE_URL`: Base URL for email links
+
+### Email Templates
+- **PPR Submitted**: Sent when public PPR is created
+- **PPR Cancelled**: Sent when PPR is cancelled (by user or admin)
+
+### Email Content
+Emails include:
+- PPR details (aircraft, times, contact info)
+- Secure edit/cancel links with tokens
+- Formatted HTML with system branding
+
+## Test Data Generation
+
+The system includes comprehensive test data generation utilities:
+
+### Generate Test PPRs
+```bash
+# Run from host
+./populate_test_data.sh
+
+# Or run directly in container
+docker exec -it ppr_nextgen_api python populate_test_data.py
+```
+
+### What It Creates
+- **93 PPR records** across all statuses (NEW, CONFIRMED, LANDED, DEPARTED, CANCELED)
+- **Diverse aircraft types** from the loaded aircraft database
+- **Realistic airport combinations** from the loaded airport data
+- **Varied captains and contact details**
+- **Proper date/time distributions** across different time periods
+- **Status distribution** matching real-world usage patterns
+
+### Test Data Features
+- Aircraft registrations with proper formatting
+- Realistic passenger counts and fuel requirements
+- Geographic distribution across UK airports
+- Time-based distribution for testing date filters
+- Email addresses for testing notification system

 ## Example Workflow

@@ -309,4 +474,90 @@ curl -X PATCH "http://localhost:8001/api/v1/pprs/1/status" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"status":"LANDED"}'
-```
+
+# Test public PPR submission (no auth required)
+curl -X POST "http://localhost:8001/api/v1/public/pprs/" \
+  -H "Content-Type: application/json" \
+  -d '{"ac_reg":"G-PUBLIC","ac_type":"PA28","captain":"Public User","in_from":"EGGW","eta":"2025-10-21T15:00:00","pob_in":1,"email":"test@example.com"}'
+
+# Test filtering by date range
+curl -s "http://localhost:8001/api/v1/pprs/?date_from=2025-10-01&date_to=2025-10-31&limit=10" \
+  -H "Authorization: Bearer $TOKEN" | jq .
+
+# Test user management (admin only)
+curl -s "http://localhost:8001/api/v1/auth/users" \
+  -H "Authorization: Bearer $TOKEN" | jq .
+
+# Generate test data
+docker exec -it ppr_nextgen_api python populate_test_data.py
+```
+
+## Enhanced Features
+
+### Intelligent Form Lookups
+- **Aircraft Lookup**: Auto-suggests aircraft type based on registration
+- **Airport Lookup**: Provides airport name and location for ICAO codes
+- **Real-time Validation**: Immediate feedback on form inputs
+
+### Advanced Date/Time Handling
+- **15-Minute Intervals**: All time pickers use 15-minute increments
+- **Separate Date/Time Fields**: Improved UX with dedicated date and time inputs
+- **UTC Storage**: All times stored in UTC for consistency
+
+### Comprehensive Reporting
+- **Date Range Filtering**: Filter PPRs by custom date ranges
+- **Multi-field Search**: Search across registration, callsign, captain, airports
+- **Export Functionality**: CSV and XLS download with all PPR data
+- **Responsive Design**: Works on desktop and mobile devices
+
+### Audit Trail System
+- **Complete Change History**: Every field change is logged
+- **User Tracking**: Who made changes and when
+- **IP Address Logging**: Security tracking for all actions
+- **Automatic Journal Entries**: No manual logging required
+
+## Example Complete Workflow
+
+1. **Public Submission**
+   ```bash
+   # User submits PPR via public form
+   curl -X POST "http://localhost:8001/api/v1/public/pprs/" \
+     -H "Content-Type: application/json" \
+     -d '{"ac_reg":"G-WORKFLOW","ac_type":"C152","captain":"Workflow Test","in_from":"EGKB","eta":"2025-10-21T14:00:00","pob_in":1,"email":"workflow@example.com"}'
+   ```
+
+2. **Email Notification Sent**
+   - User receives confirmation email with edit/cancel links
+
+3. **Admin Processing**
+   ```bash
+   # Admin confirms PPR
+   curl -X PATCH "http://localhost:8001/api/v1/pprs/1/status" \
+     -H "Content-Type: application/json" \
+     -H "Authorization: Bearer $TOKEN" \
+     -d '{"status":"CONFIRMED"}'
+   ```
+
+4. **Status Updates**
+   ```bash
+   # Mark as landed
+   curl -X PATCH "http://localhost:8001/api/v1/pprs/1/status" \
+     -H "Content-Type: application/json" \
+     -H "Authorization: Bearer $TOKEN" \
+     -d '{"status":"LANDED","timestamp":"2025-10-21T14:05:00Z"}'
+   
+   # Mark as departed
+   curl -X PATCH "http://localhost:8001/api/v1/pprs/1/status" \
+     -H "Content-Type: application/json" \
+     -H "Authorization: Bearer $TOKEN" \
+     -d '{"status":"DEPARTED","timestamp":"2025-10-21T15:30:00Z"}'
+   ```
+
+5. **Public Editing/Cancellation**
+   - User can edit or cancel using token from email
+   - System sends cancellation email if cancelled
+
+6. **Reporting**
+   - Access reports at http://localhost:8082/reports.html
+   - Filter by date range, status, search terms
+   - Export to CSV/XLS for analysis
--- a/.env.example
+++ b/.env.example
@@ -0,0 +1,22 @@
+# Database Configuration
+MYSQL_ROOT_PASSWORD=your_mysql_root_password_here
+DB_USER=ppr_user
+DB_PASSWORD=your_database_password_here
+DB_NAME=ppr
+DB_PORT=3306
+
+# API Configuration
+DB_HOST=db
+SECRET_KEY=your_secret_key_here_change_in_production
+ACCESS_TOKEN_EXPIRE_MINUTES=30
+API_V1_STR=/api/v1
+PROJECT_NAME=Airfield PPR API NextGen
+API_PORT_EXTERNAL=8001
+
+# Web Configuration
+WEB_PORT_EXTERNAL=8082
+
+# phpMyAdmin Configuration
+PMA_HOST=db
+UPLOAD_LIMIT=50M
+PMA_PORT_EXTERNAL=8083
--- a/README.md
+++ b/README.md
@@ -12,12 +12,16 @@ A modern, containerized Prior Permission Required (PPR) system for aircraft oper
 ## Features

 - 🚀 **Modern API**: RESTful API with automatic OpenAPI documentation
- 🔐 **Authentication**: JWT-based authentication system
- 📊 **Real-time Updates**: WebSocket support for live tower updates
- 🗄️ **Self-contained**: Fully dockerized with local database
+- 🔐 **Authentication**: JWT-based authentication system with role-based access
+- <EFBFBD> **Transactional Email**: SMTP-based email notifications for PPR submissions and cancellations
+- <EFBFBD>📊 **Real-time Updates**: WebSocket support for live tower updates
+- <20> **Comprehensive Reporting**: Advanced reporting with filtering, search, and CSV/XLS export
+- <20>🗄️ **Self-contained**: Fully dockerized with local database
 - 🔍 **Documentation**: Auto-generated API docs at `/docs`
- 🧪 **Testing**: Comprehensive test suite
+- 🧪 **Testing**: Comprehensive test suite with data population utilities
 - 📱 **Mobile Ready**: Responsive design for tower operations
+- 🔄 **Public Forms**: Enhanced public PPR submission and editing with intelligent field lookups
+- 📋 **Audit Trail**: Complete journal system tracking all PPR changes

 ## Quick Start

@@ -33,7 +37,11 @@ cd nextgen
 ### 2. Access the Services
 - **API Documentation**: http://localhost:8001/docs
 - **API Base URL**: http://localhost:8001/api/v1
+- **Public Web Interface**: http://localhost:8082
+- **Admin Interface**: http://localhost:8082/admin.html
+- **Reports Interface**: http://localhost:8082/reports.html
 - **Database**: localhost:3307 (user: ppr_user, password: ppr_password123)
+- **phpMyAdmin**: http://localhost:8083

 ### 3. Default Login
 - **Username**: admin
@@ -45,20 +53,68 @@ cd nextgen
 - `POST /api/v1/auth/login` - Login and get JWT token

 ### PPR Management
- `GET /api/v1/pprs` - List PPR records
+- `GET /api/v1/pprs` - List PPR records (with filtering by status, date range)
 - `POST /api/v1/pprs` - Create new PPR
 - `GET /api/v1/pprs/{id}` - Get specific PPR
 - `PUT /api/v1/pprs/{id}` - Update PPR
+- `PATCH /api/v1/pprs/{id}` - Partially update PPR
 - `PATCH /api/v1/pprs/{id}/status` - Update PPR status
 - `DELETE /api/v1/pprs/{id}` - Delete PPR
+- `GET /api/v1/pprs/{id}/journal` - Get PPR activity journal

 ### Public Endpoints (No Auth Required)
 - `GET /api/v1/public/arrivals` - Today's arrivals
 - `GET /api/v1/public/departures` - Today's departures
+- `POST /api/v1/public/pprs` - Submit public PPR
+- `GET /api/v1/public/edit/{token}` - Get PPR for public editing
+- `PATCH /api/v1/public/edit/{token}` - Update PPR publicly
+- `DELETE /api/v1/public/cancel/{token}` - Cancel PPR publicly
+
+### User Management (Admin Only)
+- `GET /api/v1/auth/users` - List users
+- `POST /api/v1/auth/users` - Create user
+- `GET /api/v1/auth/users/{id}` - Get user details
+- `PUT /api/v1/auth/users/{id}` - Update user
+- `DELETE /api/v1/auth/users/{id}` - Delete user
+
+### Reference Data
+- `GET /api/v1/airport/lookup/{code}` - Lookup airport by ICAO code
+- `GET /api/v1/aircraft/lookup/{reg}` - Lookup aircraft by registration

 ### Real-time
 - `WebSocket /ws/tower-updates` - Live updates for tower operations

+## Web Interfaces
+
+### Public PPR Forms
+- **URL**: http://localhost:8082
+- **Features**:
+  - PPR submission form with intelligent aircraft/airport lookups
+  - Date/time pickers with 15-minute intervals
+  - Email notifications for submissions
+  - Public editing/cancellation via secure tokens
+
+### Admin Interface
+- **URL**: http://localhost:8082/admin.html
+- **Features**:
+  - Complete PPR management (CRUD operations)
+  - Advanced filtering by status, date range
+  - Inline editing with modal interface
+  - Journal/audit trail viewing
+  - Quick status updates (Confirm, Land, Depart, Cancel)
+  - New PPR entry creation
+  - User management (administrators only)
+  - Real-time WebSocket updates
+
+### Reports Interface
+- **URL**: http://localhost:8082/reports.html
+- **Features**:
+  - Comprehensive PPR reporting with date range filtering
+  - Search across aircraft, captain, and airport fields
+  - Status-based filtering
+  - CSV and XLS export functionality
+  - Responsive table with all PPR details
+
 ## Development

 ### Local Development
@@ -87,6 +143,33 @@ cd backend
 pytest tests/
 ```

+## Additional Features
+
+### Email Notifications
+The system includes transactional email support for:
+- **PPR Submissions**: Automatic email confirmation to submitters
+- **PPR Cancellations**: Notification emails when PPRs are cancelled
+- **SMTP Configuration**: Configurable SMTP settings via environment variables
+
+### Test Data Generation
+A comprehensive test data population script is included:
+```bash
+# Generate test PPR records
+docker exec -it ppr_nextgen_api python populate_test_data.py
+
+# Or run the convenience script
+./populate_test_data.sh
+```
+
+This creates diverse PPR records across all statuses with realistic aircraft and airport data for testing purposes.
+
+### Audit Trail
+Complete activity logging system tracks:
+- All PPR field changes with before/after values
+- Status transitions with timestamps
+- User actions and IP addresses
+- Automatic journal entries for all modifications
+
 ## Environment Variables

 Key environment variables (configured in docker-compose.yml):
@@ -98,6 +181,15 @@ Key environment variables (configured in docker-compose.yml):
 - `SECRET_KEY` - JWT secret key
 - `ACCESS_TOKEN_EXPIRE_MINUTES` - Token expiration time

+### Email Configuration
+- `SMTP_SERVER` - SMTP server hostname
+- `SMTP_PORT` - SMTP server port (default: 587)
+- `SMTP_USERNAME` - SMTP authentication username
+- `SMTP_PASSWORD` - SMTP authentication password
+- `SMTP_TLS` - Enable TLS (default: true)
+- `FROM_EMAIL` - Sender email address
+- `BASE_URL` - Base URL for email links
+
 ## Database Schema

 The system uses an improved version of the original schema with:
--- a/db-init/02-import-data.sql
+++ b/db-init/02-import-data.sql
--- a/db-init/Dockerfile
+++ b/db-init/Dockerfile
@@ -0,0 +1,9 @@
+FROM mysql:8.0
+
+# Copy initialization SQL files
+COPY init_db.sql /docker-entrypoint-initdb.d/01-schema.sql
+COPY 02-import-data.sql /docker-entrypoint-initdb.d/02-import-data.sql
+
+# Copy CSV files for import
+COPY airports_data_clean.csv /var/lib/mysql-files/airports_data.csv
+COPY aircraft_data.csv /var/lib/mysql-files/aircraft_data.csv
--- a/db-init/aircraft_data.csv
+++ b/db-init/aircraft_data.csv
--- a/db-init/airports_data_clean.csv
+++ b/db-init/airports_data_clean.csv
--- a/db-init/init_db.sql
+++ b/db-init/init_db.sql
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1,26 +1,19 @@
 version: '3.8'

 services:
-  # MySQL Database
  db:
-    image: mysql:8.0
+    build: ./db-init
    container_name: ppr_nextgen_db
    restart: unless-stopped
    environment:
-      MYSQL_ROOT_PASSWORD: rootpassword123
-      MYSQL_DATABASE: ppr_nextgen
-      MYSQL_USER: ppr_user
-      MYSQL_PASSWORD: ppr_password123
-    ports:
-      - "3307:3306"  # Use different port to avoid conflicts
+      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
+      MYSQL_DATABASE: ${DB_NAME}
+      MYSQL_USER: ${DB_USER}
+      MYSQL_PASSWORD: ${DB_PASSWORD}
    volumes:
      - mysql_data:/var/lib/mysql
-      - ./init_db.sql:/docker-entrypoint-initdb.d/01-schema.sql
-      - ./02-import-data.sql:/docker-entrypoint-initdb.d/02-import-data.sql
-      - ./airports_data_clean.csv:/var/lib/mysql-files/airports_data.csv
-      - ./aircraft_data.csv:/var/lib/mysql-files/aircraft_data.csv
    networks:
-      - ppr_network
+      - private_network

  # FastAPI Backend
  api:
@@ -28,23 +21,24 @@ services:
    container_name: ppr_nextgen_api
    restart: unless-stopped
    environment:
-      DB_HOST: db
-      DB_USER: ppr_user
-      DB_PASSWORD: ppr_password123
-      DB_NAME: ppr_nextgen
-      DB_PORT: 3306
-      SECRET_KEY: super-secret-key-for-nextgen-ppr-system-change-in-production
-      ACCESS_TOKEN_EXPIRE_MINUTES: 30
-      API_V1_STR: /api/v1
-      PROJECT_NAME: "Airfield PPR API NextGen"
+      DB_HOST: ${DB_HOST}
+      DB_USER: ${DB_USER}
+      DB_PASSWORD: ${DB_PASSWORD}
+      DB_NAME: ${DB_NAME}
+      DB_PORT: ${DB_PORT}
+      SECRET_KEY: ${SECRET_KEY}
+      ACCESS_TOKEN_EXPIRE_MINUTES: ${ACCESS_TOKEN_EXPIRE_MINUTES}
+      API_V1_STR: ${API_V1_STR}
+      PROJECT_NAME: ${PROJECT_NAME}
    ports:
-      - "8001:8000"  # Use different port to avoid conflicts with existing system
+      - "${API_PORT_EXTERNAL}:8000"  # Use different port to avoid conflicts with existing system
    depends_on:
      - db
    volumes:
      - ./backend:/app
    networks:
-      - ppr_network
+      - private_network
+      - public_network
    command: uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

  # Redis for caching (optional for now)
@@ -52,10 +46,8 @@ services:
    image: redis:7-alpine
    container_name: ppr_nextgen_redis
    restart: unless-stopped
-    ports:
-      - "6380:6379"  # Use different port
    networks:
-      - ppr_network
+      - private_network

  # Nginx web server for public frontend
  web:
@@ -63,14 +55,14 @@ services:
    container_name: ppr_nextgen_web
    restart: unless-stopped
    ports:
-      - "8082:80"  # Public web interface
+      - "${WEB_PORT_EXTERNAL}:80"  # Public web interface
    volumes:
      - ./web:/usr/share/nginx/html
      - ./nginx.conf:/etc/nginx/nginx.conf
    depends_on:
      - api
    networks:
-      - ppr_network
+      - public_network

  # phpMyAdmin for database management
  phpmyadmin:
@@ -78,21 +70,24 @@ services:
    container_name: ppr_nextgen_phpmyadmin
    restart: unless-stopped
    environment:
-      PMA_HOST: db
-      PMA_PORT: 3306
-      PMA_USER: ppr_user
-      PMA_PASSWORD: ppr_password123
-      UPLOAD_LIMIT: 50M
+      PMA_HOST: ${PMA_HOST}
+      PMA_PORT: ${DB_PORT}
+      PMA_USER: ${DB_USER}
+      PMA_PASSWORD: ${DB_PASSWORD}
+      UPLOAD_LIMIT: ${UPLOAD_LIMIT}
    ports:
-      - "8083:80"  # phpMyAdmin web interface
+      - "${PMA_PORT_EXTERNAL}:80"  # phpMyAdmin web interface
    depends_on:
      - db
    networks:
-      - ppr_network
+      - private_network
+      - public_network

 volumes:
  mysql_data:

 networks:
-  ppr_network:
+  private_network:
+    driver: bridge
+  public_network:
    driver: bridge
--- a/populate_test_data.sh
+++ b/populate_test_data.sh
@@ -11,6 +11,6 @@ if [ ! -f "docker-compose.yml" ]; then
 fi

 # Run the population script in the backend container
-docker-compose exec backend python populate_test_data.py
+sudo docker compose exec api python populate_test_data.py

 echo "Test data population complete!"