451 lines
17 KiB
Markdown
451 lines
17 KiB
Markdown
# backupmult.ps1 - Parallel SQL Server Database Backup Script
|
|
|
|
A PowerShell script that performs parallel database backups using Ola Hallengren's DatabaseBackup stored procedure with the `DatabasesInParallel` feature for optimal performance and load distribution.
|
|
|
|
## Features
|
|
|
|
- **Parallel Processing**: Utilizes Ola Hallengren's `DatabasesInParallel=Y` feature to automatically distribute database backup workloads across multiple concurrent jobs
|
|
- **Smart Backup Type Detection**: Automatically determines whether to run FULL, DIFFERENTIAL, or LOG backups based on schedule and overdue status
|
|
- **Multi-Directory Support**: Supports striped backups across multiple directories for improved I/O performance
|
|
- **Comprehensive Logging**: Thread-safe logging with job-specific log files that are consolidated into a main log
|
|
- **Error Handling**: Robust error detection and reporting with detailed SQL error information
|
|
- **Job Monitoring**: Real-time monitoring of backup job progress and completion status
|
|
- **Mount Point Validation**: Optional mount point checking to ensure backup destinations are accessible
|
|
|
|
## Requirements
|
|
|
|
### System Requirements
|
|
- **PowerShell**: Version 5.1 or higher
|
|
- **Operating System**: Windows Server 2016 or later (or Windows 10/11 for development/testing)
|
|
- **SQL Server**: SQL Server 2016 or later (Express, Standard, Enterprise, or Developer editions)
|
|
- **Permissions**: SQL Server sysadmin privileges or appropriate database backup permissions
|
|
|
|
### Software Dependencies
|
|
- **Ola Hallengren Maintenance Solution**: Must be installed on the target SQL Server instance
|
|
- Download from: https://ola.hallengren.com/
|
|
- Install the DatabaseBackup stored procedure
|
|
- **SQL Server PowerShell Module**: Either `SqlServer` or `SQLPS` module must be available
|
|
- Install with: `Install-Module -Name SqlServer -AllowClobber`
|
|
|
|
### Network Requirements
|
|
- **SQL Server Connectivity**: The script must be able to connect to the target SQL Server instance
|
|
- **Backup Destination Access**: Read/write access to all specified backup directories
|
|
- **Mount Points**: If using mount points, they must be accessible and have sufficient space
|
|
|
|
## Installation
|
|
|
|
1. **Download the Script**:
|
|
```powershell
|
|
# Place backupmult.ps1 in your preferred scripts directory
|
|
# Example: C:\Rubrik\Scripts\
|
|
```
|
|
|
|
2. **Install Ola Hallengren Maintenance Solution**:
|
|
```sql
|
|
-- Run on your SQL Server instance
|
|
-- Download from https://ola.hallengren.com/
|
|
-- Execute the installation script
|
|
```
|
|
|
|
3. **Install SQL Server PowerShell Module** (if not already installed):
|
|
```powershell
|
|
Install-Module -Name SqlServer -AllowClobber -Force
|
|
```
|
|
|
|
4. **Verify Permissions**:
|
|
- Ensure the account running the script has SQL Server sysadmin privileges
|
|
- Verify write access to backup directories and log file locations
|
|
|
|
## Usage
|
|
|
|
### Basic Syntax
|
|
```powershell
|
|
.\backupmult.ps1 -SqlInstance "ServerName\InstanceName" [parameters]
|
|
```
|
|
|
|
### Parameters
|
|
|
|
| Parameter | Type | Required | Default | Description |
|
|
|-----------|------|----------|---------|-------------|
|
|
| `SqlInstance` | String | Yes | - | SQL Server instance name (e.g., "SERVER01\SQL2019") |
|
|
| `Directories` | String | No | Auto-generated | Comma-separated list of backup directories |
|
|
| `Jobs` | Integer | No | 2 | Number of parallel backup jobs (1-8) |
|
|
| `Force` | Switch | No | False | Skip mount point accessibility checks |
|
|
|
|
### Examples
|
|
|
|
#### Basic Usage with Default Settings
|
|
```powershell
|
|
.\backupmult.ps1 -SqlInstance "PRODSQL01\SQL2019"
|
|
```
|
|
- Uses 2 parallel jobs
|
|
- Auto-generates 4 backup directories under `C:\Rubrik\SQL2019\`
|
|
- Performs appropriate backup type based on schedule
|
|
|
|
#### Custom Directories and Job Count
|
|
```powershell
|
|
.\backupmult.ps1 -SqlInstance "TESTSQL\SQL2022" -Directories "D:\Backups\Dir1, E:\Backups\Dir2, F:\Backups\Dir3" -Jobs 3
|
|
```
|
|
- Uses 3 parallel jobs
|
|
- Backs up to specified directories
|
|
- Striped backup across 3 volumes
|
|
|
|
#### Force Mode (Skip Mount Checks)
|
|
```powershell
|
|
.\backupmult.ps1 -SqlInstance "DEVSQL" -Force
|
|
```
|
|
- Skips mount point accessibility validation
|
|
- Useful for scenarios where mount points may not be immediately accessible
|
|
|
|
### Backup Type Logic
|
|
|
|
The script automatically determines the backup type based on:
|
|
|
|
1. **FULL Backup**:
|
|
- Scheduled: Every Thursday (configurable)
|
|
- Overdue: If more than 7 days since last full backup
|
|
- Flag file: `last_full.flag` in the first backup directory
|
|
|
|
2. **DIFFERENTIAL Backup**:
|
|
- Scheduled: Daily (except full backup days)
|
|
- Flag file: `last_diff.flag` in the first backup directory
|
|
|
|
3. **LOG Backup**:
|
|
- Default: When full or diff already completed for the day
|
|
- Continuous: For transaction log backups
|
|
|
|
## Output and Logging
|
|
|
|
### Console Output
|
|
- Clean, essential status messages
|
|
- Job progress and completion notifications
|
|
- Error messages and warnings
|
|
- Suppresses verbose Ola Hallengren messages for better readability
|
|
|
|
### Log Files
|
|
- **Main Log**: `C:\Rubrik\backup-multi-{InstanceName}.log`
|
|
- Consolidated log with all job outputs
|
|
- Timestamped entries with job identification
|
|
- **Job Logs**: `C:\Rubrik\backup-multi-{InstanceName}-job{N}.log`
|
|
- Individual job-specific logs
|
|
- Automatically consolidated into main log after completion
|
|
- Include detailed SQL messages and errors
|
|
|
|
### Log Format
|
|
```
|
|
2025-10-22 14:30:15 [JOB-1] Starting backup job
|
|
2025-10-22 14:30:15 [JOB-1] Executing backup command...
|
|
2025-10-22 14:30:16 [JOB-1] Backup completed successfully
|
|
```
|
|
|
|
## Error Handling
|
|
|
|
### SQL Connection Errors
|
|
- Detected and reported with detailed error information
|
|
- Includes procedure name and line number for SQL errors
|
|
- Jobs marked as "FAILED" with comprehensive error details
|
|
|
|
### Mount Point Issues
|
|
- Validates backup directory accessibility (unless `-Force` is used)
|
|
- Reports inaccessible mount points before starting backups
|
|
- Prevents failed backups due to storage issues
|
|
|
|
### Job Monitoring
|
|
- Real-time monitoring of PowerShell background jobs
|
|
- Automatic cleanup of completed jobs
|
|
- Timeout handling for unresponsive jobs
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
#### "No SQL Server PowerShell module found"
|
|
**Solution**: Install the SqlServer module
|
|
```powershell
|
|
Install-Module -Name SqlServer -AllowClobber -Force
|
|
```
|
|
|
|
#### "Ola Hallengren procedures not found"
|
|
**Solution**: Install Ola Hallengren Maintenance Solution on the SQL Server instance
|
|
```sql
|
|
-- Download and execute the installation script from https://ola.hallengren.com/
|
|
```
|
|
|
|
#### "Access denied" to backup directories
|
|
**Solution**: Ensure the service account has read/write permissions to all backup directories
|
|
|
|
#### Jobs fail with mount point errors
|
|
**Solution**: Use `-Force` parameter to skip mount checks, or ensure mount points are accessible
|
|
|
|
#### Log files not consolidating
|
|
**Solution**: Check permissions on log file directory and ensure no file locks exist
|
|
|
|
### Debug Mode
|
|
Enable debug output by modifying the script to show debug messages:
|
|
- Look for `Write-Output "DEBUG:"` lines in the job scriptblock
|
|
- Check job-specific log files for detailed error information
|
|
|
|
### Performance Tuning
|
|
- **Job Count**: Start with 2-4 jobs and monitor SQL Server performance
|
|
- **Directories**: Use separate physical disks/volumes for striped backups
|
|
- **Memory**: Ensure adequate memory for parallel operations
|
|
- **Network**: Consider network bandwidth for remote backup destinations
|
|
|
|
## Configuration
|
|
|
|
### Backup Schedule
|
|
- **Full Backups**: Thursdays (modify `$fullBackupDay` variable)
|
|
- **Overdue Threshold**: 7 days (modify `$fullBackupOverdueDays` variable)
|
|
- **Cleanup Time**: 168 hours (7 days) for full/diff, 24 hours for log
|
|
|
|
### Directory Structure
|
|
Default structure: `C:\Rubrik\{InstanceName}\Dir1, Dir2, Dir3, Dir4`
|
|
- Modify the default directory logic in the script for custom structures
|
|
- Ensure all directories exist and are writable
|
|
|
|
### Log File Location
|
|
Default: `C:\Rubrik\backup-multi-{InstanceName}.log`
|
|
- Modify `$logFile` variable for custom log locations
|
|
- Ensure log directory exists and is writable
|
|
|
|
## Security Considerations
|
|
|
|
- **SQL Permissions**: Requires sysadmin or appropriate backup permissions
|
|
- **File System Access**: Read/write access to backup directories and log locations
|
|
- **Service Account**: Use dedicated service account with minimal required permissions
|
|
- **Log Security**: Log files may contain sensitive database information
|
|
|
|
## Support and Maintenance
|
|
|
|
### Monitoring
|
|
- Regularly review log files for errors and warnings
|
|
- Monitor backup completion times and success rates
|
|
- Check disk space usage in backup directories
|
|
|
|
### Maintenance Tasks
|
|
- Clean up old backup files according to retention policies
|
|
- Archive and rotate log files periodically
|
|
- Update Ola Hallengren scripts when new versions are available
|
|
|
|
### Version History
|
|
- Track changes to the backup script
|
|
- Test updates in development environments before production deployment
|
|
|
|
## License
|
|
|
|
This script is provided as-is for database backup automation. Ensure compliance with your organization's backup and retention policies.
|
|
|
|
---
|
|
|
|
# RestoreScript.ps1 - SQL Server Database Restore Script
|
|
|
|
A PowerShell script for restoring SQL Server databases from backup files, with support for full, differential, and log backups. It automatically catalogs backups from a specified directory and handles file relocation during restore operations.
|
|
|
|
## Features
|
|
|
|
- **Automatic Backup Cataloging**: Scans a directory for backup files and organizes them by database, type (FULL, DIFF, LOG), and timestamp
|
|
- **Flexible Restore Options**: Supports restoring from full backups only, or full + diffs + logs in the correct sequence
|
|
- **File Relocation**: Moves database and log files to specified paths during restore (e.g., for different drive layouts)
|
|
- **Backup Verification**: Validates backup integrity and provides LSN range information
|
|
- **LSN-Aware Sequencing**: Applies backups in the correct order to avoid LSN conflicts (full → all diffs → logs after latest diff)
|
|
- **Error Handling**: Robust error detection with detailed SQL error messages
|
|
|
|
## Requirements
|
|
|
|
### System Requirements
|
|
- **PowerShell**: Version 5.1 or higher
|
|
- **Operating System**: Windows Server 2016 or later (or Windows 10/11 for development/testing)
|
|
- **SQL Server**: SQL Server 2016 or later (Express, Standard, Enterprise, or Developer editions)
|
|
- **Permissions**: SQL Server sysadmin privileges or appropriate database restore permissions
|
|
|
|
### Software Dependencies
|
|
- **SQL Server PowerShell Module**: Either `SqlServer` or `SQLPS` module must be available
|
|
- Install with: `Install-Module -Name SqlServer -AllowClobber`
|
|
|
|
### Network Requirements
|
|
- **SQL Server Connectivity**: The script must be able to connect to the target SQL Server instance
|
|
- **Backup Source Access**: Read access to the backup files directory
|
|
- **Restore Destination Access**: Read/write access to the data and log file directories
|
|
|
|
## Installation
|
|
|
|
1. **Download the Script**:
|
|
```powershell
|
|
# Place RestoreScript.ps1 in your preferred scripts directory
|
|
# Example: C:\Rubrik\Scripts\
|
|
```
|
|
|
|
2. **Install SQL Server PowerShell Module** (if not already installed):
|
|
```powershell
|
|
Install-Module -Name SqlServer -AllowClobber -Force
|
|
```
|
|
|
|
3. **Verify Permissions**:
|
|
- Ensure the account running the script has SQL Server sysadmin privileges
|
|
- Verify read access to backup directories and write access to restore destinations
|
|
|
|
## Usage
|
|
|
|
### Basic Syntax
|
|
```powershell
|
|
.\RestoreScript.ps1 -SqlInstance "ServerName\InstanceName" -LiveMountRoot "C:\BackupPath" -Action <action> [parameters]
|
|
```
|
|
|
|
### Parameters
|
|
|
|
| Parameter | Type | Required | Default | Description |
|
|
|-----------|------|----------|---------|-------------|
|
|
| `SqlInstance` | String | Yes | sqlfcsql\TESTINST | SQL Server instance name (e.g., "SERVER01\SQL2019") |
|
|
| `LiveMountRoot` | String | Yes | - | Root directory containing backup files |
|
|
| `Action` | String | Yes | - | Action to perform: `catalog`, `restore`, or `verify` |
|
|
| `DatabaseName` | String | No | - | Specific database name (required for restore, optional for others) |
|
|
| `DataPath` | String | No | - | Directory for data files (required for restore) |
|
|
| `LogPath` | String | No | - | Directory for log files (required for restore) |
|
|
|
|
### Actions
|
|
|
|
#### Catalog Action
|
|
Lists all backup files found in the LiveMountRoot directory, organized by database and backup type.
|
|
|
|
**Example**:
|
|
```powershell
|
|
.\RestoreScript.ps1 -SqlInstance "SQLFC1\SQLFC1P" -LiveMountRoot "C:\Rubrik\lm\" -Action catalog
|
|
```
|
|
|
|
**Output**:
|
|
```
|
|
Database Backups Catalog:
|
|
=========================
|
|
Database: DataAW2
|
|
Type: FULL
|
|
Backup: 20251127093305
|
|
C:\Rubrik\lm\DataAW2_FULL_20251127093305_0001.bak
|
|
Type: DIFF
|
|
Backup: 20251128005350
|
|
C:\Rubrik\lm\DataAW2_DIFF_20251128005350_0001.bak
|
|
...
|
|
Type: LOG
|
|
Backup: 20251130130213
|
|
C:\Rubrik\lm\DataAW2_LOG_20251130130213_0001.trn
|
|
...
|
|
|
|
Summary of Databases:
|
|
- DataAW2
|
|
```
|
|
|
|
#### Restore Action
|
|
Restores a specific database using the latest full backup, all subsequent diffs, and logs taken after the latest diff. Files are relocated to the specified DataPath and LogPath.
|
|
|
|
**Example**:
|
|
```powershell
|
|
.\RestoreScript.ps1 -SqlInstance "SQLFC1\SQLFC1P" -LiveMountRoot "C:\Rubrik\lm\" -Action restore -DatabaseName "DataAW2" -DataPath "F:\Data" -LogPath "G:\Logs"
|
|
```
|
|
|
|
**Output**:
|
|
```
|
|
Restoring FULL backup for DataAW2...
|
|
Applying DIFF backup 20251128005350 for DataAW2...
|
|
Applying DIFF backup 20251129010111 for DataAW2...
|
|
Applying DIFF backup 20251130000053 for DataAW2...
|
|
Applying DIFF backup 20251201113159 for DataAW2...
|
|
Applying LOG backup 20251201124800 for DataAW2...
|
|
Finalizing restore for DataAW2...
|
|
Restore completed for DataAW2
|
|
```
|
|
|
|
#### Verify Action
|
|
Verifies backup file integrity and displays LSN ranges and backup details.
|
|
|
|
**Example**:
|
|
```powershell
|
|
.\RestoreScript.ps1 -SqlInstance "SQLFC1\SQLFC1P" -LiveMountRoot "C:\Rubrik\lm\" -Action verify -DatabaseName "DataAW2"
|
|
```
|
|
|
|
**Output**:
|
|
```
|
|
Verifying backups for database: DataAW2
|
|
Verifying FULL backup 20251127093305 for DataAW2...
|
|
Verification successful for FULL 20251127093305
|
|
Backup Details:
|
|
Start Date: 11/27/2025 09:33:05
|
|
Finish Date: 11/27/2025 09:33:07
|
|
First LSN: 56000000289000001
|
|
Last LSN: 56000000289300001
|
|
Database Backup LSN: 56000000182200001
|
|
Differential Base LSN:
|
|
...
|
|
Backup Summary for database: DataAW2
|
|
===================================
|
|
FULL Backups:
|
|
Date: 11/27/2025 09:33:07 | LSN Range: 56000000289000001 - 56000000289300001
|
|
DIFFERENTIAL Backups:
|
|
Date: 11/25/2025 00:21:45 | Base LSN: 56000000182200001 | LSN Range: 56000000270100001 - 56000000270400001
|
|
...
|
|
LOG Backups:
|
|
Point-in-Time Range: 11/30/2025 13:02:13 to 12/01/2025 12:48:00
|
|
LSN Range: 56000000345300001 - 56000000353400001
|
|
No gaps detected in LOG sequence
|
|
```
|
|
|
|
## Restore Logic
|
|
|
|
The script follows a specific restore sequence to ensure data consistency:
|
|
|
|
1. **Full Backup**: Always starts with the latest full backup
|
|
2. **Differential Backups**: Applies all differential backups taken after the full backup
|
|
3. **Log Backups**: Applies only log backups taken after the latest differential backup
|
|
|
|
This approach avoids LSN conflicts that can occur when mixing diffs and logs from different time ranges.
|
|
|
|
## Error Handling
|
|
|
|
### Common Issues
|
|
|
|
#### "Directory lookup for the file failed"
|
|
**Cause**: Original file paths don't exist on the restore server
|
|
**Solution**: Use `-DataPath` and `-LogPath` to relocate files to valid directories
|
|
|
|
#### "The log in this backup set terminates at LSN X, which is too early to apply"
|
|
**Cause**: Log backup is from before the applied differential backup
|
|
**Solution**: The script automatically handles this by only applying logs after the latest diff
|
|
|
|
#### "Database not found in catalog"
|
|
**Cause**: No backups found for the specified database
|
|
**Solution**: Run catalog action first to verify available databases
|
|
|
|
#### "Access denied" to directories
|
|
**Cause**: Insufficient permissions on data/log paths
|
|
**Solution**: Ensure the SQL Server service account has write access to the specified directories
|
|
|
|
## Troubleshooting
|
|
|
|
### Debug Steps
|
|
1. Run the `catalog` action to verify backup files are detected correctly
|
|
2. Run the `verify` action to check backup integrity and LSN ranges
|
|
3. Ensure DataPath and LogPath directories exist and are writable
|
|
4. Check SQL Server error logs for additional details
|
|
|
|
### Performance Considerations
|
|
- Restore operations can be time-intensive for large databases
|
|
- Ensure sufficient disk space on target directories
|
|
- Monitor SQL Server performance during restore operations
|
|
|
|
## Security Considerations
|
|
|
|
- **SQL Permissions**: Requires sysadmin or database restore permissions
|
|
- **File System Access**: Read access to backup sources, write access to restore destinations
|
|
- **Data Protection**: Ensure backup files are from trusted sources
|
|
|
|
## Support and Maintenance
|
|
|
|
### Monitoring
|
|
- Verify restore completion and database accessibility
|
|
- Check SQL Server logs for any restore-related errors
|
|
- Validate file locations after restore
|
|
|
|
### Best Practices
|
|
- Test restores in development environments first
|
|
- Keep backup files organized and accessible
|
|
- Document custom DataPath/LogPath configurations
|
|
- Regularly verify backup integrity with the verify action</content>
|
|
<parameter name="filePath">README.md |