jamesp/zf-sql
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ae9f7943772221e3102ea35d9d2e31cf97ab591b
zf-sql/README.md
T
jamesp ae9f794377 Latest update
2026-06-24 12:28:01 +01:00

17 KiB
Raw Blame History

backupmult.ps1 - Parallel SQL Server Database Backup Script

A PowerShell script that performs parallel database backups using Ola Hallengren's DatabaseBackup stored procedure with the DatabasesInParallel feature for optimal performance and load distribution.

Features

  • Parallel Processing: Utilizes Ola Hallengren's DatabasesInParallel=Y feature to automatically distribute database backup workloads across multiple concurrent jobs
  • Smart Backup Type Detection: Automatically determines whether to run FULL, DIFFERENTIAL, or LOG backups based on schedule and overdue status
  • Multi-Directory Support: Supports striped backups across multiple directories for improved I/O performance
  • Comprehensive Logging: Thread-safe logging with job-specific log files that are consolidated into a main log
  • Error Handling: Robust error detection and reporting with detailed SQL error information
  • Job Monitoring: Real-time monitoring of backup job progress and completion status
  • Mount Point Validation: Optional mount point checking to ensure backup destinations are accessible

Requirements

System Requirements

  • PowerShell: Version 5.1 or higher
  • Operating System: Windows Server 2016 or later (or Windows 10/11 for development/testing)
  • SQL Server: SQL Server 2016 or later (Express, Standard, Enterprise, or Developer editions)
  • Permissions: SQL Server sysadmin privileges or appropriate database backup permissions

Software Dependencies

  • Ola Hallengren Maintenance Solution: Must be installed on the target SQL Server instance
  • SQL Server PowerShell Module: Either SqlServer or SQLPS module must be available
    • Install with: Install-Module -Name SqlServer -AllowClobber

Network Requirements

  • SQL Server Connectivity: The script must be able to connect to the target SQL Server instance
  • Backup Destination Access: Read/write access to all specified backup directories
  • Mount Points: If using mount points, they must be accessible and have sufficient space

Installation

  1. Download the Script:

    # Place backupmult.ps1 in your preferred scripts directory
# Example: C:\Rubrik\Scripts\

  2. Install Ola Hallengren Maintenance Solution:

    -- Run on your SQL Server instance
-- Download from https://ola.hallengren.com/
-- Execute the installation script

  3. Install SQL Server PowerShell Module (if not already installed):

    Install-Module -Name SqlServer -AllowClobber -Force

  4. Verify Permissions:

    • Ensure the account running the script has SQL Server sysadmin privileges
    • Verify write access to backup directories and log file locations

Usage

Basic Syntax

.\backupmult.ps1 -SqlInstance "ServerName\InstanceName" [parameters]

Parameters

Parameter Type Required Default Description
SqlInstance String Yes - SQL Server instance name (e.g., "SERVER01\SQL2019")
Directories String No Auto-generated Comma-separated list of backup directories
Jobs Integer No 2 Number of parallel backup jobs (1-8)
Force Switch No False Skip mount point accessibility checks

Examples

Basic Usage with Default Settings

.\backupmult.ps1 -SqlInstance "PRODSQL01\SQL2019"
  • Uses 2 parallel jobs
  • Auto-generates 4 backup directories under C:\Rubrik\SQL2019\
  • Performs appropriate backup type based on schedule

Custom Directories and Job Count

.\backupmult.ps1 -SqlInstance "TESTSQL\SQL2022" -Directories "D:\Backups\Dir1, E:\Backups\Dir2, F:\Backups\Dir3" -Jobs 3
  • Uses 3 parallel jobs
  • Backs up to specified directories
  • Striped backup across 3 volumes

Force Mode (Skip Mount Checks)

.\backupmult.ps1 -SqlInstance "DEVSQL" -Force
  • Skips mount point accessibility validation
  • Useful for scenarios where mount points may not be immediately accessible

Backup Type Logic

The script automatically determines the backup type based on:

  1. FULL Backup:

    • Scheduled: Every Thursday (configurable)
    • Overdue: If more than 7 days since last full backup
    • Flag file: last_full.flag in the first backup directory

  2. DIFFERENTIAL Backup:

    • Scheduled: Daily (except full backup days)
    • Flag file: last_diff.flag in the first backup directory

  3. LOG Backup:

    • Default: When full or diff already completed for the day
    • Continuous: For transaction log backups

Output and Logging

Console Output

  • Clean, essential status messages
  • Job progress and completion notifications
  • Error messages and warnings
  • Suppresses verbose Ola Hallengren messages for better readability

Log Files

  • Main Log: C:\Rubrik\backup-multi-{InstanceName}.log
    • Consolidated log with all job outputs
    • Timestamped entries with job identification
  • Job Logs: C:\Rubrik\backup-multi-{InstanceName}-job{N}.log
    • Individual job-specific logs
    • Automatically consolidated into main log after completion
    • Include detailed SQL messages and errors

Log Format

2025-10-22 14:30:15 [JOB-1] Starting backup job
2025-10-22 14:30:15 [JOB-1] Executing backup command...
2025-10-22 14:30:16 [JOB-1] Backup completed successfully

Error Handling

SQL Connection Errors

  • Detected and reported with detailed error information
  • Includes procedure name and line number for SQL errors
  • Jobs marked as "FAILED" with comprehensive error details

Mount Point Issues

  • Validates backup directory accessibility (unless -Force is used)
  • Reports inaccessible mount points before starting backups
  • Prevents failed backups due to storage issues

Job Monitoring

  • Real-time monitoring of PowerShell background jobs
  • Automatic cleanup of completed jobs
  • Timeout handling for unresponsive jobs

Troubleshooting

Common Issues

"No SQL Server PowerShell module found"

Solution: Install the SqlServer module

Install-Module -Name SqlServer -AllowClobber -Force

"Ola Hallengren procedures not found"

Solution: Install Ola Hallengren Maintenance Solution on the SQL Server instance

-- Download and execute the installation script from https://ola.hallengren.com/

"Access denied" to backup directories

Solution: Ensure the service account has read/write permissions to all backup directories

Jobs fail with mount point errors

Solution: Use -Force parameter to skip mount checks, or ensure mount points are accessible

Log files not consolidating

Solution: Check permissions on log file directory and ensure no file locks exist

Debug Mode

Enable debug output by modifying the script to show debug messages:

  • Look for Write-Output "DEBUG:" lines in the job scriptblock
  • Check job-specific log files for detailed error information

Performance Tuning

  • Job Count: Start with 2-4 jobs and monitor SQL Server performance
  • Directories: Use separate physical disks/volumes for striped backups
  • Memory: Ensure adequate memory for parallel operations
  • Network: Consider network bandwidth for remote backup destinations

Configuration

Backup Schedule

  • Full Backups: Thursdays (modify $fullBackupDay variable)
  • Overdue Threshold: 7 days (modify $fullBackupOverdueDays variable)
  • Cleanup Time: 168 hours (7 days) for full/diff, 24 hours for log

Directory Structure

Default structure: C:\Rubrik\{InstanceName}\Dir1, Dir2, Dir3, Dir4

  • Modify the default directory logic in the script for custom structures
  • Ensure all directories exist and are writable

Log File Location

Default: C:\Rubrik\backup-multi-{InstanceName}.log

  • Modify $logFile variable for custom log locations
  • Ensure log directory exists and is writable

Security Considerations

  • SQL Permissions: Requires sysadmin or appropriate backup permissions
  • File System Access: Read/write access to backup directories and log locations
  • Service Account: Use dedicated service account with minimal required permissions
  • Log Security: Log files may contain sensitive database information

Support and Maintenance

Monitoring

  • Regularly review log files for errors and warnings
  • Monitor backup completion times and success rates
  • Check disk space usage in backup directories

Maintenance Tasks

  • Clean up old backup files according to retention policies
  • Archive and rotate log files periodically
  • Update Ola Hallengren scripts when new versions are available

Version History

  • Track changes to the backup script
  • Test updates in development environments before production deployment

License

This script is provided as-is for database backup automation. Ensure compliance with your organization's backup and retention policies.

RestoreScript.ps1 - SQL Server Database Restore Script

A PowerShell script for restoring SQL Server databases from backup files, with support for full, differential, and log backups. It automatically catalogs backups from a specified directory and handles file relocation during restore operations.

Features

  • Automatic Backup Cataloging: Scans a directory for backup files and organizes them by database, type (FULL, DIFF, LOG), and timestamp
  • Flexible Restore Options: Supports restoring from full backups only, or full + diffs + logs in the correct sequence
  • File Relocation: Moves database and log files to specified paths during restore (e.g., for different drive layouts)
  • Backup Verification: Validates backup integrity and provides LSN range information
  • LSN-Aware Sequencing: Applies backups in the correct order to avoid LSN conflicts (full → all diffs → logs after latest diff)
  • Error Handling: Robust error detection with detailed SQL error messages

Requirements

System Requirements

  • PowerShell: Version 5.1 or higher
  • Operating System: Windows Server 2016 or later (or Windows 10/11 for development/testing)
  • SQL Server: SQL Server 2016 or later (Express, Standard, Enterprise, or Developer editions)
  • Permissions: SQL Server sysadmin privileges or appropriate database restore permissions

Software Dependencies

  • SQL Server PowerShell Module: Either SqlServer or SQLPS module must be available
    • Install with: Install-Module -Name SqlServer -AllowClobber

Network Requirements

  • SQL Server Connectivity: The script must be able to connect to the target SQL Server instance
  • Backup Source Access: Read access to the backup files directory
  • Restore Destination Access: Read/write access to the data and log file directories

Installation

  1. Download the Script:

    # Place RestoreScript.ps1 in your preferred scripts directory
# Example: C:\Rubrik\Scripts\

  2. Install SQL Server PowerShell Module (if not already installed):

    Install-Module -Name SqlServer -AllowClobber -Force

  3. Verify Permissions:

    • Ensure the account running the script has SQL Server sysadmin privileges
    • Verify read access to backup directories and write access to restore destinations

Usage

Basic Syntax

.\RestoreScript.ps1 -SqlInstance "ServerName\InstanceName" -LiveMountRoot "C:\BackupPath" -Action <action> [parameters]

Parameters

Parameter Type Required Default Description
SqlInstance String Yes sqlfcsql\TESTINST SQL Server instance name (e.g., "SERVER01\SQL2019")
LiveMountRoot String Yes - Root directory containing backup files
Action String Yes - Action to perform: catalog, restore, or verify
DatabaseName String No - Specific database name (required for restore, optional for others)
DataPath String No - Directory for data files (required for restore)
LogPath String No - Directory for log files (required for restore)

Actions

Catalog Action

Lists all backup files found in the LiveMountRoot directory, organized by database and backup type.

Example:

.\RestoreScript.ps1 -SqlInstance "SQLFC1\SQLFC1P" -LiveMountRoot "C:\Rubrik\lm\" -Action catalog

Output:

Database Backups Catalog:
=========================
Database: DataAW2
  Type: FULL
    Backup: 20251127093305
      C:\Rubrik\lm\DataAW2_FULL_20251127093305_0001.bak
  Type: DIFF
    Backup: 20251128005350
      C:\Rubrik\lm\DataAW2_DIFF_20251128005350_0001.bak
    ...
  Type: LOG
    Backup: 20251130130213
      C:\Rubrik\lm\DataAW2_LOG_20251130130213_0001.trn
    ...

Summary of Databases:
  - DataAW2

Restore Action

Restores a specific database using the latest full backup, all subsequent diffs, and logs taken after the latest diff. Files are relocated to the specified DataPath and LogPath.

Example:

.\RestoreScript.ps1 -SqlInstance "SQLFC1\SQLFC1P" -LiveMountRoot "C:\Rubrik\lm\" -Action restore -DatabaseName "DataAW2" -DataPath "F:\Data" -LogPath "G:\Logs"

Output:

Restoring FULL backup for DataAW2...
Applying DIFF backup 20251128005350 for DataAW2...
Applying DIFF backup 20251129010111 for DataAW2...
Applying DIFF backup 20251130000053 for DataAW2...
Applying DIFF backup 20251201113159 for DataAW2...
Applying LOG backup 20251201124800 for DataAW2...
Finalizing restore for DataAW2...
Restore completed for DataAW2

Verify Action

Verifies backup file integrity and displays LSN ranges and backup details.

Example:

.\RestoreScript.ps1 -SqlInstance "SQLFC1\SQLFC1P" -LiveMountRoot "C:\Rubrik\lm\" -Action verify -DatabaseName "DataAW2"

Output:

Verifying backups for database: DataAW2
Verifying FULL backup 20251127093305 for DataAW2...
Verification successful for FULL 20251127093305
  Backup Details:
    Start Date: 11/27/2025 09:33:05
    Finish Date: 11/27/2025 09:33:07
    First LSN: 56000000289000001
    Last LSN: 56000000289300001
    Database Backup LSN: 56000000182200001
    Differential Base LSN:
...
Backup Summary for database: DataAW2
===================================
FULL Backups:
  Date: 11/27/2025 09:33:07 | LSN Range: 56000000289000001 - 56000000289300001
DIFFERENTIAL Backups:
  Date: 11/25/2025 00:21:45 | Base LSN: 56000000182200001 | LSN Range: 56000000270100001 - 56000000270400001
  ...
LOG Backups:
  Point-in-Time Range: 11/30/2025 13:02:13 to 12/01/2025 12:48:00
  LSN Range: 56000000345300001 - 56000000353400001
  No gaps detected in LOG sequence

Restore Logic

The script follows a specific restore sequence to ensure data consistency:

  1. Full Backup: Always starts with the latest full backup
  2. Differential Backups: Applies all differential backups taken after the full backup
  3. Log Backups: Applies only log backups taken after the latest differential backup

This approach avoids LSN conflicts that can occur when mixing diffs and logs from different time ranges.

Error Handling

Common Issues

"Directory lookup for the file failed"

Cause: Original file paths don't exist on the restore server Solution: Use -DataPath and -LogPath to relocate files to valid directories

"The log in this backup set terminates at LSN X, which is too early to apply"

Cause: Log backup is from before the applied differential backup Solution: The script automatically handles this by only applying logs after the latest diff

"Database not found in catalog"

Cause: No backups found for the specified database Solution: Run catalog action first to verify available databases

"Access denied" to directories

Cause: Insufficient permissions on data/log paths Solution: Ensure the SQL Server service account has write access to the specified directories

Troubleshooting

Debug Steps

  1. Run the catalog action to verify backup files are detected correctly
  2. Run the verify action to check backup integrity and LSN ranges
  3. Ensure DataPath and LogPath directories exist and are writable
  4. Check SQL Server error logs for additional details

Performance Considerations

  • Restore operations can be time-intensive for large databases
  • Ensure sufficient disk space on target directories
  • Monitor SQL Server performance during restore operations

Security Considerations

  • SQL Permissions: Requires sysadmin or database restore permissions
  • File System Access: Read access to backup sources, write access to restore destinations
  • Data Protection: Ensure backup files are from trusted sources

Support and Maintenance

Monitoring

  • Verify restore completion and database accessibility
  • Check SQL Server logs for any restore-related errors
  • Validate file locations after restore

Best Practices

  • Test restores in development environments first
  • Keep backup files organized and accessible
  • Document custom DataPath/LogPath configurations
  • Regularly verify backup integrity with the verify action README.md
Reference in New Issue View Git Blame Copy Permalink