From 00f3f3c92ea64c2ce6a4c641c134f23029fe696c Mon Sep 17 00:00:00 2001 From: James Pattinson Date: Tue, 25 Nov 2025 11:31:51 +0000 Subject: [PATCH] Ready for final release --- .gitignore | 2 + README.md | 237 +++++++ Update Rubrik.xml | Bin 3722 -> 0 bytes backup.ps1 | 871 ++++++++++++++++++------ backupSingle.ps1 | 271 ++++++++ failover.cmd | 2 - release_v0/backup.txt | 90 +++ release_v0/start.txt | 3 + backupmult.ps1 => release_v1/backup.ps1 | 27 +- setActiveNode.ps1 | 16 +- 10 files changed, 1304 insertions(+), 215 deletions(-) create mode 100644 README.md delete mode 100644 Update Rubrik.xml create mode 100644 backupSingle.ps1 delete mode 100644 failover.cmd create mode 100644 release_v0/backup.txt create mode 100644 release_v0/start.txt rename backupmult.ps1 => release_v1/backup.ps1 (96%) diff --git a/.gitignore b/.gitignore index 182d51e..1264592 100644 --- a/.gitignore +++ b/.gitignore @@ -4,6 +4,8 @@ # Ignore backup files and flags *.bak *.flag +*.json +*.xml # Ignore PowerShell/Batch temporary files *.ps1~ diff --git a/README.md b/README.md new file mode 100644 index 0000000..dfe4a37 --- /dev/null +++ b/README.md @@ -0,0 +1,237 @@ +# backupmult.ps1 - Parallel SQL Server Database Backup Script + +A PowerShell script that performs parallel database backups using Ola Hallengren's DatabaseBackup stored procedure with the `DatabasesInParallel` feature for optimal performance and load distribution. + +## Features + +- **Parallel Processing**: Utilizes Ola Hallengren's `DatabasesInParallel=Y` feature to automatically distribute database backup workloads across multiple concurrent jobs +- **Smart Backup Type Detection**: Automatically determines whether to run FULL, DIFFERENTIAL, or LOG backups based on schedule and overdue status +- **Multi-Directory Support**: Supports striped backups across multiple directories for improved I/O performance +- **Comprehensive Logging**: Thread-safe logging with job-specific log files that are consolidated into a main log +- **Error Handling**: Robust error detection and reporting with detailed SQL error information +- **Job Monitoring**: Real-time monitoring of backup job progress and completion status +- **Mount Point Validation**: Optional mount point checking to ensure backup destinations are accessible + +## Requirements + +### System Requirements +- **PowerShell**: Version 5.1 or higher +- **Operating System**: Windows Server 2016 or later (or Windows 10/11 for development/testing) +- **SQL Server**: SQL Server 2016 or later (Express, Standard, Enterprise, or Developer editions) +- **Permissions**: SQL Server sysadmin privileges or appropriate database backup permissions + +### Software Dependencies +- **Ola Hallengren Maintenance Solution**: Must be installed on the target SQL Server instance + - Download from: https://ola.hallengren.com/ + - Install the DatabaseBackup stored procedure +- **SQL Server PowerShell Module**: Either `SqlServer` or `SQLPS` module must be available + - Install with: `Install-Module -Name SqlServer -AllowClobber` + +### Network Requirements +- **SQL Server Connectivity**: The script must be able to connect to the target SQL Server instance +- **Backup Destination Access**: Read/write access to all specified backup directories +- **Mount Points**: If using mount points, they must be accessible and have sufficient space + +## Installation + +1. **Download the Script**: + ```powershell + # Place backupmult.ps1 in your preferred scripts directory + # Example: C:\Rubrik\Scripts\ + ``` + +2. **Install Ola Hallengren Maintenance Solution**: + ```sql + -- Run on your SQL Server instance + -- Download from https://ola.hallengren.com/ + -- Execute the installation script + ``` + +3. **Install SQL Server PowerShell Module** (if not already installed): + ```powershell + Install-Module -Name SqlServer -AllowClobber -Force + ``` + +4. **Verify Permissions**: + - Ensure the account running the script has SQL Server sysadmin privileges + - Verify write access to backup directories and log file locations + +## Usage + +### Basic Syntax +```powershell +.\backupmult.ps1 -SqlInstance "ServerName\InstanceName" [parameters] +``` + +### Parameters + +| Parameter | Type | Required | Default | Description | +|-----------|------|----------|---------|-------------| +| `SqlInstance` | String | Yes | - | SQL Server instance name (e.g., "SERVER01\SQL2019") | +| `Directories` | String | No | Auto-generated | Comma-separated list of backup directories | +| `Jobs` | Integer | No | 2 | Number of parallel backup jobs (1-8) | +| `Force` | Switch | No | False | Skip mount point accessibility checks | + +### Examples + +#### Basic Usage with Default Settings +```powershell +.\backupmult.ps1 -SqlInstance "PRODSQL01\SQL2019" +``` +- Uses 2 parallel jobs +- Auto-generates 4 backup directories under `C:\Rubrik\SQL2019\` +- Performs appropriate backup type based on schedule + +#### Custom Directories and Job Count +```powershell +.\backupmult.ps1 -SqlInstance "TESTSQL\SQL2022" -Directories "D:\Backups\Dir1, E:\Backups\Dir2, F:\Backups\Dir3" -Jobs 3 +``` +- Uses 3 parallel jobs +- Backs up to specified directories +- Striped backup across 3 volumes + +#### Force Mode (Skip Mount Checks) +```powershell +.\backupmult.ps1 -SqlInstance "DEVSQL" -Force +``` +- Skips mount point accessibility validation +- Useful for scenarios where mount points may not be immediately accessible + +### Backup Type Logic + +The script automatically determines the backup type based on: + +1. **FULL Backup**: + - Scheduled: Every Thursday (configurable) + - Overdue: If more than 7 days since last full backup + - Flag file: `last_full.flag` in the first backup directory + +2. **DIFFERENTIAL Backup**: + - Scheduled: Daily (except full backup days) + - Flag file: `last_diff.flag` in the first backup directory + +3. **LOG Backup**: + - Default: When full or diff already completed for the day + - Continuous: For transaction log backups + +## Output and Logging + +### Console Output +- Clean, essential status messages +- Job progress and completion notifications +- Error messages and warnings +- Suppresses verbose Ola Hallengren messages for better readability + +### Log Files +- **Main Log**: `C:\Rubrik\backup-multi-{InstanceName}.log` + - Consolidated log with all job outputs + - Timestamped entries with job identification +- **Job Logs**: `C:\Rubrik\backup-multi-{InstanceName}-job{N}.log` + - Individual job-specific logs + - Automatically consolidated into main log after completion + - Include detailed SQL messages and errors + +### Log Format +``` +2025-10-22 14:30:15 [JOB-1] Starting backup job +2025-10-22 14:30:15 [JOB-1] Executing backup command... +2025-10-22 14:30:16 [JOB-1] Backup completed successfully +``` + +## Error Handling + +### SQL Connection Errors +- Detected and reported with detailed error information +- Includes procedure name and line number for SQL errors +- Jobs marked as "FAILED" with comprehensive error details + +### Mount Point Issues +- Validates backup directory accessibility (unless `-Force` is used) +- Reports inaccessible mount points before starting backups +- Prevents failed backups due to storage issues + +### Job Monitoring +- Real-time monitoring of PowerShell background jobs +- Automatic cleanup of completed jobs +- Timeout handling for unresponsive jobs + +## Troubleshooting + +### Common Issues + +#### "No SQL Server PowerShell module found" +**Solution**: Install the SqlServer module +```powershell +Install-Module -Name SqlServer -AllowClobber -Force +``` + +#### "Ola Hallengren procedures not found" +**Solution**: Install Ola Hallengren Maintenance Solution on the SQL Server instance +```sql +-- Download and execute the installation script from https://ola.hallengren.com/ +``` + +#### "Access denied" to backup directories +**Solution**: Ensure the service account has read/write permissions to all backup directories + +#### Jobs fail with mount point errors +**Solution**: Use `-Force` parameter to skip mount checks, or ensure mount points are accessible + +#### Log files not consolidating +**Solution**: Check permissions on log file directory and ensure no file locks exist + +### Debug Mode +Enable debug output by modifying the script to show debug messages: +- Look for `Write-Output "DEBUG:"` lines in the job scriptblock +- Check job-specific log files for detailed error information + +### Performance Tuning +- **Job Count**: Start with 2-4 jobs and monitor SQL Server performance +- **Directories**: Use separate physical disks/volumes for striped backups +- **Memory**: Ensure adequate memory for parallel operations +- **Network**: Consider network bandwidth for remote backup destinations + +## Configuration + +### Backup Schedule +- **Full Backups**: Thursdays (modify `$fullBackupDay` variable) +- **Overdue Threshold**: 7 days (modify `$fullBackupOverdueDays` variable) +- **Cleanup Time**: 168 hours (7 days) for full/diff, 24 hours for log + +### Directory Structure +Default structure: `C:\Rubrik\{InstanceName}\Dir1, Dir2, Dir3, Dir4` +- Modify the default directory logic in the script for custom structures +- Ensure all directories exist and are writable + +### Log File Location +Default: `C:\Rubrik\backup-multi-{InstanceName}.log` +- Modify `$logFile` variable for custom log locations +- Ensure log directory exists and is writable + +## Security Considerations + +- **SQL Permissions**: Requires sysadmin or appropriate backup permissions +- **File System Access**: Read/write access to backup directories and log locations +- **Service Account**: Use dedicated service account with minimal required permissions +- **Log Security**: Log files may contain sensitive database information + +## Support and Maintenance + +### Monitoring +- Regularly review log files for errors and warnings +- Monitor backup completion times and success rates +- Check disk space usage in backup directories + +### Maintenance Tasks +- Clean up old backup files according to retention policies +- Archive and rotate log files periodically +- Update Ola Hallengren scripts when new versions are available + +### Version History +- Track changes to the backup script +- Test updates in development environments before production deployment + +## License + +This script is provided as-is for database backup automation. Ensure compliance with your organization's backup and retention policies. +README.md \ No newline at end of file diff --git a/Update Rubrik.xml b/Update Rubrik.xml deleted file mode 100644 index b4e164f12a1dca5ed2961ba24e6fc1646fb4fc2e..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 3722 zcmc(iQBNC36ov0|rTzztheWDmZ0w{^YMfG_#ZuCw1;hj7!DStw*bDM5Q2u<|@7(cr zch=Z)qDs{YJUese-gD2KxikCsUp;$cf7-;x*02W~+0-(-wp&YW$JT7a)~(HNgOSuO z=ua3;?b1h2?AU&?CRX2K^Tzu2wS8qhdWU$qS?ICj9z)a=OGUb}oafHAt;LfO>x!5Y z_q5LMH9b?}XZ(G_a^2BPuzhqy3H^_ZwyXo{jETRFZCV#R8?<`h zC_RX+((|s}6a9*qQ@rnDcTTJX{7w6V)&!&(49Q1q8~ZsSrvV<$?F8#&hPOfQz5R~w zYgR@+3cF&q`c`K{t1MraJV7;erVd?~v4`h>g7H9T@{X#C%=#IU#|B9!V_K~cI&Tj)Hh*2qO!0y=b9rc=>^^vDo z#}-IS#_FrwswiRESY0L3;bN6l6&Vy;;9R79S;b=+sxSQXoL?#B0*3F%e+>8Kc-S*q zgV?6w>!E5|tq;|DSMHe;JW7AkdY{}>#hYHuI^OEq7RrBdL%u2c1RjOU`J3pi9RTlGXm+SHTcy+~ku1cM7VgFah!fL()vuc&r zTgV1vAiUE41@9RMZW&Sic_O0$*i&a_#LgSRl2ClEmpIFPyl1X9Rj)@dEB=)K1J1(t z#QTwVWf3ZwP{pMq=dtPefs1SKj<-8^(QDd#xX z)n>#|N=8rpev`6OAMh?U7Uy4|yu*$U+FWf;94F{moQ+Y7n3P8H*vc!3q_SH zC=C0c(!F**yVYjIcabc+sm_+x%`BhhKDeVmi$2V!h;LLGX&Ty0JqC!7R2ljKpgc+-8& z_ZRb|n#F37Qgpium^1D>d$6Ee&WMb1zjrP>&wJd9LgwX(RGzHzHL2)rkFS&S=v&|% tY~{^4V^sC0dV|%s)&!*}kGj7i$FSpcTd!l$tT@AfQ~k$8(wAc2{sA+OT7Cck diff --git a/backup.ps1 b/backup.ps1 index 615d110..6436f24 100644 --- a/backup.ps1 +++ b/backup.ps1 @@ -1,271 +1,742 @@ param( [Parameter(Mandatory=$true)] [string]$SqlInstance, + + [Parameter(Mandatory=$true)] + [string]$MvName, [Parameter(Mandatory=$false)] - [string]$Directory, + [int]$Jobs = 1, [Parameter(Mandatory=$false)] - [switch]$Force + [int]$LogRetentionDays = 30 + , + [Parameter(Mandatory=$false)] + [switch]$Nuke ) -# backup.ps1 +# backup.ps1 - Parallel database backup script using Ola H # -# TODO: Parallelize backups for multiple DBs in the instance +# Uses Ola H's built-in parallel processing by starting multiple concurrent backup jobs +# Each job will automatically share the database load using DatabasesInParallel=Y if Jobs>1 + +# TODO: See if there is way to query QueueDatabase during backup to monitor progress +# TODO: Better trapping when RSC connection fails + +$fullBackupDay = 'Thursday' +$fullBackupOverdueDays = 7 +$SAFile = "C:\Rubrik\scripts\rbksql.xml" +$logDir = "C:\Rubrik\logs" + +function Write-Log($message, $jobId = "") { + $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss" + $jobPrefix = if ($jobId) { "[JOB-$jobId] " } else { "" } + $logEntry = "$timestamp $jobPrefix$message" + + # Use mutex for thread-safe logging to main log file + $mutex = $null + try { + $mutex = [System.Threading.Mutex]::new($false, "BackupLogMutex") + if ($mutex.WaitOne(5000)) { # 5 second timeout + Add-Content -Path $logFile -Value $logEntry -Encoding UTF8 + } else { + Write-Warning "Could not acquire log mutex, writing to console only" + } + } catch { + Write-Warning "Logging error: $($_.Exception.Message)" + } finally { + if ($mutex) { + $mutex.ReleaseMutex() + $mutex.Dispose() + } + } + + Write-Host $logEntry +} + +# Parse instance name from SQL instance parameter +$instanceParts = $SqlInstance -split '\\' +if ($instanceParts.Length -eq 2) { + # Format: HOSTNAME\INSTANCENAME or CLUSTERNAME\INSTANCENAME + $instanceName = $instanceParts[1] + # SqlInstance stays as provided (HOSTNAME\INSTANCENAME) +} elseif ($instanceParts.Length -eq 1) { + # Single value provided - could be hostname (default instance) or instance name on local host + $singleValue = $instanceParts[0] + + # If it's "MSSQLSERVER" (default instance name), treat as default instance on local host + if ($singleValue -eq "MSSQLSERVER") { + $instanceName = "MSSQLSERVER" + $SqlInstance = $env:COMPUTERNAME # Connect to default instance (no instance name) + } else { + # Assume it's an instance name on the local host + $instanceName = $singleValue + $SqlInstance = "$($env:COMPUTERNAME)\$singleValue" + } +} else { + $instanceName = $SqlInstance.Replace('\\', '_').Replace('/', '_') +} + +if ([string]::IsNullOrEmpty($instanceName)) { + Write-Host "ERROR: Could not determine instance name from SqlInstance: '$SqlInstance'" + exit 1 +} + +# Sanitize and trim the instance name for safe filenames +$instanceName = $instanceName.Trim() +$invalidChars = [IO.Path]::GetInvalidFileNameChars() +foreach ($c in $invalidChars) { + $escaped = [regex]::Escape($c) + $instanceName = $instanceName -replace $escaped, '_' +} + +$timestamp = Get-Date -Format "yyyyMMdd_HHmmss" +# Ensure log directory exists before building/using log file +if (-not (Test-Path $logDir)) { + try { + New-Item -ItemType Directory -Path $logDir -Force | Out-Null + } catch { + Write-Host "ERROR: Could not create log directory $logDir : $($_.Exception.Message)" + exit 1 + } +} + +$logFileName = "backup_{0}_{1}.log" -f $instanceName, $timestamp +$logFile = Join-Path $logDir $logFileName + +Write-Log "DEBUG: SqlInstance='$SqlInstance', instanceName='$instanceName', logFile='$logFile'" + +# Function to clean up old log files +function Remove-OldLogs { + param([int]$retentionDays) + + if (-not (Test-Path $logDir)) { + try { + New-Item -ItemType Directory -Path $logDir -Force | Out-Null + Write-Log "INFO: Created log directory: $logDir" + } catch { + Write-Log "ERROR: Failed to create log directory $logDir. $($_.Exception.Message)" + return + } + } + + $cutoffDate = (Get-Date).AddDays(-$retentionDays) + Write-Log "INFO: Cleaning up log files older than $retentionDays days (before $($cutoffDate.ToString('yyyy-MM-dd')))" + + $oldLogs = Get-ChildItem -Path $logDir -Filter "*.log" | Where-Object { $_.LastWriteTime -lt $cutoffDate } + $deletedCount = 0 + + foreach ($logFile in $oldLogs) { + try { + Remove-Item $logFile.FullName -Force + $deletedCount++ + } catch { + Write-Log "WARNING: Failed to delete old log file $($logFile.Name): $($_.Exception.Message)" + } + } + + Write-Log "INFO: Cleaned up $deletedCount old log files" +} + +# Clean up old logs before starting +Remove-OldLogs -retentionDays $LogRetentionDays # Import SQL Server PowerShell module try { - # Try to import the newer SqlServer module first if (Get-Module -ListAvailable -Name SqlServer) { Import-Module SqlServer -ErrorAction Stop - Write-Host "INFO: SqlServer PowerShell module loaded successfully." + Write-Log "INFO: SqlServer PowerShell module loaded successfully." } - # Fall back to older SQLPS module if available elseif (Get-Module -ListAvailable -Name SQLPS) { Import-Module SQLPS -ErrorAction Stop - Write-Host "INFO: SQLPS PowerShell module loaded successfully." + Write-Log "INFO: SQLPS PowerShell module loaded successfully." } else { throw "No SQL Server PowerShell module found" } - # Verify Invoke-Sqlcmd is available if (-not (Get-Command Invoke-Sqlcmd -ErrorAction SilentlyContinue)) { throw "Invoke-Sqlcmd command not available" } } catch { - Write-Host "ERROR: Failed to import SQL Server PowerShell module. Please install it using: Install-Module -Name SqlServer -AllowClobber" - Write-Host "ERROR: $($_.Exception.Message)" + Write-Log "ERROR: Failed to import SQL Server PowerShell module. Please install it using: Install-Module -Name SqlServer -AllowClobber" + Write-Log "ERROR: $($_.Exception.Message)" exit 1 } -$instanceName = $SqlInstance.Split('\')[1] - -# Use provided directory parameter or default to instance-based path -if ($Directory) { - $directory = $Directory - Write-Host "INFO: Using provided directory: $directory" -} else { - $directory = "C:\Rubrik\$instanceName" - Write-Host "INFO: Using default directory: $directory" +# Import Rubrik Security Cloud module +try { + Import-Module RubrikSecurityCloud -ErrorAction Stop + Write-Log "INFO: RubrikSecurityCloud module loaded successfully." +} catch { + Write-Log "ERROR: Failed to import RubrikSecurityCloud module. $($_.Exception.Message)" + exit 1 } -$fullBackupDay = 'Thursday' -$fullBackupOverdueDays = 7 # Force full backup if last full backup is older than this many days -$checkCluster = $false -#$logFile = "C:\Rubrik\backup-$instanceName.log" -$logFile = "H:\Backup\backup-$instanceName.log" +$localNode = $env:COMPUTERNAME +$clusterInstance = Get-ClusterResource | Where-Object { $_.ResourceType -eq "SQL Server" -and $_.Name -eq "SQL Server ($instanceName)" } + +if ($clusterInstance) { + $ownerNode = $clusterInstance.OwnerNode + if ($ownerNode -ne $localNode) { + Write-Log "SQL instance '$SqlInstance' is not running on local node '$localNode'. Updating the MV." + + Connect-Rsc -ServiceAccountFile $SAFile + Write-Log "Connected to Rubrik Security Cloud." + + $newHost = Get-RscHost -Name $ownerNode -OsType WINDOWS + + $query = New-RscQuery -GqlQuery slaManagedVolumes -AddField Nodes.HostDetail, Nodes.SmbShare, Nodes.ClientConfig, Nodes.ClientConfig.BackupScript, Nodes.ClientConfig.PreBackupScript + $query.var.filter = @(Get-RscType -Name Filter) + $query.var.filter[0].field = "NAME_EXACT_MATCH" + $query.var.filter[0].Texts = $mvName + $mvResult = $query.Invoke() + if (-not $mvResult.nodes -or $mvResult.nodes.Count -eq 0) { + Write-Log "ERROR: Managed Volume '$mvName' not found. This may be due to insufficient permissions or the volume not existing." + Disconnect-Rsc + exit 1 + } + $mvDetail = $mvResult.nodes[0] + + Write-Log "Found Managed Volume: $($mvDetail.Name) (ID: $($mvDetail.Id), Status: $($mvDetail.hostDetail.Status), HostDetail Name: $($mvDetail.hostDetail.Name))" + + $query = New-RscMutation -GqlMutation updateManagedVolume + $query.Var.input = Get-RscType -Name UpdateManagedVolumeInput + $query.Var.input.update = Get-RscType -Name ManagedVolumeUpdateInput + $query.Var.input.update.config = Get-RscType -Name ManagedVolumePatchConfigInput + $query.Var.input.update.slaClientConfig = Get-RscType -Name ManagedVolumePatchSlaClientConfigInput + + $query.Var.input.Id = $mvDetail.Id + $query.Var.input.update.Name = $mvName + $query.Var.input.update.config.SmbDomainName = $mvDetail.SmbShare.DomainName + $query.Var.input.update.config.SmbValidIps = $newHost.Name + $query.Var.input.update.config.SmbValidUsers = $mvDetail.SmbShare.ValidUsers + $mvDetail.SmbShare.ActiveDirectoryGroups + $query.Var.input.update.slaClientConfig.clientHostId = $newHost.Id + $query.Var.input.update.slaClientConfig.channelHostMountPaths = $mvDetail.ClientConfig.ChannelHostMountPaths + $query.Var.input.update.slaClientConfig.backupScriptCommand = $mvDetail.ClientConfig.BackupScript.ScriptCommand + + # Only set pre-backup script fields if a pre-backup script was configured + if ($mvDetail.ClientConfig.PreBackupScript.ScriptCommand) { + $query.Var.input.update.slaClientConfig.preBackupScriptCommand = $mvDetail.ClientConfig.PreBackupScript.ScriptCommand + $query.Var.input.update.slaClientConfig.preBackupScriptTimeout = $mvDetail.ClientConfig.PreBackupScript.Timeout + $query.Var.input.update.slaClientConfig.shouldCancelBackupOnPreBackupScriptFailure = $mvDetail.ClientConfig.ShouldCancelBackupOnPreBackupScriptFailure + $query.Var.input.update.slaClientConfig.shouldDisablePreBackupScript = $false + } else { + $query.Var.input.update.slaClientConfig.shouldDisablePreBackupScript = $true + } + + $query.Var.input.update.slaClientConfig.shouldDisablePostBackupScriptOnBackupFailure = $true + $query.Var.input.update.slaClientConfig.shouldDisablePostBackupScriptOnBackupSuccess = $true + + $query.gqlRequest().Variables + + if (-not $dryrun) { + $query.Invoke() + } else { + Write-Log "Dry run mode: Managed Volume update not invoked." + } + + # Now must exit 1 to stop the backup continuing on the wrong node + Disconnect-Rsc + exit 1 + + } else { + Write-Log "SQL instance '$SqlInstance' is running on local node '$localNode'. No action needed." + } +} else { + Write-Log "ERROR: SQL instance '$SqlInstance' not found in cluster resources. Continuing assuming standalone instance." +} + +# Connect to Rubrik and retrieve managed volume paths +try { + Connect-Rsc -ServiceAccountFile $SAFile + Write-Log "INFO: Connected to Rubrik Security Cloud." + + $query = New-RscQuery -GqlQuery slaManagedVolumes -AddField Nodes.HostDetail, Nodes.SmbShare, Nodes.ClientConfig, Nodes.ClientConfig.BackupScript, Nodes.ClientConfig.PreBackupScript + $query.var.filter = @(Get-RscType -Name Filter) + $query.var.filter[0].field = "NAME_EXACT_MATCH" + $query.var.filter[0].Texts = $MvName + $mvDetail = $query.Invoke() + + if (-not $mvDetail.nodes -or $mvDetail.nodes.Count -eq 0) { + Write-Log "ERROR: Managed Volume '$MvName' not found. This may be due to insufficient permissions or the volume not existing." + exit 1 + } + + $paths = $mvDetail.nodes[0].ClientConfig.ChannelHostMountPaths + Write-Log "INFO: Retrieved paths: $($paths -join ', ')" +} catch { + Write-Log "ERROR: Failed to retrieve paths from Rubrik. $($_.Exception.Message)" + exit 1 +} + +# If -Nuke is set, delete the contents of each retrieved path (but keep the folder itself). +if ($Nuke) { + Write-Log "INFO: -nuke flag set. Beginning recursive deletion of contents for retrieved paths." + foreach ($p in $paths) { + if (-not $p) { continue } + $pathToCheck = $p.Trim() + + # Determine root to avoid deleting drive root like C:\ + try { $root = [IO.Path]::GetPathRoot($pathToCheck) } catch { $root = $null } + + if ([string]::IsNullOrEmpty($pathToCheck)) { + Write-Log "WARNING: Skipping empty path entry" + continue + } + + if ($root -and ($pathToCheck.TrimEnd('\') -eq $root.TrimEnd('\'))) { + Write-Log "ERROR: Refusing to nuke root path '$pathToCheck'. Skipping." + continue + } + + if (-not (Test-Path -LiteralPath $pathToCheck)) { + Write-Log "WARNING: Path '$pathToCheck' does not exist. Skipping." + continue + } + + Write-Log "INFO: NUKING contents of '$pathToCheck' (deleting all files & subfolders inside)." + try { + # Enumerate children and delete each item so the folder itself remains + Get-ChildItem -LiteralPath $pathToCheck -Force -ErrorAction SilentlyContinue | ForEach-Object { + try { + Remove-Item -LiteralPath $_.FullName -Recurse -Force -ErrorAction Stop + Write-Log "INFO: Deleted: $($_.FullName)" + } catch { + Write-Log "WARNING: Failed to delete $($_.FullName): $($_.Exception.Message)" + } + } + } catch { + Write-Log "ERROR: Failed to enumerate or delete contents of '$pathToCheck': $($_.Exception.Message)" + } + } + Write-Log "INFO: -nuke operation complete. Continuing with backup flow." +} + +$directoryParam = $paths -join ', ' + +# Validate job count +if ($Jobs -lt 1 -or $Jobs -gt 4) { + Write-Log "ERROR: Jobs parameter must be between 1 and 4. Provided: $Jobs" + exit 1 +} + +Write-Log "INFO: Starting $Jobs parallel backup jobs" -$fullFlag = $directory + "\last_full.flag" -$diffFlag = $directory + "\last_diff.flag" $today = (Get-Date).Date -function FlagTakenToday($flagPath) { - if (Test-Path $flagPath) { - $flagDate = (Get-Content $flagPath | Out-String).Trim() - return ($flagDate -eq $today.ToString("yyyy-MM-dd")) +function Get-BackupType($directoryParam) { + # Support multiple candidate directories. Scan them in deterministic order for existing flags. + $dirs = @() + if ($directoryParam) { + $dirs = $directoryParam -split ',' | ForEach-Object { $_.Trim() } | Where-Object { $_ -ne '' } } - return $false -} -function GetLastFullBackupDate($flagPath) { - if (Test-Path $flagPath) { - $flagDate = (Get-Content $flagPath | Out-String).Trim() + # Build lists of found flags (in candidate order) + $foundFull = @() + $foundDiff = @() + foreach ($d in $dirs) { + $full = Join-Path $d "last_full.flag" + $diff = Join-Path $d "last_diff.flag" + if (Test-Path $full) { $foundFull += $full } + if (Test-Path $diff) { $foundDiff += $diff } + } + + # Determine if full backup is overdue using the first-found full flag (if any) + $isFullBackupOverdue = $false + if ($foundFull.Count -gt 0) { + $fullFlag = $foundFull[0] try { - return [DateTime]::ParseExact($flagDate, "yyyy-MM-dd", $null) + $lastFullDate = [DateTime]::ParseExact((Get-Content $fullFlag).Trim(), "yyyy-MM-dd", $null) + $daysSinceLastFull = ($today - $lastFullDate).Days + $isFullBackupOverdue = $daysSinceLastFull -gt $fullBackupOverdueDays + Write-Log "INFO: Last full backup was $daysSinceLastFull days ago (from $fullFlag). Overdue threshold: $fullBackupOverdueDays days." + } catch { + $isFullBackupOverdue = $true + Write-Log "WARNING: Could not parse last full backup date in $fullFlag. Treating as overdue." } - catch { - Write-Log "WARNING: Could not parse last full backup date from flag file: $flagDate" - return $null + } else { + $isFullBackupOverdue = $true + Write-Log "WARNING: No last full backup date found in any candidate directories. Treating as overdue." + } + + # Helper to ensure directory exists + function Ensure-DirExists([string]$path) { + if (-not (Test-Path $path)) { + try { New-Item -ItemType Directory -Path $path -Force | Out-Null } catch { } } } - return $null -} -function IsFullBackupOverdue($flagPath, $overdueDays) { - $lastFullDate = GetLastFullBackupDate $flagPath - if ($null -eq $lastFullDate) { - Write-Log "WARNING: No last full backup date found. Full backup is considered overdue." - return $true - } - - $daysSinceLastFull = ($today - $lastFullDate).Days - $isOverdue = $daysSinceLastFull -gt $overdueDays - - Write-Log "INFO: Last full backup was $daysSinceLastFull days ago on $($lastFullDate.ToString('yyyy-MM-dd')). Overdue threshold: $overdueDays days." - - return $isOverdue -} + # Determine preferred write location: prefer existing related flag location, otherwise first candidate dir + $firstDir = $dirs[0] -function Write-Log($message) { - $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss" - $logEntry = "$timestamp $message" - Add-Content -Path $logFile -Value $logEntry - Write-Host $logEntry -} + # If it's a full backup day or overdue, plan for full backup + if ((Get-Date).DayOfWeek -eq $fullBackupDay -or $isFullBackupOverdue) { + # If a full flag exists, use its location; else use firstDir + $targetFullFlag = if ($foundFull.Count -gt 0) { $foundFull[0] } else { Join-Path $firstDir "last_full.flag" } + $targetDir = Split-Path $targetFullFlag -Parent + Ensure-DirExists $targetDir -# Check if directory exists and is a symbolic link (unless -Force is specified) -if (-not (Test-Path $directory)) { - Write-Log "ERROR: Directory '$directory' does not exist. Exiting script." - exit 1 -} + $currentValue = $null + if (Test-Path $targetFullFlag) { + try { $currentValue = (Get-Content $targetFullFlag).Trim() } catch { $currentValue = $null } + } -if (-not $Force) { - $directoryInfo = Get-Item $directory - if (-not ($directoryInfo.Attributes -band [System.IO.FileAttributes]::ReparsePoint)) { - Write-Log "ERROR: Directory '$directory' is not a symbolic link. Exiting script." - exit 1 - } - Write-Log "INFO: Directory '$directory' exists and is a symbolic link. Target: $($directoryInfo.Target). Proceeding." -} else { - Write-Log "INFO: Force parameter specified. Skipping symbolic link check for directory '$directory'." -} - -if ($checkCluster) { - # Check if SQL instance is running locally - $localNode = $env:COMPUTERNAME - - $clusterInstance = Get-ClusterResource | Where-Object { $_.ResourceType -eq "SQL Server" -and $_.Name -eq "SQL Server ($instanceName)" } - if ($clusterInstance) { - $ownerNode = $clusterInstance.OwnerNode.Name - if ($ownerNode -ne $localNode) { - Write-Log "SQL instance '$SqlInstance' is not running on local node '$localNode'. Exiting script." - exit 1 + if (-not $currentValue -or $currentValue -ne $today.ToString("yyyy-MM-dd")) { + try { + Set-Content -Path $targetFullFlag -Value $today.ToString("yyyy-MM-dd") -Encoding UTF8 + Write-Log "INFO: Created/Updated full backup flag file: $targetFullFlag" + } catch { + Write-Log "ERROR: Failed to create/update full backup flag file: $targetFullFlag. $($_.Exception.Message)" + } + $reason = if ($isFullBackupOverdue) { "overdue" } else { "scheduled" } + return @{ Type = "FULL"; CleanupTime = 168; Reason = $reason } } else { - Write-Log "SQL instance '$SqlInstance' is running on local node '$localNode'. Proceeding." + return @{ Type = "LOG"; CleanupTime = 24; Reason = "full already taken today" } } - } else { - Write-Log "ERROR: SQL instance '$SqlInstance' not found in cluster resources." - exit 1 } -} else { - Write-Log "INFO: Cluster check is disabled. Proceeding without verification." -} -# Check if full backup is overdue regardless of the day -$isFullBackupOverdue = IsFullBackupOverdue $fullFlag $fullBackupOverdueDays + # Otherwise, plan for differential + # Prefer an existing diff flag location if present; else prefer the existing full flag location (write diff alongside full); otherwise firstDir + if ($foundDiff.Count -gt 0) { + $targetDiffFlag = $foundDiff[0] + } elseif ($foundFull.Count -gt 0) { + $targetDiffFlag = Join-Path (Split-Path $foundFull[0] -Parent) "last_diff.flag" + } else { + $targetDiffFlag = Join-Path $firstDir "last_diff.flag" + } -if ((Get-Date).DayOfWeek -eq $fullBackupDay) { - if (-not (FlagTakenToday $fullFlag)) { - $backupType = "FULL" - $cleanupTime = 168 - Set-Content $fullFlag $today.ToString("yyyy-MM-dd") - Write-Log "Selected FULL backup (scheduled day). Flag updated." - } else { - $backupType = "LOG" - $cleanupTime = 24 - Write-Log "FULL backup already taken today. Selected LOG backup." + $targetDir = Split-Path $targetDiffFlag -Parent + Ensure-DirExists $targetDir + + $currentDiffValue = $null + if (Test-Path $targetDiffFlag) { + try { $currentDiffValue = (Get-Content $targetDiffFlag).Trim() } catch { $currentDiffValue = $null } } -} elseif ($isFullBackupOverdue) { - if (-not (FlagTakenToday $fullFlag)) { - $backupType = "FULL" - $cleanupTime = 168 - Set-Content $fullFlag $today.ToString("yyyy-MM-dd") - Write-Log "Selected FULL backup (overdue - forcing full backup). Flag updated." + + if (-not $currentDiffValue -or $currentDiffValue -ne $today.ToString("yyyy-MM-dd")) { + try { + Set-Content -Path $targetDiffFlag -Value $today.ToString("yyyy-MM-dd") -Encoding UTF8 + Write-Log "INFO: Created/Updated diff backup flag file: $targetDiffFlag" + } catch { + Write-Log "ERROR: Failed to create/update diff backup flag file: $targetDiffFlag. $($_.Exception.Message)" + } + return @{ Type = "DIFF"; CleanupTime = 168; Reason = "differential scheduled" } } else { - $backupType = "LOG" - $cleanupTime = 24 - Write-Log "FULL backup already taken today (was overdue). Selected LOG backup." - } -} else { - if (-not (FlagTakenToday $diffFlag)) { - $backupType = "DIFF" - $cleanupTime = 168 - Set-Content $diffFlag $today.ToString("yyyy-MM-dd") - Write-Log "Selected DIFF backup. Flag updated." - } else { - $backupType = "LOG" - $cleanupTime = 24 - Write-Log "DIFF backup already taken today. Selected LOG backup." + return @{ Type = "LOG"; CleanupTime = 24; Reason = "diff already taken today" } } } -$query = "EXECUTE [dbo].[DatabaseBackup] @Databases = 'ALL_DATABASES', @Directory = '$directory', @BackupType = '$backupType', @Verify = 'N', @CleanupTime = $cleanupTime, @CheckSum = 'Y', @LogToTable = 'Y'" -Write-Log "Executing backup type: $backupType" +# Determine backup type +$backupInfo = Get-BackupType $directoryParam +Write-Log "Selected $($backupInfo.Type) backup ($($backupInfo.Reason))" + +# Build the Ola H query. Include DatabasesInParallel only when multiple jobs are used +# Build parameter lines so we can avoid leaving a trailing comma when omitting DatabasesInParallel +$paramLines = @( + "@Databases = 'ALL_DATABASES'", + "@Directory = '$directoryParam'", + "@BackupType = '$($backupInfo.Type)'", + "@Verify = 'N'", + "@CleanupTime = $($backupInfo.CleanupTime)", + "@CheckSum = 'Y'", + "@LogToTable = 'Y'" +) + +# Only enable DatabasesInParallel when we run more than one job +if ($Jobs -gt 1) { + $paramLines += "@DatabasesInParallel = 'Y'" +} + +# Join with commas and indentation to produce clean SQL parameter list +$params = $paramLines -join ",`n " + +$query = "EXECUTE [dbo].[DatabaseBackup] `n $params" + Write-Log "SQL Query: $query" -try { - # Execute the backup using PowerShell SQL module with better error handling - # Capture verbose output from Ola H scripts - $infoMessages = @() - - # Create event handlers to capture SQL Server messages - $connection = New-Object System.Data.SqlClient.SqlConnection - $connection.ConnectionString = "Server=$SqlInstance;Integrated Security=true;Connection Timeout=30" - - # Event handler for informational messages (PRINT statements) - $connection.add_InfoMessage({ - param($sqlSender, $e) - $message = $e.Message - if ($message -and $message.Trim() -ne "") { - $script:infoMessages += $message - Write-Log "SQL INFO: $message" +# Function to execute backup job with message capture +function Start-BackupJob { + param( + [int]$jobId, + [string]$sqlInstance, + [string]$query, + [string]$baseLogFile + ) + $scriptBlock = { + param($JobId, $SqlInstance, $Query, $BaseLogFile) + + # Debug the base log file parameter + Write-Output "DEBUG: BaseLogFile parameter = '$BaseLogFile'" + + # Create job-specific log file path with fallback + if ($BaseLogFile -and $BaseLogFile.Trim() -ne "") { + $jobLogFile = $BaseLogFile -replace '\.log$', "-job$JobId.log" + } else { + # Fallback log file path using logDir + $jobLogFile = Join-Path $using:logDir "backup-multi-job$JobId.log" } - }) - - try { - $connection.Open() - $command = New-Object System.Data.SqlClient.SqlCommand - $command.Connection = $connection - $command.CommandText = $query - $command.CommandTimeout = 0 # No timeout for backup operations + Write-Output "DEBUG: Job log file will be: '$jobLogFile'" - Write-Log "Executing SQL command with message capture..." - - # Execute and capture any result sets - $reader = $command.ExecuteReader() - - # Process any result sets - while ($reader.Read()) { - $rowData = @() - for ($i = 0; $i -lt $reader.FieldCount; $i++) { - $rowData += "$($reader.GetName($i)): $($reader.GetValue($i))" + function Write-JobLog($message, $suppressConsole = $false) { + $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss" + $logEntry = "$timestamp [JOB-$JobId] $message" + if ($jobLogFile -and $jobLogFile.Trim() -ne "") { + try { + Add-Content -Path $jobLogFile -Value $logEntry -Encoding UTF8 + # Output to console for debugging (unless suppressed) + if (-not $suppressConsole) { + Write-Output "LOGGED TO $jobLogFile : $logEntry" + } + } catch { + Write-Output "LOG ERROR: $($_.Exception.Message) - File: $jobLogFile" + } + } else { + Write-Output "NO LOG FILE: jobLogFile is empty or null" } - if ($rowData.Count -gt 0) { - Write-Log "SQL RESULT: $($rowData -join ', ')" + # Always output to console for job monitoring (unless suppressed) + if (-not $suppressConsole) { + Write-Output $logEntry } } - $reader.Close() - } - finally { - if ($connection.State -eq [System.Data.ConnectionState]::Open) { - $connection.Close() - } - $connection.Dispose() - } - - Write-Log "$backupType Backup execution completed successfully." - Write-Log "Total informational messages captured: $($infoMessages.Count)" -} -catch { - Write-Log "ERROR: Backup execution failed with exception: $($_.Exception.Message)" - - # Log additional SQL Server error details if available - if ($_.Exception.InnerException) { - Write-Log "ERROR: Inner Exception: $($_.Exception.InnerException.Message)" - } - - # Check for SQL Server specific errors - if ($_.Exception -is [System.Data.SqlClient.SqlException]) { - Write-Log "ERROR: SQL Server Error Details:" - foreach ($sqlError in $_.Exception.Errors) { - Write-Log "ERROR: Severity: $($sqlError.Class), State: $($sqlError.State), Number: $($sqlError.Number)" - Write-Log "ERROR: Message: $($sqlError.Message)" - if ($sqlError.Procedure) { - Write-Log "ERROR: Procedure: $($sqlError.Procedure), Line: $($sqlError.LineNumber)" + try { + Write-JobLog "Starting backup job" + + # Create SQL connection with message capture + $connection = New-Object System.Data.SqlClient.SqlConnection + $connection.ConnectionString = "Server=$SqlInstance;Integrated Security=true;Connection Timeout=30" + + $infoMessages = @() + + # Event handler for informational messages (PRINT statements) + $connection.add_InfoMessage({ + param($sqlSender, $e) + $message = $e.Message + if ($message -and $message.Trim() -ne "") { + $script:infoMessages += $message + Write-JobLog "SQL INFO: $message" $true # Suppress console output for verbose messages + } + }) + + try { + Write-JobLog "Attempting to connect to SQL Server: $SqlInstance" + $connection.Open() + Write-JobLog "Connected to SQL Server successfully" + + $command = New-Object System.Data.SqlClient.SqlCommand + $command.Connection = $connection + $command.CommandText = $Query + $command.CommandTimeout = 0 # No timeout for backup operations + + Write-JobLog "Executing backup command..." + + # Execute and capture any result sets + $reader = $command.ExecuteReader() + + # Process any result sets + while ($reader.Read()) { + $rowData = @() + for ($i = 0; $i -lt $reader.FieldCount; $i++) { + $rowData += "$($reader.GetName($i)): $($reader.GetValue($i))" + } + if ($rowData.Count -gt 0) { + $resultLine = "SQL RESULT: $($rowData -join ', ')" + Write-JobLog $resultLine + Write-Output $resultLine # Also output for Receive-Job + } + } + + $reader.Close() + $summaryMessage = "Backup completed successfully. Captured $($infoMessages.Count) messages." + Write-JobLog $summaryMessage + Write-Output $summaryMessage # Also output for Receive-Job + + # Output all captured SQL messages for debugging (only to log file, not console) + Write-JobLog "=== SQL MESSAGES START ===" $true + foreach ($msg in $infoMessages) { + Write-JobLog "SQL: $msg" $true + } + Write-JobLog "=== SQL MESSAGES END ===" $true + + # Don't return hashtable - just output success message + Write-Output "JOB-${JobId}: SUCCESS" + } + finally { + if ($connection.State -eq [System.Data.ConnectionState]::Open) { + $connection.Close() + } + $connection.Dispose() } } + catch { + $errorMessage = "ERROR: Backup failed - $($_.Exception.Message)" + Write-JobLog $errorMessage + Write-Output $errorMessage # Also output for Receive-Job + + # Check for specific connection errors + if ($_.Exception.Message -like "*server*not found*" -or + $_.Exception.Message -like "*network-related*" -or + $_.Exception.Message -like "*instance*" -or + $_.Exception.Message -like "*login*failed*") { + $connError = "ERROR: CONNECTION FAILURE - Check SQL Server instance name and connectivity" + Write-JobLog $connError + Write-Output $connError + } + + # Log SQL Server specific errors + if ($_.Exception -is [System.Data.SqlClient.SqlException]) { + Write-JobLog "ERROR: SQL Server Error Details:" + Write-Output "ERROR: SQL Server Error Details:" + foreach ($sqlError in $_.Exception.Errors) { + $errorDetail = "ERROR: Severity: $($sqlError.Class), State: $($sqlError.State), Number: $($sqlError.Number)" + Write-JobLog $errorDetail + Write-Output $errorDetail + + $errorMsg = "ERROR: Message: $($sqlError.Message)" + Write-JobLog $errorMsg + Write-Output $errorMsg + + if ($sqlError.Procedure) { + $procError = "ERROR: Procedure: $($sqlError.Procedure), Line: $($sqlError.LineNumber)" + Write-JobLog $procError + Write-Output $procError + } + } + } + + # Log full exception details for debugging + $fullError = "ERROR: Full Exception Type: $($_.Exception.GetType().Name)" + Write-JobLog $fullError + Write-Output $fullError + + if ($_.Exception.InnerException) { + $innerError = "ERROR: Inner Exception: $($_.Exception.InnerException.Message)" + Write-JobLog $innerError + Write-Output $innerError + } + + Write-Output "JOB-${JobId}: FAILED" + } } - # Clean up connection if it exists - if ($connection -and $connection.State -eq [System.Data.ConnectionState]::Open) { - $connection.Close() - $connection.Dispose() - } - - exit 1 + return Start-Job -ScriptBlock $scriptBlock -ArgumentList $jobId, $sqlInstance, $query, $baseLogFile } +# Start parallel backup jobs +Write-Log "Starting $Jobs parallel backup jobs" +[System.Collections.ArrayList]$jobList = @() + +for ($i = 1; $i -le $Jobs; $i++) { + $job = Start-BackupJob -jobId $i -sqlInstance $SqlInstance -query $query -baseLogFile $logFile + $null = $jobList.Add($job) + Write-Log "Started backup job $i (Job ID: $($job.Id))" + Start-Sleep -Milliseconds 4000 # Delay to stagger job starts +} + +# Monitor jobs and capture output +Write-Log "Monitoring $($jobList.Count) backup jobs..." +$allJobsCompleted = $false +[System.Collections.ArrayList]$completedJobs = @() + +while (-not $allJobsCompleted) { + Start-Sleep -Seconds 5 + + foreach ($job in $jobList) { + if ($job.Id -notin $completedJobs) { + # Check if job is no longer running + if ($job.State -eq "Completed" -or $job.State -eq "Failed" -or $job.State -eq "Stopped") { + $null = $completedJobs.Add($job.Id) + + # Get all job output + $jobOutput = Receive-Job -Job $job -Keep # Use -Keep to preserve output + + if ($job.State -eq "Completed") { + Write-Log "Job $($job.Id) completed successfully" + + # Log all job output to main log + if ($jobOutput) { + Write-Log "=== Job $($job.Id) Output ===" + foreach ($line in $jobOutput) { + Write-Log "$line" + } + Write-Log "=== End Job $($job.Id) Output ===" + } + } else { + Write-Log "ERROR: Job $($job.Id) failed with state: $($job.State)" + if ($jobOutput) { + Write-Log "=== Job $($job.Id) Error Output ===" + foreach ($line in $jobOutput) { + Write-Log "ERROR: $line" + } + Write-Log "=== End Job $($job.Id) Error Output ===" + } + } + } + } + } + + $allJobsCompleted = $completedJobs.Count -eq $jobList.Count + + # Progress update + $runningCount = ($jobList | Where-Object { $_.State -eq "Running" }).Count + if ($runningCount -gt 0) { + Write-Log "Progress: $($completedJobs.Count)/$($jobList.Count) jobs completed, $runningCount still running..." + } +} + +Write-Log "All backup jobs completed" + +# Collect job states and outputs before cleanup for final status check +$jobResults = @{} +foreach ($job in $jobList) { + $jobOutput = Receive-Job -Job $job -Keep -ErrorAction SilentlyContinue + $hasFailed = $false + + # Check if job output contains failure indicator + if ($jobOutput) { + foreach ($line in $jobOutput) { + if ($line -like "*JOB-*: FAILED") { + $hasFailed = $true + break + } + } + } + + $jobResults[$job.Id] = @{ + State = $job.State + Failed = $hasFailed + } +} + +# Clean up jobs +Write-Log "Cleaning up completed jobs..." +foreach ($job in $jobList) { + try { + if ($job.State -eq "Running") { + Write-Log "WARNING: Job $($job.Id) still running, stopping it..." + Stop-Job -Job $job -Force + Start-Sleep -Seconds 2 + } + Remove-Job -Job $job -Force -ErrorAction SilentlyContinue + Write-Log "Cleaned up job $($job.Id)" + } catch { + Write-Log "WARNING: Could not clean up job $($job.Id): $($_.Exception.Message)" + } +} + +# Final status check using job output analysis +$failedJobIds = $jobResults.Keys | Where-Object { $jobResults[$_].Failed -eq $true } + +if ($failedJobIds.Count -gt 0) { + Write-Log "ERROR: $($failedJobIds.Count) out of $($jobResults.Count) backup jobs failed" + foreach ($jobId in $failedJobIds) { + Write-Log "ERROR: Job ID $jobId failed" + } + Write-Log "CRITICAL: Backup operation failed - check errors above" + exit 1 +} else { + Write-Log "SUCCESS: All $($jobResults.Count) backup jobs completed successfully" +} \ No newline at end of file diff --git a/backupSingle.ps1 b/backupSingle.ps1 new file mode 100644 index 0000000..615d110 --- /dev/null +++ b/backupSingle.ps1 @@ -0,0 +1,271 @@ +param( + [Parameter(Mandatory=$true)] + [string]$SqlInstance, + + [Parameter(Mandatory=$false)] + [string]$Directory, + + [Parameter(Mandatory=$false)] + [switch]$Force +) + +# backup.ps1 +# +# TODO: Parallelize backups for multiple DBs in the instance + +# Import SQL Server PowerShell module +try { + # Try to import the newer SqlServer module first + if (Get-Module -ListAvailable -Name SqlServer) { + Import-Module SqlServer -ErrorAction Stop + Write-Host "INFO: SqlServer PowerShell module loaded successfully." + } + # Fall back to older SQLPS module if available + elseif (Get-Module -ListAvailable -Name SQLPS) { + Import-Module SQLPS -ErrorAction Stop + Write-Host "INFO: SQLPS PowerShell module loaded successfully." + } + else { + throw "No SQL Server PowerShell module found" + } + + # Verify Invoke-Sqlcmd is available + if (-not (Get-Command Invoke-Sqlcmd -ErrorAction SilentlyContinue)) { + throw "Invoke-Sqlcmd command not available" + } +} +catch { + Write-Host "ERROR: Failed to import SQL Server PowerShell module. Please install it using: Install-Module -Name SqlServer -AllowClobber" + Write-Host "ERROR: $($_.Exception.Message)" + exit 1 +} + +$instanceName = $SqlInstance.Split('\')[1] + +# Use provided directory parameter or default to instance-based path +if ($Directory) { + $directory = $Directory + Write-Host "INFO: Using provided directory: $directory" +} else { + $directory = "C:\Rubrik\$instanceName" + Write-Host "INFO: Using default directory: $directory" +} + +$fullBackupDay = 'Thursday' +$fullBackupOverdueDays = 7 # Force full backup if last full backup is older than this many days +$checkCluster = $false +#$logFile = "C:\Rubrik\backup-$instanceName.log" +$logFile = "H:\Backup\backup-$instanceName.log" + +$fullFlag = $directory + "\last_full.flag" +$diffFlag = $directory + "\last_diff.flag" +$today = (Get-Date).Date + +function FlagTakenToday($flagPath) { + if (Test-Path $flagPath) { + $flagDate = (Get-Content $flagPath | Out-String).Trim() + return ($flagDate -eq $today.ToString("yyyy-MM-dd")) + } + return $false +} + +function GetLastFullBackupDate($flagPath) { + if (Test-Path $flagPath) { + $flagDate = (Get-Content $flagPath | Out-String).Trim() + try { + return [DateTime]::ParseExact($flagDate, "yyyy-MM-dd", $null) + } + catch { + Write-Log "WARNING: Could not parse last full backup date from flag file: $flagDate" + return $null + } + } + return $null +} + +function IsFullBackupOverdue($flagPath, $overdueDays) { + $lastFullDate = GetLastFullBackupDate $flagPath + if ($null -eq $lastFullDate) { + Write-Log "WARNING: No last full backup date found. Full backup is considered overdue." + return $true + } + + $daysSinceLastFull = ($today - $lastFullDate).Days + $isOverdue = $daysSinceLastFull -gt $overdueDays + + Write-Log "INFO: Last full backup was $daysSinceLastFull days ago on $($lastFullDate.ToString('yyyy-MM-dd')). Overdue threshold: $overdueDays days." + + return $isOverdue +} + +function Write-Log($message) { + $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss" + $logEntry = "$timestamp $message" + Add-Content -Path $logFile -Value $logEntry + Write-Host $logEntry +} + +# Check if directory exists and is a symbolic link (unless -Force is specified) +if (-not (Test-Path $directory)) { + Write-Log "ERROR: Directory '$directory' does not exist. Exiting script." + exit 1 +} + +if (-not $Force) { + $directoryInfo = Get-Item $directory + if (-not ($directoryInfo.Attributes -band [System.IO.FileAttributes]::ReparsePoint)) { + Write-Log "ERROR: Directory '$directory' is not a symbolic link. Exiting script." + exit 1 + } + Write-Log "INFO: Directory '$directory' exists and is a symbolic link. Target: $($directoryInfo.Target). Proceeding." +} else { + Write-Log "INFO: Force parameter specified. Skipping symbolic link check for directory '$directory'." +} + +if ($checkCluster) { + # Check if SQL instance is running locally + $localNode = $env:COMPUTERNAME + + $clusterInstance = Get-ClusterResource | Where-Object { $_.ResourceType -eq "SQL Server" -and $_.Name -eq "SQL Server ($instanceName)" } + if ($clusterInstance) { + $ownerNode = $clusterInstance.OwnerNode.Name + if ($ownerNode -ne $localNode) { + Write-Log "SQL instance '$SqlInstance' is not running on local node '$localNode'. Exiting script." + exit 1 + } else { + Write-Log "SQL instance '$SqlInstance' is running on local node '$localNode'. Proceeding." + } + } else { + Write-Log "ERROR: SQL instance '$SqlInstance' not found in cluster resources." + exit 1 + } +} else { + Write-Log "INFO: Cluster check is disabled. Proceeding without verification." +} + +# Check if full backup is overdue regardless of the day +$isFullBackupOverdue = IsFullBackupOverdue $fullFlag $fullBackupOverdueDays + +if ((Get-Date).DayOfWeek -eq $fullBackupDay) { + if (-not (FlagTakenToday $fullFlag)) { + $backupType = "FULL" + $cleanupTime = 168 + Set-Content $fullFlag $today.ToString("yyyy-MM-dd") + Write-Log "Selected FULL backup (scheduled day). Flag updated." + } else { + $backupType = "LOG" + $cleanupTime = 24 + Write-Log "FULL backup already taken today. Selected LOG backup." + } +} elseif ($isFullBackupOverdue) { + if (-not (FlagTakenToday $fullFlag)) { + $backupType = "FULL" + $cleanupTime = 168 + Set-Content $fullFlag $today.ToString("yyyy-MM-dd") + Write-Log "Selected FULL backup (overdue - forcing full backup). Flag updated." + } else { + $backupType = "LOG" + $cleanupTime = 24 + Write-Log "FULL backup already taken today (was overdue). Selected LOG backup." + } +} else { + if (-not (FlagTakenToday $diffFlag)) { + $backupType = "DIFF" + $cleanupTime = 168 + Set-Content $diffFlag $today.ToString("yyyy-MM-dd") + Write-Log "Selected DIFF backup. Flag updated." + } else { + $backupType = "LOG" + $cleanupTime = 24 + Write-Log "DIFF backup already taken today. Selected LOG backup." + } +} + +$query = "EXECUTE [dbo].[DatabaseBackup] @Databases = 'ALL_DATABASES', @Directory = '$directory', @BackupType = '$backupType', @Verify = 'N', @CleanupTime = $cleanupTime, @CheckSum = 'Y', @LogToTable = 'Y'" +Write-Log "Executing backup type: $backupType" +Write-Log "SQL Query: $query" + +try { + # Execute the backup using PowerShell SQL module with better error handling + # Capture verbose output from Ola H scripts + $infoMessages = @() + + # Create event handlers to capture SQL Server messages + $connection = New-Object System.Data.SqlClient.SqlConnection + $connection.ConnectionString = "Server=$SqlInstance;Integrated Security=true;Connection Timeout=30" + + # Event handler for informational messages (PRINT statements) + $connection.add_InfoMessage({ + param($sqlSender, $e) + $message = $e.Message + if ($message -and $message.Trim() -ne "") { + $script:infoMessages += $message + Write-Log "SQL INFO: $message" + } + }) + + try { + $connection.Open() + + $command = New-Object System.Data.SqlClient.SqlCommand + $command.Connection = $connection + $command.CommandText = $query + $command.CommandTimeout = 0 # No timeout for backup operations + + Write-Log "Executing SQL command with message capture..." + + # Execute and capture any result sets + $reader = $command.ExecuteReader() + + # Process any result sets + while ($reader.Read()) { + $rowData = @() + for ($i = 0; $i -lt $reader.FieldCount; $i++) { + $rowData += "$($reader.GetName($i)): $($reader.GetValue($i))" + } + if ($rowData.Count -gt 0) { + Write-Log "SQL RESULT: $($rowData -join ', ')" + } + } + + $reader.Close() + } + finally { + if ($connection.State -eq [System.Data.ConnectionState]::Open) { + $connection.Close() + } + $connection.Dispose() + } + + Write-Log "$backupType Backup execution completed successfully." + Write-Log "Total informational messages captured: $($infoMessages.Count)" +} +catch { + Write-Log "ERROR: Backup execution failed with exception: $($_.Exception.Message)" + + # Log additional SQL Server error details if available + if ($_.Exception.InnerException) { + Write-Log "ERROR: Inner Exception: $($_.Exception.InnerException.Message)" + } + + # Check for SQL Server specific errors + if ($_.Exception -is [System.Data.SqlClient.SqlException]) { + Write-Log "ERROR: SQL Server Error Details:" + foreach ($sqlError in $_.Exception.Errors) { + Write-Log "ERROR: Severity: $($sqlError.Class), State: $($sqlError.State), Number: $($sqlError.Number)" + Write-Log "ERROR: Message: $($sqlError.Message)" + if ($sqlError.Procedure) { + Write-Log "ERROR: Procedure: $($sqlError.Procedure), Line: $($sqlError.LineNumber)" + } + } + } + + # Clean up connection if it exists + if ($connection -and $connection.State -eq [System.Data.ConnectionState]::Open) { + $connection.Close() + $connection.Dispose() + } + + exit 1 +} + diff --git a/failover.cmd b/failover.cmd deleted file mode 100644 index b2d9f49..0000000 --- a/failover.cmd +++ /dev/null @@ -1,2 +0,0 @@ -@echo off -"C:\Program Files\PowerShell\7\pwsh.exe" -ExecutionPolicy Bypass -File "%~dp0claimInstance.ps1" diff --git a/release_v0/backup.txt b/release_v0/backup.txt new file mode 100644 index 0000000..d402935 --- /dev/null +++ b/release_v0/backup.txt @@ -0,0 +1,90 @@ +# +# backup.ps1 +# +# TODO: Update cleanup time based on backup type + +$sqlInstance = "sqlfcsql\TESTINST" +#$directory = "H:\Backup" +$directory = "C:\Rubrik\mount" +$fullBackupDay = 'Thursday' +$checkCluster = $false +$logFile = "C:\Rubrik\backup.log" + +$fullFlag = $directory + "\last_full.flag" +$diffFlag = $directory + "\last_diff.flag" +$today = (Get-Date).Date + +function FlagTakenToday($flagPath) { + if (Test-Path $flagPath) { + $flagDate = (Get-Content $flagPath | Out-String).Trim() + return ($flagDate -eq $today.ToString("yyyy-MM-dd")) + } + return $false +} + +function Write-Log($message) { + $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss" + $logEntry = "$timestamp $message" + Add-Content -Path $logFile -Value $logEntry + Write-Host $logEntry +} + +if ($checkCluster) { + # Check if SQL instance is running locally + $localNode = $env:COMPUTERNAME + $instanceName = $sqlInstance.Split('\')[1] + $clusterInstance = Get-ClusterResource | Where-Object { $_.ResourceType -eq "SQL Server" -and $_.Name -eq "SQL Server ($instanceName)" } + if ($clusterInstance) { + $ownerNode = $clusterInstance.OwnerNode.Name + if ($ownerNode -ne $localNode) { + Write-Log "SQL instance '$sqlInstance' is not running on local node '$localNode'. Exiting script." + exit 1 + } else { + Write-Log "SQL instance '$sqlInstance' is running on local node '$localNode'. Proceeding." + } + } else { + Write-Log "ERROR: SQL instance '$sqlInstance' not found in cluster resources." + exit 1 + } +} else { + Write-Log "INFO: Cluster check is disabled. Proceeding without verification." +} + +if ((Get-Date).DayOfWeek -eq $fullBackupDay) { + if (-not (FlagTakenToday $fullFlag)) { + $backupType = "FULL" + $cleanupTime = 168 + Set-Content $fullFlag $today.ToString("yyyy-MM-dd") + Write-Log "Selected FULL backup. Flag updated." + } else { + $backupType = "LOG" + Write-Log "FULL backup already taken today. Selected LOG backup." + } +} else { + if (-not (FlagTakenToday $diffFlag)) { + $backupType = "DIFF" + $cleanupTime = 168 + Set-Content $diffFlag $today.ToString("yyyy-MM-dd") + Write-Log "Selected DIFF backup. Flag updated." + } else { + $backupType = "LOG" + $cleanupTime = 24 + Write-Log "DIFF backup already taken today. Selected LOG backup." + } +} + +$query = "EXECUTE [dbo].[DatabaseBackup] @Databases = 'ALL_DATABASES', @Directory = '$directory', @BackupType = '$backupType', @Verify = 'N', @CleanupTime = $cleanupTime, @CheckSum = 'Y', @LogToTable = 'Y'" +Write-Log "Executing backup type: $backupType" + +$sqlcmdOutput = & sqlcmd -S $sqlInstance -Q $query 2>&1 +$sqlcmdExitCode = $LASTEXITCODE + +if ($sqlcmdExitCode -eq 0) { + foreach ($line in $sqlcmdOutput) { + Write-Log $line + } + Write-Log "$backupType Backup execution completed." +} else { + Write-Log "ERROR: Backup execution failed. Exit code: $sqlcmdExitCode. Output: $sqlcmdOutput" +} + diff --git a/release_v0/start.txt b/release_v0/start.txt new file mode 100644 index 0000000..ee7e0ce --- /dev/null +++ b/release_v0/start.txt @@ -0,0 +1,3 @@ +:: filepath: c:\Rubrik\Scripts\start.cmd +@echo off +powershell -ExecutionPolicy Bypass -File "%~dp0backup.ps1" %* \ No newline at end of file diff --git a/backupmult.ps1 b/release_v1/backup.ps1 similarity index 96% rename from backupmult.ps1 rename to release_v1/backup.ps1 index b7c13b7..6436f24 100644 --- a/backupmult.ps1 +++ b/release_v1/backup.ps1 @@ -15,7 +15,7 @@ param( [switch]$Nuke ) -# backupmult.ps1 - Parallel database backup script using Ola H +# backup.ps1 - Parallel database backup script using Ola H # # Uses Ola H's built-in parallel processing by starting multiple concurrent backup jobs # Each job will automatically share the database load using DatabasesInParallel=Y if Jobs>1 @@ -26,6 +26,7 @@ param( $fullBackupDay = 'Thursday' $fullBackupOverdueDays = 7 $SAFile = "C:\Rubrik\scripts\rbksql.xml" +$logDir = "C:\Rubrik\logs" function Write-Log($message, $jobId = "") { $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss" @@ -56,9 +57,22 @@ function Write-Log($message, $jobId = "") { # Parse instance name from SQL instance parameter $instanceParts = $SqlInstance -split '\\' if ($instanceParts.Length -eq 2) { + # Format: HOSTNAME\INSTANCENAME or CLUSTERNAME\INSTANCENAME $instanceName = $instanceParts[1] + # SqlInstance stays as provided (HOSTNAME\INSTANCENAME) } elseif ($instanceParts.Length -eq 1) { - $instanceName = $instanceParts[0] + # Single value provided - could be hostname (default instance) or instance name on local host + $singleValue = $instanceParts[0] + + # If it's "MSSQLSERVER" (default instance name), treat as default instance on local host + if ($singleValue -eq "MSSQLSERVER") { + $instanceName = "MSSQLSERVER" + $SqlInstance = $env:COMPUTERNAME # Connect to default instance (no instance name) + } else { + # Assume it's an instance name on the local host + $instanceName = $singleValue + $SqlInstance = "$($env:COMPUTERNAME)\$singleValue" + } } else { $instanceName = $SqlInstance.Replace('\\', '_').Replace('/', '_') } @@ -77,7 +91,6 @@ foreach ($c in $invalidChars) { } $timestamp = Get-Date -Format "yyyyMMdd_HHmmss" -$logDir = "C:\Rubrik\logs" # Ensure log directory exists before building/using log file if (-not (Test-Path $logDir)) { try { @@ -97,7 +110,6 @@ Write-Log "DEBUG: SqlInstance='$SqlInstance', instanceName='$instanceName', logF function Remove-OldLogs { param([int]$retentionDays) - $logDir = "C:\Rubrik\logs" if (-not (Test-Path $logDir)) { try { New-Item -ItemType Directory -Path $logDir -Force | Out-Null @@ -233,8 +245,7 @@ if ($clusterInstance) { Write-Log "SQL instance '$SqlInstance' is running on local node '$localNode'. No action needed." } } else { - Write-Log "ERROR: SQL instance '$SqlInstance' not found in cluster resources." - exit 1 + Write-Log "ERROR: SQL instance '$SqlInstance' not found in cluster resources. Continuing assuming standalone instance." } # Connect to Rubrik and retrieve managed volume paths @@ -463,8 +474,8 @@ function Start-BackupJob { if ($BaseLogFile -and $BaseLogFile.Trim() -ne "") { $jobLogFile = $BaseLogFile -replace '\.log$', "-job$JobId.log" } else { - # Fallback log file path - $jobLogFile = "C:\Rubrik\logs\backup-multi-job$JobId.log" + # Fallback log file path using logDir + $jobLogFile = Join-Path $using:logDir "backup-multi-job$JobId.log" } Write-Output "DEBUG: Job log file will be: '$jobLogFile'" diff --git a/setActiveNode.ps1 b/setActiveNode.ps1 index b5e1cb8..ebdbb4c 100644 --- a/setActiveNode.ps1 +++ b/setActiveNode.ps1 @@ -33,7 +33,7 @@ param ( ) # SA File must be an absolute path -$SAFile = "C:\Rubrik\scripts\sa.xml" +$SAFile = "C:\Rubrik\scripts\rbksql.xml" $logFile = "C:\Rubrik\scripts\setActiveNode.log" $checkCluster = $true @@ -90,12 +90,18 @@ if ($checkCluster) { $query.Var.input.update.slaClientConfig.clientHostId = $newHost.Id $query.Var.input.update.slaClientConfig.channelHostMountPaths = $mvDetail.ClientConfig.ChannelHostMountPaths $query.Var.input.update.slaClientConfig.backupScriptCommand = $mvDetail.ClientConfig.BackupScript.ScriptCommand - $query.Var.input.update.slaClientConfig.preBackupScriptCommand = $mvDetail.ClientConfig.PreBackupScript.ScriptCommand - $query.Var.input.update.slaClientConfig.preBackupScriptTimeout = $mvDetail.ClientConfig.PreBackupScript.Timeout + # Only set pre-backup script fields if a pre-backup script was configured + if ($mvDetail.ClientConfig.PreBackupScript.ScriptCommand) { + $query.Var.input.update.slaClientConfig.preBackupScriptCommand = $mvDetail.ClientConfig.PreBackupScript.ScriptCommand + $query.Var.input.update.slaClientConfig.preBackupScriptTimeout = $mvDetail.ClientConfig.PreBackupScript.Timeout + $query.Var.input.update.slaClientConfig.shouldCancelBackupOnPreBackupScriptFailure = $mvDetail.ClientConfig.ShouldCancelBackupOnPreBackupScriptFailure + $query.Var.input.update.slaClientConfig.shouldDisablePreBackupScript = $false + } else { + $query.Var.input.update.slaClientConfig.shouldDisablePreBackupScript = $true + } + $query.Var.input.update.slaClientConfig.shouldDisablePostBackupScriptOnBackupFailure = $true $query.Var.input.update.slaClientConfig.shouldDisablePostBackupScriptOnBackupSuccess = $true - $query.Var.input.update.slaClientConfig.shouldDisablePreBackupScript = $false - $query.Var.input.update.slaClientConfig.shouldCancelBackupOnPreBackupScriptFailure = $mvDetail.ClientConfig.ShouldCancelBackupOnPreBackupScriptFailure $query.gqlRequest().Variables