PowerShell DSC: Remote Monitoring Configuration Propagation

So if you are like me you are not really interested in crossing your fingers and hoping your servers are working right. Which is why it is uniquely frustrating that DSC does not have anything resembling a dashboard (not a complaint really, it is early days, but in practical application not knowing something went down is...not really an option unless you like being sloppy).

The way I build my servers is, I have an XML file with a list of servers, their role, and their role GUID. Baked into the master image is a simple bootstrap script that goes and gets the build script, since I'm using DSC the "build" script doesn't really build much, itself mostly just bootstrapping the DSC process. The first script to run is:

$nodeloc = "\\dscserver\DSC\Nodes\nodes.xml"

# Get node information.
try {
	[xml]$nodes = Get-Content -Path $nodeloc -ErrorAction 'Stop'
	$role = $nodes.hostname.$env:COMPUTERNAME.role
}
catch{ Write-Host "Could not find matching node, exiting.";Break }

# Set correct build script location.
switch($role) {
	"XenAppPKG" { $scriptloc = "\\dscserver\DSC\Scripts\pkgbuild.ps1" }
	"XenAppQA" { $scriptloc = "\\dscserver\DSC\Scripts\qabuild.ps1" }
	"XenAppProd" { $scriptloc = "\\dscserver\DSC\Scripts\prodbuild.ps1" }
}

Write-Host "Script location set to:"$scriptloc
if((Test-Path -Path "C:\scripts") -ne $true){ New-Item -Path "C:\scripts" -ItemType Directory -Force -ErrorAction 'Stop' }
Write-Host "Checking build script availability..."
while((Test-Path -Path $scriptloc) -ne $true){ Start-Sleep -Seconds 15 }
Write-Host "Fetching build script..."
while((Test-Path -Path "C:\scripts\build.ps1") -ne $true){ Copy-Item -Path $scriptloc -Destination "C:\scripts\build.ps1" -ErrorAction 'SilentlyContinue' }
Write-Host "Executing build script..."
& C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -file "C:\scripts\build.ps1"

The information it looks for in the nodes.xml file looks like this:

<hostname>
	<A01 role="XenAppProd" guid="22e35281-49c6-40f3-9fd7-ad7f8d69c84d" />
	<A02 role="XenAppProd" guid="22e35281-49c6-40f3-9fd7-ad7f8d69c84d" />
	<A03 role="XenAppProd" guid="22e35281-49c6-40f3-9fd7-ad7f8d69c84d" />
	<A04 role="XenAppProd" guid="22e35281-49c6-40f3-9fd7-ad7f8d69c84d" />
	<B01 role="XenAppProd" guid="22e35281-49c6-40f3-9fd7-ad7f8d69c84d" />
	<B02 role="XenAppProd" guid="22e35281-49c6-40f3-9fd7-ad7f8d69c84d" />
	<B03 role="XenAppProd" guid="22e35281-49c6-40f3-9fd7-ad7f8d69c84d" />
	<B04 role="XenAppProd" guid="22e35281-49c6-40f3-9fd7-ad7f8d69c84d" />
</hostname>

I wont go any further into this as most of it has already been covered here before, the main gist of this is, my solution to this problem relies on the fact that I use the XML file to provision DSC on these machines.

There are a couple modifications I need to make to my DSC config to enable tracking, note the first item are only there so I can override the GUID from the CMDLine if I want. In reality you could just set the ValueData to ([GUID]::NewGUID()).ToString() and be fine.

The first bit of code take place before I start my Configuration block, the actual Registry resource is the very last resource in the Configuration block (less chance of false-positives due to an error mid-config).

param (
	[string]$guid = ([GUID]::NewGuid()).ToString()
)

...

Registry verGUID {
	Ensure = "Present"
	Key = "HKLM:\SOFTWARE\PostBuild"
	ValueName = "verGuid"
	ValueData = $verGUID
	ValueType = "String"
}

From here we get to the important part:

[regex]$node = '(\[Registry\]verGUID[A-Za-z0-9\";\r\n\s=:\\ \-\.\{]*)'
[regex]$guid = '([a-z0-9\-]{36})'
$path = "\\dscserver\Configuration\"
$pkg = @()
$qa = @()
$prod = @()
$watch = @{}
$complete = @{}
[xml]$nodes = (Get-Content "\\dscserver\DSC\Nodes\nodes.xml")

# Find a list of machine names and role guids.
foreach($child in $nodes.hostname.ChildNodes) {
	switch($child.Role)
	{
		"XenAppPKG" { $pkg += $child.Name;$pkgGuid = $child.guid }
		"XenAppQA" { $qa += $child.Name;$qaGuid = $child.guid }
		"XenAppProd" { $prod += $child.Name;$prodGuid = $child.guid }
	}
}

# Convert DSC GUID's to latest verGUID.
$pkgGuid = $guid.Match(($node.Match((Get-Content -Path ($path+$pkgGuid+".mof")))).Captures.Value).Captures.Value
$qaGuid = $guid.Match(($node.Match((Get-Content -Path ($path+$qaGuid+".mof")))).Captures.Value).Captures.Value
$prodGuid = $guid.Match(($node.Match((Get-Content -Path ($path+$prodGuid+".mof")))).Captures.Value).Captures.Value

# See if credentials exist in this session.
if($creds -eq $null){ $creds = (Get-Credential) }

# Make an initial pass, determine configured/incomplete servers.
if($pkg.Count -gt 0 -and $pkgGuid.Length -eq 36) {
	foreach($server in $pkg) {
		$test = Invoke-Command -ComputerName $server -Credential $creds -ScriptBlock{ (Get-ItemProperty -Path "HKLM:\SOFTWARE\PostBuild" -Name verGUID -ErrorAction 'SilentlyContinue').verGUID }
		if($test -ne $pkgGuid) {
			Write-Host ("Server {0} does not appear to be configured, adding to watchlist." -f $server)
			$watch[$server] = $pkgGuid
		}else{
			Write-Host ("Server {0} appears to be configured. Adding to completed list." -f $server)
			$complete[$server] = $true
		}
	}
}else{
	Write-Host "No Pkg server nodes found or no verGUID detected in Pkg config. Skipping."
}

if($qa.Count -gt 0 -and $qaGuid.Length -eq 36) {
	foreach($server in $qa) {
		$test = Invoke-Command -ComputerName $server -Credential $creds -ScriptBlock{ (Get-ItemProperty -Path "HKLM:\SOFTWARE\PostBuild" -Name verGUID -ErrorAction 'SilentlyContinue').verGUID }
		if($test -ne $qaGuid) {
			Write-Host ("Server {0} does not appear to be configured, adding to watchlist." -f $server)
			$watch[$server] = $qaGuid
		}else{
			Write-Host ("Server {0} appears to be configured. Adding to completed list." -f $server)
			$complete[$server] = $true
		}
	}
}else{
	Write-Host "No QA server nodes found or no verGUID detected in QA config. Skipping."
}

if($prod.Count -gt 0 -and $prodGuid.Length -eq 36) {
	foreach($server in $prod) {
		$test = Invoke-Command -ComputerName $server -Credential $creds -ScriptBlock{ (Get-ItemProperty -Path "HKLM:\SOFTWARE\PostBuild" -Name verGUID -ErrorAction 'SilentlyContinue').verGUID }
		if($test -ne $prodGuid) {
			Write-Host ("Server {0} does not appear to be configured, adding to watchlist." -f $server)
			$watch[$server] = $prodGuid
		}else{
			Write-Host ("Server {0} appears to be configured. Adding to completed list." -f $server)
			$complete[$server] = $true
		}
	}
}else{
	Write-Host "No Production server nodes found or no verGUID detected in Production config. Skipping."
}

# Pause for meatbag digestion.
Start-Sleep -Seconds 10

# Monitor incomplete servers until all servers return matching verGUID's.
if($watch.Count -gt 0){ $monitor = $true }else{ $monitor = $false }
while($monitor -ne $false) {
	$monitor = $false
	$cleaner = @()
	foreach($server in $watch.Keys) {
		$test = Invoke-Command -ComputerName $server -Credential $creds -ScriptBlock{ (Get-ItemProperty -Path "HKLM:\SOFTWARE\PostBuild" -Name verGUID -ErrorAction 'SilentlyContinue').verGUID }
		if($test -eq $watch[$server]) {
			$complete[$server] = $true
			$cleaner += $server
		}else{
			$monitor = $true
		}
	}
	
	foreach($item in $cleaner){ $watch.Remove($item) }
	
	Clear-Host
	Write-Host "mConfigured Servers:`r`n"$complete.Keys
	Write-Host "`r`n`r`nmIncomplete Servers:`r`n"$watch.Keys
	if($monitor -eq $true){ Start-Sleep -Seconds 10 }
}

Clear-Host
Write-Host "Configured Servers:`r`n"$complete.Keys
Write-Host "`r`n`r`nIncomplete Servers:`r`n"$watch.Keys

End of the day is this a perfect solution? No. Bear in mind I just slapped this together to fill a void, things could be objectified, cleaned up, probably streamlined, but honestly a powershell script is not a good dashboard. I would also rather the servers themselves flag their progress in a centralized location rather than being pinged by a script.

But that is really something best implemented by the PowerShell devs, as anything 3rd party would, IMO, be rather ugly. So if all we have right now is ugly, I'll take ugly and fast.

As always, use at your own risk, I cannot imagine how you could eat a server with this script but don't go using it as some definitive health-metric. Just use it as a way to get a rough idea of the health of your latest configuration push.