PowerShell code runs considerably slower in Automation Account hybrid worker than it does directly on the hybrid worker server

Question

I have a web scraping and HTTP posting automation written in PowerShell that I developed partially locally and partially on my hybrid worker with the authoring toolkit. During development and testing on the hybrid worker server, it run as expected in a few hours but when the code runs on the hybrid worker from the automation account, the code is considerably slower.

There are 2 primary sections that are slowing it down in the automation account and taking around 50 seconds each, and these have to run tens of thousands of times throughout the automation:

$unparsedJSON = ($detailsSession.Scripts)[-1].outerHTML
There are only 31 script blocks in the $detailsSession and the script block is 806 character. This takes 0.77 millisecond on the server.

$linkedRecord = ($linkedSession.allelements.innertext | Where-Object {$_ -like 'Record -*'})

There are 419 elements in the $linkedSession. This line takes 13 milliseconds directly on the server.

Answer 1

1. To optimize the JSON parsing process for $detailsSession.Scripts, leveraging the ConvertFrom-Json cmdlet is recommended over direct property access. Given that there are only 31 script blocks, each averaging around 806 characters, the parsing operation should not be excessively slow. By employing ConvertFrom-Json, the parsing process can be streamlined and potentially enhanced for improved efficiency.

2. When dealing with large datasets like the 419 elements in $linkedSession, filtering elements using Where-Object can introduce performance bottlenecks. To address this, consider leveraging XPath for optimization, particularly if the structure of the elements allows for it. XPath offers a more efficient approach to filtering based on specific patterns.

3. Optimize looping by utilizing foreach over ForEach-Object for improved speed. Additionally, leverage PowerShell's parallel processing capabilities, such as jobs or runspaces, to execute independent tasks concurrently, thereby boosting overall performance.

Powershell code:-

$detailsSession = @"
{
    "Scripts": [
        {"outerHTML": "<script>Script 1</script>"},
        {"outerHTML": "<script>Script 2</script>"},
        {"outerHTML": "<script>Script 3</script>"}
    ]
}
"@

$linkedSession = @"
<elements>
    <element>Record - 1</element>
    <element>Record - 2</element>
    <element>Record - 3</element>
    <element>Other Element</element>
</elements>
"@

# Optimized JSON parsing using ConvertFrom-Json
$unparsedJSON = $detailsSession | ConvertFrom-Json | Select-Object -ExpandProperty Scripts | Select-Object -Last 1 | Select-Object -ExpandProperty outerHTML
Write-Host "Unparsed JSON: $unparsedJSON"

# Optimized element filtering using XPath
$linkedRecord = $linkedSession | Select-Xml "//text()[contains(., 'Record -')]" | ForEach-Object { $_.Node.InnerText }
Write-Host "Linked Record: $($linkedRecord -join ', ')"

Output:-

You can also start the runbook by using the Powershell command here to get more insights on the Run.

Start-AzAutomationRunbook -AutomationAccountName "siliconautom" -Name "ps2" -ResourceGroupName "siddheshrg"