How to download a webstring from multiple web-servers in parallel via Powershell?

Question

I need to download some webcontent from many servers in parallel as part of a scheduled job, but I cannot find a correct way to run the download in parallel/async. How can this be done?

Without any parallelism I can do it this way, but it is very slow:

$web = [System.Net.WebClient]::new()
[Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

# $srvList is a list of servers of viariable length
$allData = ""
foreach ($srv in $srvList) {
    $url = "https:\\$srv\MyWebPage"
    $data = $web.DownloadString($url)
    $allData += $data
}

But how to do this in parallel via "$web.DownloadStringAsync"? I found this snippet, but I dont see how to get the result of each call and how to concatenate it:

$job = Register-ObjectEvent -InputObject $web -EventName DownloadStringCompleted -Action {
    Write-Host 'Download completed'
    write-host $EventArgs.Result
}
$web.DownloadString($url)

Does someone know, how to get this solved in a short & smart way?

Answer 1

Finally I found a simple solution via events. Here is my code-snippet:

cls
Remove-Variable * -ea 0

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = $null
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

$srvList = @('srv1','srv2','srv3')
$webObjList = [System.Collections.ArrayList]::new()
$eventList  = [System.Collections.ArrayList]::new()
$resultList = [System.Collections.ArrayList]::new()

$i=0
foreach ($srv in $srvList) {
    $null  = $webObjList.add([System.Net.WebClient]::new())
    $null  = $eventList.add($(Register-ObjectEvent -InputObject $webObjList[$i] -EventName DownloadStringCompleted -SourceIdentifier $srv))
    $null  = $resultList.add($webObjList[$i].DownloadStringTaskAsync("https://$srv/MyWebPage"))
    $i++
}
do {sleep -Milliseconds 10} until ($resultList.IsCompleted -notcontains $false)
foreach ($srv in $srvList) {Unregister-Event $srv}

# show all Results:
$resultList.result

Answer 2

The best and fastest way is using runspaces:

Add-Type -AssemblyName System.Collections

$GH = [hashtable]::Synchronized(@{})

[System.Collections.Generic.List[PSObject]]$GH.results = [System.Collections.Generic.List[string]]::new()
[System.Collections.Generic.List[string]]$GH.servers = @('server1','server2');
[System.Collections.Generic.List[string]]$GH.functions = @('Download-Content');

[System.Collections.Generic.List[PSObject]]$jobs = @()


#-----------------------------------------------------------------
function Download-Content {
#-----------------------------------------------------------------

   # a function which runs parallel

   param(
    [string]$server
   )

   $web = [System.Net.WebClient]::new()
   [Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
   [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

   $url = "https:\\$server\MyWebPage"
   $data = $web.DownloadString($url)
   $GH.results.Add( $data )
}

#-----------------------------------------------------------------
function Create-InitialSessionState {
#-----------------------------------------------------------------

    param(
        [System.Collections.Generic.List[string]]$functionNameList
    )

    # Setting up an initial session state object
    $initialSessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()

    foreach( $functionName in $functionNameList ) {

        # Getting the function definition for the functions to add
        $functionDefinition = Get-Content function:\$functionName
        $functionEntry = New-Object System.Management.Automation.Runspaces.SessionStateFunctionEntry -ArgumentList $functionName, $functionDefinition

        # And add it to the iss object
        [void]$initialSessionState.Commands.Add($functionEntry)
    }

    return $initialSessionState
}

#-----------------------------------------------------------------
function Create-RunspacePool {
#-----------------------------------------------------------------

    param(
        [InitialSessionState]$initialSessionState
    )

    $runspacePool = [RunspaceFactory]::CreateRunspacePool(1, ([int]$env:NUMBER_OF_PROCESSORS + 1), $initialSessionState, $Host)
    $runspacePool.ApartmentState = 'MTA'
    $runspacePool.ThreadOptions  = "ReuseThread"
    [void]$runspacePool.Open()

    return $runspacePool
}


#-----------------------------------------------------------------
function Release-Runspaces {
#-----------------------------------------------------------------

    $runspaces = Get-Runspace | Where { $_.Id -gt 1 }

    foreach( $runspace in $runspaces ) {
        try{
            [void]$runspace.Close()
            [void]$runspace.Dispose()
            }
        catch {
        }
    }
}


$initialSessionState = Create-InitialSessionState -functionNameList $GH.functions
$runspacePool        = Create-RunspacePool -initialSessionState $initialSessionState


foreach ($server in $GH.servers)
{
    Write-Host $server

    $job = [System.Management.Automation.PowerShell]::Create($initialSessionState)
    $job.RunspacePool = $runspacePool

    $scriptBlock = { param ( [hashtable]$GH, [string]$server ); Download-Content -server $server }

    [void]$job.AddScript( $scriptBlock ).AddArgument( $GH ).AddArgument( $server )
    $jobs += New-Object PSObject -Property @{
                                    RunNum = $jobCounter++
                                    JobObj = $job
                                    Result = $job.BeginInvoke() }

    do {
        Sleep -Seconds 1 
    } while( $runspacePool.GetAvailableRunspaces() -lt 1 )

}

Do {
    Sleep -Seconds 1
} While( $jobs.Result.IsCompleted -contains $false)


$GH.results

Release-Runspaces | Out-Null

[void]$runspacePool.Close()
[void]$runspacePool.Dispose()