Can Powershell Run Commands in Parallel?

I have a powershell script to do some batch processing on a bunch of images and I'd like to do some parallel processing. Powershell seems to have some background processing options such as start-job, wait-job, etc, but the only good resource I found for doing parallel work was writing the text of a script out and running those (PowerShell Multithreading)

Ideally, I'd like something akin to parallel foreach in .net 4.

Something pretty seemless like:

foreach-parallel -threads 4 ($file in (Get-ChildItem $dir))
{ .. Do Work
}

Maybe I'd be better off just dropping down to c#...

2

10 Answers

You can execute parallel jobs in Powershell 2 using Background Jobs. Check out Start-Job and the other job cmdlets.

# Loop through the server list
Get-Content "ServerList.txt" | %{ # Define what each job does $ScriptBlock = { param($pipelinePassIn) Test-Path "\\$pipelinePassIn\c`$\Something" Start-Sleep 60 } # Execute the jobs in parallel Start-Job $ScriptBlock -ArgumentList $_
}
Get-Job
# Wait for it all to complete
While (Get-Job -State "Running")
{ Start-Sleep 10
}
# Getting the information back from the jobs
Get-Job | Receive-Job

6

The answer from Steve Townsend is correct in theory but not in practice as @likwid pointed out. My revised code takes into account the job-context barrier--nothing crosses that barrier by default! The automatic $_ variable can thus be used in the loop but cannot be used directly within the script block because it is inside a separate context created by the job.

To pass variables from the parent context to the child context, use the -ArgumentList parameter on Start-Job to send it and use param inside the script block to receive it.

cls
# Send in two root directory names, one that exists and one that does not.
# Should then get a "True" and a "False" result out the end.
"temp", "foo" | %{ $ScriptBlock = { # accept the loop variable across the job-context barrier param($name) # Show the loop variable has made it through! Write-Host "[processing '$name' inside the job]" # Execute a command Test-Path "\$name" # Just wait for a bit... Start-Sleep 5 } # Show the loop variable here is correct Write-Host "processing $_..." # pass the loop variable across the job-context barrier Start-Job $ScriptBlock -ArgumentList $_
}
# Wait for all to complete
While (Get-Job -State "Running") { Start-Sleep 2 }
# Display output from all jobs
Get-Job | Receive-Job
# Cleanup
Remove-Job *

(I generally like to provide a reference to the PowerShell documentation as supporting evidence but, alas, my search has been fruitless. If you happen to know where context separation is documented, post a comment here to let me know!)

4

There's so many answers to this these days:

1. jobs (or threadjobs in PS 6/7 or the module for PS 5)
2. start-process
3. workflows (PS 5 only)
4. powershell api with another runspace
5. invoke-command with multiple computers, which can all be localhost (have to be admin)
6. multiple session (runspace) tabs in the ISE, or remote powershell ISE tabs
7. Powershell 7 has a foreach-object -parallel as an alternative for #4

Using start-threadjob in powershell 5.1. I wish this worked like I expect, but it doesn't:

# test-netconnection has a miserably long timeout
echo yahoo.com facebook.com | start-threadjob { test-netconnection $input } | receive-job -wait -auto
WARNING: Name resolution of yahoo.com microsoft.com facebook.com failed

It works this way. Not quite as nice and foreach-object -parallel in powershell 7 but it'll do.

echo yahoo.com facebook.com | % { $_ | start-threadjob { test-netconnection $input } } | receive-job -wait -auto | ft -a
ComputerName RemotePort RemoteAddress PingSucceeded PingReplyDetails (RTT) TcpTestS ucceeded
------------ ---------- ------------- ------------- ---------------------- --------
facebook.com 0 31.13.71.36 True 17 ms False
yahoo.com 0 98.137.11.163 True 97 ms False

Here's workflows with literally a foreach -parallel:

workflow work { foreach -parallel ($i in 1..3) { sleep 5 "$i done" }
}
work
3 done
1 done
2 done

Or a workflow with a parallel block:

function sleepfor($time) { sleep $time; "sleepfor $time done"}
workflow work { parallel { sleepfor 3 sleepfor 2 sleepfor 1 } 'hi'
}
work
sleepfor 1 done
sleepfor 2 done
sleepfor 3 done
hi

Here's an api with runspaces example:

$a = [PowerShell]::Create().AddScript{sleep 5;'a done'}
$b = [PowerShell]::Create().AddScript{sleep 5;'b done'}
$c = [PowerShell]::Create().AddScript{sleep 5;'c done'}
$r1,$r2,$r3 = ($a,$b,$c).begininvoke() # run in background
$a.EndInvoke($r1); $b.EndInvoke($r2); $c.EndInvoke($r3) # wait
($a,$b,$c).streams.error # check for errors
($a,$b,$c).dispose() # clean
a done
b done
c done

2

i created an invoke-async which allows you do run multiple script blocks/cmdlets/functions at the same time. this is great for small jobs (subnet scan or wmi query against 100's of machines) because the overhead for creating a runspace vs the startup time of start-job is pretty drastic. It can be used like so.

with scriptblock,

$sb = [scriptblock] {param($system) gwmi win32_operatingsystem -ComputerName $system | select csname,caption}
$servers = Get-Content servers.txt
$rtn = Invoke-Async -Set $server -SetParam system -ScriptBlock $sb

just cmdlet/function

$servers = Get-Content servers.txt
$rtn = Invoke-Async -Set $servers -SetParam computername -Params @{count=1} -Cmdlet Test-Connection -ThreadCount 50

Backgrounds jobs are expensive to setup and are not reusable. PowerShell MVP Oisin Grehan has a good example of PowerShell multi-threading.

(10/25/2010 site is down, but accessible via the Web Archive).

I'e used adapted Oisin script for use in a data loading routine here:

1

To complete previous answers, you can also use Wait-Job to wait for all jobs to complete:

For ($i=1; $i -le 3; $i++) { $ScriptBlock = { Param ( [string] [Parameter(Mandatory=$true)] $increment ) Write-Host $increment } Start-Job $ScriptBlock -ArgumentList $i
}
Get-Job | Wait-Job | Receive-Job

In Powershell 7 you can use ForEach-Object -Parallel

$Message = "Output:"
Get-ChildItem $dir | ForEach-Object -Parallel { "$using:Message $_"
} -ThrottleLimit 4

If you're using latest cross platform powershell (which you should btw) , you can add single & to run parallel scripts. (Use ; to run sequentially)

In my case I needed to run 2 npm scripts in parallel: npm run hotReload & npm run dev

---

You can also setup npm to use powershell for its scripts (by default it uses cmd on windows).

Run from project root folder: npm config set script-shell pwsh --userconfig ./.npmrcand then use single npm script command: npm run start

"start":"npm run hotReload & npm run dev"

This has been answered thoroughly. Just want to post this method i have created based on Powershell-Jobs as a reference.

Jobs are passed on as a list of script-blocks. They can be parameterized. Output of the jobs is color-coded and prefixed with a job-index (just like in a vs-build-process, as this will be used in a build) Can be used to startup multiple servers at a time or running build steps in parallel or so..

function Start-Parallel { param( [ScriptBlock[]] [Parameter(Position = 0)] $ScriptBlock, [Object[]] [Alias("arguments")] $parameters ) $jobs = $ScriptBlock | ForEach-Object { Start-Job -ScriptBlock $_ -ArgumentList $parameters } $colors = "Blue", "Red", "Cyan", "Green", "Magenta" $colorCount = $colors.Length try { while (($jobs | Where-Object { $_.State -ieq "running" } | Measure-Object).Count -gt 0) { $jobs | ForEach-Object { $i = 1 } { $fgColor = $colors[($i - 1) % $colorCount] $out = $_ | Receive-Job $out = $out -split [System.Environment]::NewLine $out | ForEach-Object { Write-Host "$i> "-NoNewline -ForegroundColor $fgColor Write-Host $_ } $i++ } } } finally { Write-Host "Stopping Parallel Jobs ..." -NoNewline $jobs | Stop-Job $jobs | Remove-Job -Force Write-Host " done." }
}

sample output:

2

There is a new built-in solution in PowerShell 7.0 Preview 3.PowerShell ForEach-Object Parallel Feature

So you could do:

Get-ChildItem $dir | ForEach-Object -Parallel {
.. Do Work $_ # this will be your file
}-ThrottleLimit 4