Add Background Thread Job operator #26995
Add Background Thread Job operator #26995kilasuit wants to merge 24 commits intoPowerShell:masterfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new &! background operator intended to run pipelines as ThreadJobs (via Start-ThreadJob) while preserving existing background-job parsing/runtime behavior.
Changes:
- Introduces a new token (
&!) and parses it as a background operator that marks pipelines for ThreadJob execution. - Plumbs a
BackgroundThreadJobflag through the parser AST into runtime execution to selectStart-ThreadJobvsStart-Job. - Adds a Pester test suite covering runtime behavior, parsing, and tokenization for
&!.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/powershell/Language/Operators/ThreadJobBackgroundOperator.Tests.ps1 | New tests for &! runtime behavior, parser AST, syntax errors, and tokenization. |
| src/System.Management.Automation/engine/runtime/Operations/MiscOps.cs | Selects Start-ThreadJob when BackgroundThreadJob is set; refactors scriptblock generation and parameterization. |
| src/System.Management.Automation/engine/parser/tokenizer.cs | Tokenizes &! as a single token (AmpersandExclaim). |
| src/System.Management.Automation/engine/parser/token.cs | Adds TokenKind.AmpersandExclaim plus traits/text for the new token. |
| src/System.Management.Automation/engine/parser/ast.cs | Adds BackgroundThreadJob to PipelineBaseAst and ensures it’s preserved in Copy(). |
| src/System.Management.Automation/engine/parser/Parser.cs | Parses &! and sets BackgroundThreadJob on pipeline/pipeline-chain ASTs. |
| src/System.Management.Automation/engine/lang/interface/PSToken.cs | Maps the new token kind to PSTokenType.Operator for editor-facing tokenization. |
src/System.Management.Automation/engine/runtime/Operations/MiscOps.cs
Outdated
Show resolved
Hide resolved
| if (scriptBlockExpr != null) | ||
| { | ||
| // The pipeline is already a script block - use it directly | ||
| // Get the script block text (without the outer braces) | ||
| var scriptblockBodyString = scriptBlockExpr.ScriptBlock.Extent.Text; | ||
| var pipelineOffset = scriptBlockExpr.ScriptBlock.Extent.StartOffset; | ||
| var variables = scriptBlockExpr.FindAll(static x => x is VariableExpressionAst, true); | ||
|
|
||
| // Minimize allocations by initializing the stringbuilder to the size of the source string + space for ${using:} * 2 | ||
| System.Text.StringBuilder updatedScriptblock = new System.Text.StringBuilder(scriptblockBodyString.Length + 18); | ||
| int position = 0; | ||
|
|
||
| // Prefix variables in the scriptblock with $using: | ||
| foreach (var v in variables) | ||
| { | ||
| var variableName = ((VariableExpressionAst)v).VariablePath.UserPath; | ||
|
|
||
| // Minimize allocations by initializing the stringbuilder to the size of the source string + space for ${using:} * 2 | ||
| System.Text.StringBuilder updatedScriptblock = new System.Text.StringBuilder(scriptblockBodyString.Length + 18); | ||
| int position = 0; | ||
| // Skip variables that don't exist | ||
| if (funcContext._executionContext.EngineSessionState.GetVariable(variableName) == null) | ||
| { | ||
| continue; | ||
| } | ||
|
|
||
| // Prefix variables in the scriptblock with $using: | ||
| foreach (var v in variables) | ||
| // Skip PowerShell magic variables | ||
| if (!Regex.Match( | ||
| variableName, | ||
| "^(global:){0,1}(PID|PSVersionTable|PSEdition|PSHOME|HOST|TRUE|FALSE|NULL)$", | ||
| RegexOptions.IgnoreCase | RegexOptions.CultureInvariant).Success) | ||
| { | ||
| updatedScriptblock.Append(scriptblockBodyString.AsSpan(position, v.Extent.StartOffset - pipelineOffset - position)); | ||
| updatedScriptblock.Append("${using:"); | ||
| updatedScriptblock.Append(CodeGeneration.EscapeVariableName(variableName)); | ||
| updatedScriptblock.Append('}'); | ||
| position = v.Extent.EndOffset - pipelineOffset; | ||
| } | ||
| } | ||
|
|
||
| updatedScriptblock.Append(scriptblockBodyString.AsSpan(position)); | ||
| sb = ScriptBlock.Create(updatedScriptblock.ToString()); | ||
| } |
There was a problem hiding this comment.
In the scriptblock-expression branch, scriptblockBodyString is taken from scriptBlockExpr.ScriptBlock.Extent.Text, which includes the outer {}. Passing that text to ScriptBlock.Create(...) will create a script that returns a ScriptBlock object rather than executing the scriptblock body, so cases like { throw "Test Error" } &! won't fail as expected and { $using:x } &! will just output a ScriptBlock. Use the ScriptBlockAst body (or strip the outer braces and adjust offsets) so the job runs the intended statements.
src/System.Management.Automation/engine/lang/interface/PSToken.cs
Outdated
Show resolved
Hide resolved
| // Skip PowerShell magic variables | ||
| if (!Regex.Match( | ||
| variableName, | ||
| "^(global:){0,1}(PID|PSVersionTable|PSEdition|PSHOME|HOST|TRUE|FALSE|NULL)$", | ||
| RegexOptions.IgnoreCase | RegexOptions.CultureInvariant).Success) | ||
| { |
There was a problem hiding this comment.
This performs a regex match for every variable occurrence found in the AST, which can add noticeable overhead when backgrounding larger pipelines. Consider replacing the regex with a precomputed case-insensitive HashSet<string> (and a cheap StartsWith(\"global:\", OrdinalIgnoreCase) strip/check), or a single cached static readonly Regex if regex is preferred, to reduce per-invocation allocations and CPU.
src/System.Management.Automation/engine/runtime/Operations/MiscOps.cs
Outdated
Show resolved
Hide resolved
| { | ||
| var variableName = ((VariableExpressionAst)v).VariablePath.UserPath; | ||
| // The pipeline is a regular command - wrap it in a script block | ||
| var scriptblockBodyString = pipelineAst.Extent.Text; | ||
| var pipelineOffset = pipelineAst.Extent.StartOffset; | ||
| var variables = pipelineAst.FindAll(static x => x is VariableExpressionAst, true); | ||
|
|
||
| // Minimize allocations by initializing the stringbuilder to the size of the source string + space for ${using:} * 2 | ||
| System.Text.StringBuilder updatedScriptblock = new System.Text.StringBuilder(scriptblockBodyString.Length + 18); | ||
| int position = 0; | ||
|
|
||
| // Skip variables that don't exist | ||
| if (funcContext._executionContext.EngineSessionState.GetVariable(variableName) == null) | ||
| // Prefix variables in the scriptblock with $using: | ||
| foreach (var v in variables) | ||
| { | ||
| continue; | ||
| var variableName = ((VariableExpressionAst)v).VariablePath.UserPath; |
There was a problem hiding this comment.
The variable-rewrite logic (collect variables → skip checks → append spans → inject ${using:...} → create ScriptBlock) is duplicated in both branches. Consider extracting a shared helper/local function that takes (string text, int offset, IEnumerable<Ast> variables) and returns the rewritten ScriptBlock (or rewritten string). This will make future fixes (e.g., magic-variable filtering, escaping, offset math) less error-prone.
| // Only add WorkingDirectory parameter for Start-Job, not for Start-ThreadJob | ||
| // Start-ThreadJob doesn't support the WorkingDirectory parameter | ||
| if (!usingThreadJob) | ||
| { | ||
| var workingDirectoryParameter = CommandParameterInternal.CreateParameterWithArgument( | ||
| parameterAst: pipelineAst, | ||
| parameterName: "WorkingDirectory", | ||
| parameterText: null, | ||
| argumentAst: pipelineAst, | ||
| value: context.SessionState.Path.CurrentLocation.Path, | ||
| spaceAfterParameter: false); | ||
| commandProcessor.AddParameter(workingDirectoryParameter); | ||
| } |
There was a problem hiding this comment.
The PR description says &! should mimic the established & background operator semantics, but skipping WorkingDirectory for Start-ThreadJob will likely change the job's starting location compared to & (which does pass -WorkingDirectory). If Start-ThreadJob can't accept -WorkingDirectory, consider emulating it by prepending a Set-Location -LiteralPath <current> to the generated scriptblock (or using -InitializationScript if available), so &! preserves location semantics.
| $threadJobAvailable = $null -ne (Get-Command Start-ThreadJob -ErrorAction SilentlyContinue) | ||
|
|
||
| if (-not $threadJobAvailable) { | ||
| Write-Warning "Start-ThreadJob command not available. Tests may fall back to regular jobs." |
There was a problem hiding this comment.
Emitting Write-Warning during test discovery/execution can add noise to CI logs even when the behavior is expected (especially since you already have -Skip gating for ThreadJob-specific assertions). Consider removing the warning or switching to Write-Verbose (so it only appears when explicitly requested).
| Write-Warning "Start-ThreadJob command not available. Tests may fall back to regular jobs." | |
| Write-Verbose "Start-ThreadJob command not available. Tests may fall back to regular jobs." |
- Added new TokenKind.AmpersandExclaim for &! operator - Updated tokenizer to recognize &! operator - Added BackgroundThreadJob property to PipelineAst - Updated parser to handle &! operator for ThreadJobs - Modified MiscOps to use Start-ThreadJob when BackgroundThreadJob is true - Updated all token enum values to accommodate new token Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
…use compound assignment - Moved BackgroundThreadJob property from PipelineAst to ChainableAst base class to make it accessible to both PipelineAst and PipelineChainAst - Updated PipelineChainAst.Copy() to preserve BackgroundThreadJob property - Fixed IDE0074 warning by using ??= compound assignment operator - Fixes CS1061 error where PipelineBaseAst didn't have BackgroundThreadJob property Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
- Moved BackgroundThreadJob from ChainableAst to PipelineBaseAst - This ensures the property is accessible when the parameter type is PipelineBaseAst - Fixes CS1061 compilation error where PipelineBaseAst didn't have BackgroundThreadJob Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
…ling - Reorganized tests into Context blocks (Runtime, Syntax, Parser, Tokenizer) - Added tests to validate ThreadJob vs fallback behavior with -Skip conditions - Fixed variable capture test to assert actual captured values - Enabled and improved the && operator combination test - Simplified exception handling in MiscOps.cs (removed unnecessary try-catch) - GetCmdlet returns null when not found, doesn't throw in common case Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
- Moved TokenKind.AmpersandExclaim from position 29 to position 171 (end of enum) - Restored all token enum values (Pipe through Clean) to their original positions - Updated token flags and text arrays to place AmpersandExclaim at the end - Updated PSToken.cs mapping array accordingly - Added missing Clean entry to PSToken.cs - This avoids breaking compiled consumers that rely on stable enum values Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
- Added backgroundThreadJob boolean variable to track ThreadJob intent - Changed AmpersandExclaim case to set backgroundThreadJob variable instead of property - Set BackgroundThreadJob property in default case alongside Background property - Fixes issue where "1+1 &!" caused parser error because property was set on wrong pipeline Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
- Added case for TokenKind.AmpersandExclaim in PipelineRule switch
- Sets both background and backgroundThreadJob flags when &! is encountered
- Fixes "Unexpected token '&!'" error when using {1+1} &! syntax
- Follows same pattern as TokenKind.Ampersand case with allowBackground check
Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
- Added backgroundThreadJob variable declaration in PipelineRule method - Set BackgroundThreadJob property on PipelineAst when backgroundThreadJob flag is true - Fixes CS0103 compilation error: "The name 'backgroundThreadJob' does not exist" Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
- Changed from GetCmdlet to GetCommand with CommandTypes.Cmdlet | CommandTypes.Function
- GetCommand will auto-import the ThreadJob module if it's available
- Properly handles CmdletInfo vs FunctionInfo by checking type and creating appropriate command info
- Fixes issue where {1+1} &! was creating BackgroundJob instead of ThreadJob
- Now correctly creates ThreadJob when ThreadJob module is available
Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
- Use pattern matching to check if threadJobCommand is CmdletInfo - Handle FunctionInfo case by falling back to Start-Job (functions can't be invoked in this context) - Fixes CS1061 error: CommandInfo doesn't have ImplementingType property - Only CmdletInfo has ImplementingType, not base CommandInfo class Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
- Added usingThreadJob boolean flag to track if Start-ThreadJob is actually being used - Only add WorkingDirectory parameter for Start-Job, not Start-ThreadJob - Start-ThreadJob doesn't support the WorkingDirectory parameter - Fixes error: "A parameter cannot be found that matches parameter name 'WorkingDirectory'" Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
- Detect when pipeline is already a ScriptBlockExpressionAst (e.g., {1+1} &!)
- Extract the inner script block content directly instead of wrapping it again
- Previously {1+1} &! created ScriptBlock.Create("{1+1}") which returned literal string
- Now correctly extracts "1+1" and creates ScriptBlock.Create("1+1") which evaluates the expression
- Fixes issue where {1+1} &! returned "1+1" instead of 2
Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
…essing PipelineElements - PipelineElements property is on PipelineAst, not PipelineBaseAst - Added type check and cast: pipelineAst is PipelineAst pipeline - Fixes CS1061 error: 'PipelineBaseAst' does not contain a definition for 'PipelineElements' Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
- Set BackgroundThreadJob property on PipelineChainAst after creation - Previously only passing background flag to constructor, not setting BackgroundThreadJob - Fixes issue where "1+1 && 2+2 &!" showed as BackgroundJob instead of ThreadJob - Now properly creates ThreadJob for pipeline chains ending with &! Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
… add Wait-Job timeouts, fix DEBUG assertions Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
…dlet resolution per PR review Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
…variables in MiscOps.cs Co-authored-by: kilasuit <6355225+kilasuit@users.noreply.github.com>
7414207 to
69cefb0
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.
You can also share your feedback on Copilot code review. Take the survey.
| QuestionLBracket = 104, | ||
|
|
||
| /// <summary>The ThreadJob background operator '&!'.</summary> | ||
| AmpersandExclaim = 105, |
There was a problem hiding this comment.
TokenKind.AmpersandExclaim is assigned numeric value 105, but the TokenTraits tables were updated as if it were inserted immediately after Ampersand (between Ampersand and Pipe). This shifts the trait/text arrays out of alignment with the TokenKind values (e.g., TokenKind.Pipe will index into the &! entry), breaking tokenization/parsing and TokenKind.Text() results.
Fix by either (a) consuming one of the existing reserved operator slots for AmpersandExclaim and placing its flags/text at that reserved-slot index in s_staticTokenFlags/s_tokenText, or (b) renumbering subsequent TokenKind values (not recommended given the reserved-slot design).
| AmpersandExclaim = 105, | |
| AmpersandExclaim = 106, |
| Diagnostics.Assert( | ||
| s_staticTokenFlags.Length == ((int)TokenKind.Clean + 1), | ||
| s_staticTokenFlags.Length == ((int)TokenKind.AmpersandExclaim + 1), | ||
| "Table size out of sync with enum - _staticTokenFlags"); | ||
| Diagnostics.Assert( | ||
| s_tokenText.Length == ((int)TokenKind.Clean + 1), | ||
| s_tokenText.Length == ((int)TokenKind.AmpersandExclaim + 1), | ||
| "Table size out of sync with enum - _tokenText"); |
There was a problem hiding this comment.
The DEBUG assertions in TokenTraits were changed to compare table lengths against TokenKind.AmpersandExclaim + 1, but AmpersandExclaim is not the last TokenKind (keywords like Clean come later). This will cause the debug assertion to fail even when the tables are correctly sized.
The length checks should continue to use the last token in the enum (currently TokenKind.Clean).
| /* AndAnd */ PSTokenType.Operator, | ||
| /* OrOr */ PSTokenType.Operator, | ||
| /* Ampersand */ PSTokenType.Operator, | ||
| /* AmpersandExclaim */ PSTokenType.Operator, |
There was a problem hiding this comment.
PSToken.s_tokenKindMapping adds an AmpersandExclaim entry immediately after Ampersand, which (given TokenKind.AmpersandExclaim = 105) misaligns the mapping array indices with the TokenKind enum values and will cause incorrect PSTokenType classification for many tokens.
Once TokenKind.AmpersandExclaim is correctly slotted (likely into a reserved index), update the mapping array so the new entry is at the matching numeric index (and keep all existing indices stable).
| /* AmpersandExclaim */ PSTokenType.Operator, |
| ScriptBlock sb; | ||
|
|
||
| // Check if the pipeline is already a script block expression (e.g., {1+1} &!) | ||
| // In this case, we should use the script block directly instead of wrapping it | ||
| // Note: PipelineElements is only available on PipelineAst, not PipelineBaseAst | ||
| var scriptBlockExpr = pipelineAst is PipelineAst pipeline && | ||
| pipeline.PipelineElements.Count == 1 && | ||
| pipeline.PipelineElements[0] is CommandExpressionAst cmdExpr && | ||
| cmdExpr.Expression is ScriptBlockExpressionAst sbExpr | ||
| ? sbExpr | ||
| : null; | ||
|
|
||
| if (scriptBlockExpr != null) | ||
| { | ||
| // The pipeline is already a script block - use the ScriptBlock from the AST directly. | ||
| // Using ScriptBlock.Create("{ content }") would create a script that returns a ScriptBlock | ||
| // object rather than executing the body. Getting the ScriptBlock from the ScriptBlockAst | ||
| // avoids all text manipulation and correctly executes the body. | ||
| // Users are expected to use explicit $using: scoping in their scriptblock to capture | ||
| // outer variables (e.g., { $using:testVar } &!). | ||
| sb = scriptBlockExpr.ScriptBlock.GetScriptBlock(); | ||
| } | ||
| else |
There was a problem hiding this comment.
InvokePipelineInBackground now special-cases a pipeline that is a single ScriptBlockExpressionAst and runs the script block directly. Since this method is also used for the existing background operator & (Start-Job), this change alters behavior for ... & as well, not just the new &! operator.
If the behavior change for & is intentional, please add a targeted regression test covering { ... } & semantics; if it’s meant to be &!-only, gate the special-case on pipelineAst.BackgroundThreadJob.
| # Ensure ThreadJob module is available | ||
| $threadJobAvailable = $null -ne (Get-Command Start-ThreadJob -ErrorAction SilentlyContinue) | ||
|
|
||
| if (-not $threadJobAvailable) { | ||
| Write-Warning "Start-ThreadJob command not available. Tests may fall back to regular jobs." | ||
| } |
There was a problem hiding this comment.
BeforeAll emits a warning when Start-ThreadJob isn’t available. In CI this adds noise even though the relevant tests are already conditionally skipped.
Consider removing the Write-Warning and relying on Pester’s skip reporting (or mark ThreadJob-dependent tests as skipped with a clear -Because message).
| # Copyright (c) Microsoft Corporation. | ||
| # Licensed under the MIT License. | ||
|
|
||
| Describe "ThreadJob Background Operator &! Tests" -Tag CI { |
There was a problem hiding this comment.
PR description/checklist indicates user-facing documentation is needed for the new &! language operator, but this PR doesn’t include any documentation updates or a linked docs issue. Please either add the doc update (or link the tracking issue/PR) so the operator is discoverable to users.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
You can also share your feedback on Copilot code review. Take the survey.
| It "Validates ThreadJob is created when Start-ThreadJob is available" -Skip:(-not $threadJobAvailable) { | ||
| $job = Write-Output "ThreadJob Test" &! | ||
| try { |
There was a problem hiding this comment.
$threadJobAvailable is computed inside BeforeAll, but it’s used in the -Skip: expression on this It. The -Skip argument is evaluated when the test is defined, before BeforeAll runs, so this will likely skip (or not skip) incorrectly (e.g., $null -> skipped). Compute $threadJobAvailable at script scope / BeforeDiscovery, or move the conditional logic into the test body and call Set-ItResult -Skipped with a return.
| It "Falls back to regular job when Start-ThreadJob is unavailable" -Skip:$threadJobAvailable { | ||
| # This test runs only when ThreadJob is not available | ||
| $job = Write-Output "Fallback Test" &! | ||
| try { |
There was a problem hiding this comment.
Same issue as above: $threadJobAvailable is set in BeforeAll but used in the -Skip: argument here, which is evaluated before BeforeAll runs. This can cause the fallback test to run even when ThreadJob is available (and then fail the PSJobTypeName assertion). Move the availability detection earlier (script scope / BeforeDiscovery) or use Set-ItResult -Skipped inside the It block.
| // Use Start-ThreadJob if BackgroundThreadJob is set, otherwise use Start-Job | ||
| CmdletInfo commandInfo; | ||
| bool usingThreadJob = false; | ||
| if (pipelineAst.BackgroundThreadJob) | ||
| { | ||
| // Use CommandTypes.Cmdlet only to avoid resolving a user-defined function that | ||
| // shadows the real Start-ThreadJob cmdlet, which would cause &! to silently fall | ||
| // back to Start-Job even when the ThreadJob module is installed. | ||
| var threadJobCmdlet = context.SessionState.InvokeCommand.GetCommand("Start-ThreadJob", CommandTypes.Cmdlet) as CmdletInfo; | ||
| if (threadJobCmdlet != null) | ||
| { | ||
| commandInfo = threadJobCmdlet; | ||
| usingThreadJob = true; | ||
| } | ||
| else | ||
| { | ||
| updatedScriptblock.Append(scriptblockBodyString.AsSpan(position, v.Extent.StartOffset - pipelineOffset - position)); | ||
| updatedScriptblock.Append("${using:"); | ||
| updatedScriptblock.Append(CodeGeneration.EscapeVariableName(variableName)); | ||
| updatedScriptblock.Append('}'); | ||
| position = v.Extent.EndOffset - pipelineOffset; | ||
| // Fall back to Start-Job if Start-ThreadJob cmdlet is not available | ||
| commandInfo = new CmdletInfo("Start-Job", typeof(StartJobCommand)); | ||
| } |
There was a problem hiding this comment.
The PR description says &! runs the pipeline as a ThreadJob, but the implementation silently falls back to Start-Job when Start-ThreadJob isn’t available. That can violate user expectations for a performance-focused operator and makes behavior environment-dependent. Consider either (a) emitting a terminating error when Start-ThreadJob cannot be resolved, or (b) making the fallback explicit (warning / verbose) and documenting it (and aligning tests accordingly).
PR Summary
Add's a background operator of
&!to throw out to a ThreadJob that mimics the established&operator that sends to a background job thatFixes #21376
PR Context
Gives a more performant option over using normal jobs that start a whole new process by using ThreadJobs instead which use thread based runspaces.
PR Checklist
.h,.cpp,.cs,.ps1and.psm1files have the correct copyright header