CLI: Stdin input does not implicitly become pipeline input

Currently, stdin input streamed to pwsh from the outside:

does not automatically become pipeline input;
- accessing stdin input currently requires constructing a separate pipeline in PowerShell code using $Input
only automatically becomes input to interactive prompts, such as Read-Host calls or the automatic prompts displayed when mandatory arguments are missing.

This has the following implications:

You cannot directly pipe data to PowerShell commands from the outside:

Contrast the following two commands:

# Bash:
# stdin input is "relayed" to the 'wc' utility
$ printf 'one\ntwo' | bash -c 'wc'
2

# PowerShell:
PS> printf 'one\ntwo' | pwsh -noprofile -c 'Measure-Object -Word | % Words'
0 # !! Measure-Object did NOT receive stdin input.

# Explicit reference to $Input in an internal pipeline is required:
PS> printf 'one\ntwo\n' | pwsh -noprofile -c '$Input | Measure-Object -Word | % Words'
2

printf 'one\ntwo\n' | pwsh -noprofile -c 'Measure-Object -Word'

If you're calling an advanced script that binds pipeline input, it cannot be invoked with just -File, because it won't receive pipeline input:

# Create an advanced script that echoes each pipeline input enclosed in [...]
'[CmdletBinding()] param([Parameter(ValueFromPipeline)] $var) process { "[$var]" }' > t.ps1

# Call from either Bash or PowerShell (same behavior).
PS> echo hi | pwsh -noprofile -file t.ps1
[]  # !! t.ps1 did NOT receive stdin input

# The workaround is again non-obvious and cumbersome:
PS> echo hi | pwsh -noprofile -command '$Input | ./t.ps1'
[hi]  # OK

Similarly, if you author a pipeline-binding advanced script as shebang-line-based script on Unix, it cannot receive stdin input directly:

# Create a shebang-line-based script...
@'
#!/usr/bin/env pwsh
[CmdletBinding()] param([Parameter(ValueFromPipeline)] $var) process { "[$var]" }
'@ > foo

# ... and make it executable.
chmod +x foo

# Call from either Bash or PowerShell (same behavior).
PS> 1..2 | ./foo
[]  # !! Pipeline input wasn't bound, not even with pipeline input from PowerShell.

PowerShell's behavior here is at odds with traditional shells and arguably makes the typical use case unnecessarily difficult.

Note: I'm unclear on whether this can be fixed without breaking existing behavior.

The challenge is to know when to provide stdin input via an implied PowerShell pipeline, and when to use stdin input line by line, on demand, to respond to interactive prompts.

E.g., the following currently works - the two distinct Read-Host prompts are "answered" by 1 stdin input line each:

$ 'one\ntwo\n' | pwsh -noprofile -c 'read-host first; get-date;  read-host second'
first: one
one

Monday, April 29, 2019 10:52:04 AM
second: two
two

Environment data

PowerShell Core 6.2.0

I am hit by the same problem, I have to work with the databricks CLI, which has a terrible UX that requires me to pipe stdin to the the native binary and it seems I have to use bash for now as I could not translate the following to PowerShell:

databricks configure --token <<EOF
https://$(databricks_location).azuredatabricks.net
$(databricks-token)
EOF

I've tried piping a string to it

$newline = [System.Environment]::NewLine
"${newline}https://$DatabricksLocationShort.azuredatabricks.net${newline}$DatabricksToken${newline}" | databricks configure --token

with variations of ${newline} or also trying to pipe an array with no luck....

It sounds like you're describing the opposite direction: piping to an external program.

In my experience that usually does work; e.g.:

@'
one
two
three
'@ | bash -c 'cat -n'  # outputs the stdin input lines with line numbers

Ditto:

'one', 'two', 'three'  | bash -c 'cat -n'  # outputs the stdin input lines with line numbers

Is there something special about the databricks utility?

Note that, as usual, PowerShell uses the $OutputEncoding character encoding for sending text to external programs, which in PS Core is now UTF-8.

Thanks for the info. Yes, it is piping to an external binary that expects input via stdin as it prompts.
I decided to use authentication via environment variables on the PowerShell process level instead

Christoph Bergmeister (@bergmeister)

Christoph Bergmeister (@bergmeister) Can you share code how you are able to authenticate Databricks through Environment variables. I am pretty much struck in this case

mvalikar

mvalikar You set the DATABRICKS_HOST and DATABRICKS_TOKEN environment variables as per docs, I suggest to set them only for the current process for security purposes and clean them up at the end
https://docs.databricks.com/dev-tools/cli/index.html

Databricks CLI — Databricks Documentation
Learn how to install and configure your environment to run the Databricks command-line interface.

The same happens when directly calling ps1.

Bash

test.sh has

python -c "print(input())"

We can call it interactively:

$ ./test.sh
sometext  < input here
sometext

When piping to test.sh, the inner python receives from stdin automatically:

$ echo "sometext" | ./test.sh
sometext

PowerShell

test.ps1 have the same content:

python -c "print(input())"

Interactive input works:

> ./test.ps1
sometext  < input here
sometext

But when piping to test.ps1, the inner python can't receive from stdin and waits for user input.

> echo "sometext" | ./test.ps1
<waiting for user input>

If we explicitly pipe $input to python in test.ps1:

$input | python -c "print(input())"

Now piping works:

> echo "sometext" | ./test.ps1
sometext

but interactive input fails:

> ./test.ps1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
EOFError: EOF when reading a line

A workaround I can think of is to check if $input is empty:

If $input is not empty, pipe it to python
If $input is empty, don't pipe it to python and leave python to read stdin from TTY

(We need to save $input to another variable $stdin since $input is an enumerator and can be accessed only once.)

$stdin = "$input"
if ($stdin) {
    $stdin | python -c "print(input())"
}
else {
    python -c "print(input())"
}

This solution requires additional efforts and is cumbersome. It is also fundamentally different from Bash since we are deciding whether to connect stdin to python based on the content of stdin, instead of whether piping is used.

In Bash, the behavior can be expressed as:

If a sh script is directly called, the inner executable's stdin is connected to TTY
If a sh script is called with pipe |, the output of the source is sent to inner executable's stdin

For example, in Bash, even when piping is used with empty content, it leads to failure:

$ python -c "print('', end='')" | ./test.sh
Traceback (most recent call last):
  File "<string>", line 1, in <module>
EOFError: EOF when reading a line

but the above workaround doesn't work as expected for empty pipe and falls back to interactive input:

> python -c "print('', end='')" | ./test.ps1
userinput  < input here
userinput

So questions are:

Can we detect if piping is used from $input?
How can we detect if $input is empty (pipe is not used) or $input is empty string (an empty pipe is used)?
What if stdin contains binary stream?

Can we detect if piping is used from $input?

Use $MyInvocation.ExpectingInput

& { $MyInvocation.ExpectingInput } # -> $false
1 | & { $MyInvocation.ExpectingInput } # -> $true

How can we detect if [...] $input is empty string (an empty pipe is used)?

If it's OK to collect all input up front:

python -c "print('', end='')" | & { $allLines = @($Input); $allLines.Count -eq 0 }

Otherwise:

# -> $true
python -c "print('', end='')" | & { begin { $emptyInput = $true } process { $emptyInput = $false  } end { $emptyInput }  }

Note that without pipeline input, the process block is also called, namely exactly once.

What if stdin contains binary stream?

PowerShell currently has no support for binary data in its pipelines when communicating with external programs - everything is treated as text - both on sending and receiving - see

Don’t parse the pipeline as text when it is directed from an EXE to another EXE or file. Keep the bytes as-is. #1908

Michael Klement (@mklement0)

Michael Klement (@mklement0), thanks a lot for the insights.

This solution you mentioned doesn't seem to work. It returns True even with empty stdin (scenario 2):

# 1. pipeline is not used
> & { $allLines = @($Input); $allLines.Count -eq 0 }
True

# 2. pipeline is used, but empty
> python -c "print('', end='')" | & { $allLines = @($Input); $allLines.Count -eq 0 }
True

# 3. pipeline is used, with some chars in it
> python -c "print('a', end='')" | & { $allLines = @($Input); $allLines.Count -eq 0 }
False

The other solution also has some unexpected behavior:

> & { begin { $emptyInput = $true } process { $emptyInput = $false  } end { $emptyInput }  }
False
> python -c "print('', end='')" | & { begin { $emptyInput = $true } process { $emptyInput = $false  } end { $emptyInput }  }
True
> python -c "print('a', end='')" | & { begin { $emptyInput = $true } process { $emptyInput = $false  } end { $emptyInput }  }
False

The expected result is to return

False if pipeline is not used (scenario 1)
True is pipeline is used (scenario 2, 3)

(Or otherwise - doesn't really matter.)

I answered your questions in isolation, You need to combine your logic with $MyInvocation.ExpectingInput as needed, to distinguish the pipeline from the non-pipeline scenarios.
Apart from that, I see no unexpected behavior. & { $allLines = @($Input); $allLines.Count -eq 0 } is designed to return $true if there is either no input from the pipeline at all or if that input is empty.
& { begin { $emptyInput = $true } process { $emptyInput = $false } end { $emptyInput } } is designed to return $true only if there's pipeline input and it is empty.

I'm running into the same limitation. I'm trying to use PowerShell to write a script to handle the pre-push git hook.

Since I am unable to grab stdin from PowerShell, I am unable to process this data.

pre-push
This hook is called by [git-push[1]](https://git-scm.com/docs/git-push) and can be used to prevent a push from taking place. The hook is called with two parameters which provide the name and location of the destination remote, if a named remote is not being used both values will be the same.

Information about what is to be pushed is provided on the hook’s standard input with lines of the form:

<local ref> SP <local object name> SP <remote ref> SP <remote object name> LF

For instance, if the command git push origin master:foreign were run the hook would receive a line like the following:

refs/heads/master 67890 refs/heads/foreign 12345

although the full object name would be supplied. If the foreign ref does not yet exist the <remote object name> will be the all-zeroes object name. If a ref is to be deleted, the <local ref> will be supplied as (delete) and the <local object name> will be the all-zeroes object name. If the local commit was specified by something other than a name which could be expanded (such as HEAD~, or an object name) it will be supplied as it was originally given.

If this hook exits with a non-zero status, git push will abort without pushing anything. Information about why the push is rejected may be sent to the user by writing to standard error.

Workaround

The only workaround that I've seen, to read StdIn from PowerShell, is to use $Host.UI.ReadLine(), however that has a bug where it results in duplicate output. Also, the ReadLine() method hangs script execution. If you reach the last line of input, and you call ReadLine() again, your script will completely hang.

Yeah, let's just waste all the time and effort that the community has put into reporting bugs and usability issues. Good plan as usual, Microsoft.

Trevor Sullivan (@pcgeek86)

Trevor Sullivan (@pcgeek86) I had a similar issue while creating a mechanism to handle powershell hooks natively in git for Windows. The trick is to grab stdin directly and read from it, as shown under the cookbooks heading here https://github.com/fireflycons/git-hook-dispatcher

GitHub
GitHub - fireflycons/git-hook-dispatcher: Git hooks for Windows - native without git-bash etc.
Git hooks for Windows - native without git-bash etc. - GitHub - fireflycons/git-hook-dispatcher: Git hooks for Windows - native without git-bash etc.

Navigation Menu

Repository navigation

CLI: Stdin input does not implicitly become pipeline input #9497

Environment data

Bash

PowerShell

Workaround

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CLI: Stdin input does not implicitly become pipeline input #9497

Description

Environment data

Activity

bergmeister commented on Nov 14, 2019

mklement0 commented on Nov 14, 2019

bergmeister commented on Nov 14, 2019

mvalikar commented on Dec 18, 2019

bergmeister commented on Dec 18, 2019

jiasli commented on Sep 14, 2022

Bash

PowerShell

jiasli commented on Sep 14, 2022

mklement0 commented on Sep 14, 2022

jiasli commented on Sep 15, 2022

mklement0 commented on Sep 15, 2022

pcgeek86 commented on Jan 14, 2023

Workaround

pcgeek86 commented on Nov 16, 2023

fireflycons commented on Dec 4, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions