Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out-String should not append a trailing newline #14444

Open
mklement0 opened this issue Dec 16, 2020 · 20 comments
Open

Out-String should not append a trailing newline #14444

mklement0 opened this issue Dec 16, 2020 · 20 comments
Labels
Issue-Enhancement WG-Cmdlets-Utility

Comments

@mklement0
Copy link
Contributor

@mklement0 mklement0 commented Dec 16, 2020

Summary of the new feature/enhancement

It should be possible to use Out-String in a manner that doesn't append a trailing newline, which is what it currently does and has always done.

Note that -NoNewLine is not a solution, because it suppresses all newlines, so that two or more output lines are then simply directly concatenated (like -join ''; e.g. 1, 2 | Out-String -NoNewLine yields 12).

Here's a demonstration of the problem:

# CURRENT behavior:
PS> ('foo' | Out-String) -replace '\r?\n', '<newline>'
foo<newline>   #  A newline was appended.

I see two options:

  • (a) Consider a breaking change - assuming it is justified to consider it a Bucket 3: Unlikely Grey Area change - and make Out-String never append a trailing newline.

  • (b) Introduce a new switch, -NoTrailingNewLine (related: #5108)

@mklement0 mklement0 added the Issue-Enhancement label Dec 16, 2020
@iSazonov iSazonov added WG-Cmdlets-Core WG-Cmdlets-Utility and removed WG-Cmdlets-Core labels Dec 17, 2020
@bpayette
Copy link
Contributor

@bpayette bpayette commented Dec 17, 2020

Out-String formats an object for presentation as a string. It should add a new line because that's how formatting works. Given that, I don't understand what the problem is - can you clarify? Thanks.

@mklement0
Copy link
Contributor Author

@mklement0 mklement0 commented Dec 18, 2020

It should add a new line

I don't think it should. Why do you think so?

I would expect:

Get-Date

and

Get-Date | Out-String

to produce the same output, yet they don't (the latter has an extra newline).

Given that piping to Out-String is a convenient way to capture stdout from an external program in a single string - and remember that strings are out-of-band objects with respect to formatting and are passed through - it is inconvenient to have a trailing newline in the resulting string with respect to further programmatic processing.

In other words:

PS> $out = cmd /c 'echo a&echo b' | Out-String

should be equivalent to:

PS>  $out = (cmd /c 'echo a&echo b') -join [Environment]::NewLine

yet it isn't, due to the trailing newline.

In short: there's no good reason to append a newline to the (formatted representation) of the last input object.

@bpayette
Copy link
Contributor

@bpayette bpayette commented Dec 19, 2020

there's no good reason to append a newline to the (formatted representation) of the last input object.

The purpose of Out-String is to render objects as presentable text which includes trailing newlines. If what you actually want to do is join a bunch of strings then use -join because that's it for.

@mklement0
Copy link
Contributor Author

@mklement0 mklement0 commented Dec 19, 2020

Yes, -join is a workaround, but that still doesn't justify the trailing newline:

  • If you output the string resulting from Out-String, it is the standard output formatting that will add the trailing newline for you - just like sending the output to a file will do that for you - just like interactively outputting 'foo' prints a newline before the next prompt is displayed.

  • If you programmatically process Out-String output, the trailing newline is an unnecessary nuisance.

Each formatted output individually may or may not have trailing newlines (e.g., Get-Date vs. 'foo').

Newlines are only needed as separators between multiple formatted representations, not as a terminators.

Again - there's no justification for Get-Date and Get-Date | Out-String to produce different output (which they do, due to the trailing newline) - both in the console and if you redirect to a file - and there's no reason why
((cmd /c 'echo a & echo b' | Out-String) -split [Environment]::NewLine).Count should report 3 rather than 2, given that there are only two output lines.

@ExE-Boss
Copy link

@ExE-Boss ExE-Boss commented Jan 15, 2021

As I said in #13071 (comment):

This is the expected behaviour, because both Out‑String and Set‑Content add a trailing newline by default.


You can use the ‑NoNewline switch with Set‑Content to disable adding the extra trailing newline:

@(
	\"$($PSStyle.Foreground.Blue)Hello$($PSStyle.Reset)\",
	'string2'
) `
	| Out-String `
	| Set-Content -Path test2.txt -NoNewline

@mklement0
Copy link
Contributor Author

@mklement0 mklement0 commented Jan 15, 2021

@ExE-Boss: Out-String is an in-memory operation, whereas Set-Content is a file operation.

When you use Out-String, you typically want to process the result in memory - otherwise, you'd just use > / Out-File directly.

For in-memory processing, it is reasonable for multiple formatted representations to be joined with a separator (a newline), and not for each representation to have a terminator, which is a concept that applies to (line-oriented text) files, especially on Unix.

If Out-String behaved that way, you could later use it with Set-Content as-is, without having to resort to the -NoNewline workaround.

@kilasuit
Copy link
Collaborator

@kilasuit kilasuit commented Jan 15, 2021

Isn't the crux of this that in interactive output in the terminal shows a trailing new line and when run in a script (where there often isn't an interactive session) it shouldn't?

If so then that can be checked within the cmdlet whether or not the session is in non-interactive mode or not & I don't believe would be a breaking change

@mklement0
Copy link
Contributor Author

@mklement0 mklement0 commented Jan 15, 2021

@kilasuit, this is primarily about the trailing newline being unhelpful in programmatic processing of Out-String output.

As for the display side of things:

Both default host output and file-creation cmdlets (Out-File / >, Set-Content) are clearly designed not to expect the data to already have a trailing newline, and therefore blindly append one (by default).
(And I don't see how interactive vs. script processing figures into this, but perhaps you can elaborate.)

Given that Out-String currently itself appends a trailing newline to the resulting in-memory string, you end up having to fight that design:

  • If you later split the string into lines, you'll end up with an extra empty array element at the end (("foo`nbar`n" -split '\n').Count is 3).

  • On printing to the host / saving to a file, you must then suppress the extra trailing newline:

    • With -NoNewLine, if you use Out-File / Set-Content (> is then not an option)
    • For to-host output you're stuck with something awkward like Write-Host -NoNewline (Get-Date | Out-String)

Of course, you can avoid the empty array element / the need for these workarounds by stripping the trailing newline manually first - ((Get-Date | Out-String) -replace '\n\z') - but the whole point of this issue is that the trailing newline shouldn't be added to the in-memory data (string) to begin with: there is no upside to doing so, only downsides.

@ExE-Boss
Copy link

@ExE-Boss ExE-Boss commented Jan 15, 2021

Out‑String also supports ‑NoNewline.

@mklement0
Copy link
Contributor Author

@mklement0 mklement0 commented Jan 15, 2021

@ExE-Boss, to quote from the OP:

Note that -NoNewLine is not a solution, because it suppresses all newlines, so that two or more output lines are then simply directly concatenated (like -join '').

For instance:

PS> 1, 2 | Out-String -NoNewline
12

See also: #5108

@jazzdelightsme
Copy link
Contributor

@jazzdelightsme jazzdelightsme commented Jan 16, 2021

I think maybe mklement0 chose Get-Date as an example because it produces a simple object that isn't a string (a DateTime), so that Out-String would be doing "some work" to convert something to a string.

But consider even just a plain string!

'asdf'

versus

'asdf' | Out-String

How would one could consider that 'asdf' is not "rendered as presentable text"? In other words, who would consider the output of the latter to be "presentable text", but not the former?

Adding newlines in is ONLY interesting where there are multiple input objects: newlines in that case are used to separate them (unless belayed by -NoNewline). The current last (trailing) newline is separating the last object from ... what? It serves no purpose; it is not separating it from anything. We should get rid of it.

Consider also this pathological scenario:

'asdf' | Out-String | Out-String | Out-String | Out-String | Out-String 

Let's stop the madness and pull this burr out from under the saddle.

(edit: I called that last scenario "pathological", because nobody is going to pipe through a long chain of Out-String commands, but I think the nerve it hits is quite reasonable: it feels like it ought to be idempotent)

@SteveL-MSFT
Copy link
Member

@SteveL-MSFT SteveL-MSFT commented Jan 20, 2021

Out-String already has a -NoNewline switch I believe for this specific reason

@mklement0
Copy link
Contributor Author

@mklement0 mklement0 commented Jan 20, 2021

@SteveL-MSFT, as previously stated, -NoNewline does not address the complaint at hand, because it omits all newlines, even between input objects.

To provide another example:

# E.g., on macOS
PS> ls / | Out-String -NoNewline
ApplicationsLibrarySystemUsersVolumesbincoresdevetchomeoptprivatesbintmpusrvar

@BobbyTables10
Copy link

@BobbyTables10 BobbyTables10 commented Dec 22, 2021

Any updates on this @SteveL-MSFT @mklement0? I'm using Out-String to generate an .xml (.nfo) file that will be picked up by Kodi to show movie details that I've scraped from an online video archive. Building up the .xml I use:

" <title>" | Out-File -LiteralPath $FileName -Encoding "UTF8" -Append

.. then insert the scraped title. What I want to end up with in the .xml file is:

<movie>
<title>Some Movie Title</title>
<plot>The story begins in 1939.
A soldier ... 
</plot>
</movie>

but what I actually end up with is something like this, which embeds newlines into the title/plot text and looks scruffy when presented in the media centre.

<movie>
<title>
Some Movie Title

</title>
<plot>
The story begins in 1939.
A soldier ... 
</plot>
</movie>

As others have said, a -NoTrailingNewLine switch would alleviate this without having to do loads of string processing?

@mklement0
Copy link
Contributor Author

@mklement0 mklement0 commented Dec 23, 2021

@BobbyTables10, the -NoTrailingNewLine / -NoLastNewLine proposal that also covers Out-String is in #5108 - but, as argued above, the much preferable option is to simply omit a trailing newline from Out-String's output to begin with.

What you're showing is an Out-File call; however, as with Out-String, if there's only one input object (string), -NoNewLine does work as expected to suppress a trailing newline.
.

@jaydeepmehta15
Copy link

@jaydeepmehta15 jaydeepmehta15 commented Jan 10, 2022

Upvote on -NoTrailingNewLine / -NoLastNewLine

@BEEDELLROKEJULIANLOCKHART
Copy link

@BEEDELLROKEJULIANLOCKHART BEEDELLROKEJULIANLOCKHART commented May 21, 2022

Is “[System.Console]::Write()” and optional manual addition of “[System.Environment]::Newline” inadequate? I may have misunderstood why Out-String exists, although it appears to me to be useless.

As demonstration, “[System.Console]::Write($(get-date))” or “[System.Console]::Write($(get-date),[System.Environment]::Newline)” appear perfect.

@BEEDELLROKEJULIANLOCKHART
Copy link

@BEEDELLROKEJULIANLOCKHART BEEDELLROKEJULIANLOCKHART commented May 21, 2022

This problem appears to have been sneakily bothersome since Out-String was added, which "http://stackoverflow.com/a/21564398/9731176" demonstrates. (I believe that it is relevant, although that example demonstrates potentially erroneous utilization of the command.)

@ImportTaste
Copy link

@ImportTaste ImportTaste commented May 23, 2022

Is [System.Console]::Write() and optional manual addition of [System.Environment]::Newline inadequate? I may have misunderstood why Out-String exists, although it appears to me to be useless.

As demonstration, [System.Console]::Write($(get-date)) or [System.Console]::Write($(get-date),[System.Environment]::Newline) appear perfect.

Out-String does special handling of some PSObjects to ensure the output looks more-or-less alike to what it would've been without Out-String. Get-Date is actually a great example of this, since [Console]::Write will change 'Monday, May 23, 2022 12:02:54 PM' to '2022-05-23 12:02:50 PM', while using Out-String makes it remain the same, but with an extra trailing newline.

Having a parameter like -NoTrailingNewline (perhaps with a -ntn alias) would be ideal.

@ImportTaste
Copy link

@ImportTaste ImportTaste commented Jun 22, 2022

Is [System.Console]::Write() and optional manual addition of [System.Environment]::Newline inadequate? I may have misunderstood why Out-String exists, although it appears to me to be useless.

As demonstration, [System.Console]::Write($(get-date)) or [System.Console]::Write($(get-date),[System.Environment]::Newline) appear perfect.

Also, I just noticed today, these methods write to host rather than stdout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Issue-Enhancement WG-Cmdlets-Utility
Projects
None yet
Development

No branches or pull requests