Pipes between native processes are too slow #2176

jhclark · 2016-09-03T16:29:57Z

Problem: Native (non-object) pipes between native processes are ~1000X faster versus PowerShell object pipes. This is especially problematic for us systems people who like to work on 500MB-5GB files routinely.

Proposed solution: Automatically detect 2 native processes (neither is a function nor commandlet), and use a direct system pipe.

Timings from my box:

Powershell object pipe

PS> time { cat bigfile.txt | measure -line }
Minutes           : 15
Seconds           : 14
TotalSeconds      : 914.4296154

Using cygwin utilities

PS> time { FastPipe { C:\cygwin\bin\cat.exe bigfile.txt | c:\cygwin\bin\wc -l } }
TotalSeconds      : 1.7950218

*FastPipe is a hack I wrote that stringifies the scriptblock and passes it to cmd.exe for its faster raw pipes. Perhaps this is the best solution for now?

The text was updated successfully, but these errors were encountered:

GeeLaw · 2016-09-05T07:07:00Z

No. There is NOT pipes between native commands.

When invoking a native command, PowerShell runs the command, interprets the stdout of it and make it a typed output.

If you are sending an object to a native command, PowerShell outputs it into the stdin of that command.

With those two combined, if you do native | native on PowerShell, PowerShell will "help" you corrupt the stdout of the first native and of course, the stdin of the second native.

Use Start-Process or System.Diagnostics.Process to invoke native commands.

jhclark · 2016-09-05T16:00:00Z

I am aware that this is not the current implementation and that is what I would like changed.

The object in the middle model makes sense for functions and commandlets, but not native processes, both in terms of efficiency and possible data corruption.

If you believe this is a feature and not a deficiency, could you provide a concrete user case?

Get Outlook for Android

On Mon, Sep 5, 2016 at 12:07 AM -0700, "Gee Law" notifications@github.com wrote:

No. There is NOT pipes between native commands.

When invoking a native command, PowerShell runs the command, interprets the stdout of it and make it a typed output.

If you are sending an object to a native command, PowerShell outputs it into the stdin of that command.

With those two combined, if you do native | native on PowerShell, PowerShell will "help" you corrupt the stdout of the first native and of course, the stdin of the second native.

Use Start-Process or System.Diagnostics.Process to invoke native commands.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

GeeLaw · 2016-09-05T16:55:02Z

I don't believe that this is a useful feature. There is already an issue addressing this. See #559. This issue is mostly a duplicate/sunset of that issue.

And do note my sarcasm on this "smart oo pipe between native utilities", I said PS "helped" you currput the streams.

jhclark · 2016-09-06T16:07:38Z

I could go for better language than "this is not a useful feature" -- I'm quite certain that it is.

However, I do buy that it is a duplicate of #1908, which appears to be discussed in #559 now and actively worked on by @vors on the PowerShell team.

jhclark changed the title ~~Pipes between natives processes are too slow~~ Pipes between native processes are too slow Sep 3, 2016

joeyaiello added Resolution-Duplicate The issue is a duplicate. Issue-Enhancement the issue is more of a feature request than a bug WG-Engine core PowerShell engine, interpreter, and runtime labels Sep 5, 2016

jhclark closed this as completed Sep 6, 2016

Pipes between native processes are too slow #2176

Pipes between native processes are too slow #2176

jhclark commented Sep 3, 2016

GeeLaw commented Sep 5, 2016

jhclark commented Sep 5, 2016

GeeLaw commented Sep 5, 2016 •

edited

jhclark commented Sep 6, 2016

Pipes between native processes are too slow #2176

Pipes between native processes are too slow #2176

Comments

jhclark commented Sep 3, 2016

GeeLaw commented Sep 5, 2016

jhclark commented Sep 5, 2016

GeeLaw commented Sep 5, 2016 • edited

jhclark commented Sep 6, 2016

GeeLaw commented Sep 5, 2016 •

edited