Thanks for putting up the fix!
I left some comments above, but my biggest question mark is whether it's worth doing this given that we will change to the public API once it's available in the Windows App SDK, or are we not going to use that public API anymore?

With the current fix, it's still incomplete in that we will allocate console when it's unnecessary (check out the use of IsWindowsApplication at here and here), and this has to be done before the exe starts. This doesn't sound like a big deal because this likely matters only when a non-console application is hosting PowerShell, but it's surely incomplete.

The upside to stick with this fix is that we can avoid the dependency on the Windows App SDK, and that's actually not bad from the maintenance point of view ...

However, if this is meant to be a temp workaround and we are going to depend on Windows App SDK, then a new issue needs to be opened to track that because the current tracking issue #9970 will be closed by this fix.

@iSazonov @jborean93 @SteveL-MSFT what do you think?

With the current fix, it's still incomplete in that we will allocate console when it's unnecessary

What's the purpose of this, if Windows was to start a console application and one wasn't already allocated it will do so automatically. A brief look at the code shows it might be to hide the Window but surely the .NET Process has this exposed already. I also don't think we need to care about redirection with stdio as that is governed by whether there is input or the output is redirected somewhere which is down to the code rather than the executable being used.

With the current fix, it's still incomplete in that we will allocate console when it's unnecessary

What's the purpose of this, if Windows was to start a console application and one wasn't already allocated it will do so automatically. A brief look at the code shows it might be to hide the Window but surely the .NET Process has this exposed already. I also don't think we need to care about redirection with stdio as that is governed by whether there is input or the output is redirected somewhere which is down to the code rather than the executable being used.

If an user run a console application the user expects to get output from the application. This is why pwsh always creates a console (whether it is a console application or not) before run a console application.
The fact force me think we should always allocate a console and grab its output.
I believe we could do this for GUI apps too - if GUI allocates a console why do it so? (Skype (from Store) writes something to a console at start, I don't know whether it allocate a console always.) At least we could do that until we have a way to always identify the type of application before running. Nevertheless if GUI app inherits a console and writes something to the console the design assumes they want we read the output; otherwise the GUI application shouldn't inherits a console but allocate new one if needed.

Also we should add redirection here too for consistency.
Also we should/could do the same for associated file scenario: if we create file.findstr file, create assoc for *.findstr with findstr /f: and run file.findstr we'd want to grab the output.

I left some comments above, but my biggest question mark is whether it's worth doing this given that we will change to the public API once it's available in the Windows App SDK, or are we not going to use that public API anymore?

The best fix would be in SHGetFileInfo so that it works with AppX. Otherwise we have to do one p/invoke more.
But I guess this would be never backported to early Windows 10 versions.
And if the fix will be created it will be only in Windows 11. It is not helps us since we should support old Windows versions.
The same for Windows App SDK - will we get the API for Windows 11 only? I think yes...

So we have two questions:

1. Console allocations and output grabbing
   As I said we could always allocate a console and always grab app output since if app does output it assumes the output is by-design and we should read it.
2. Waiting application exit.
   The fix resolves the issue but better fix would be in SHGetFileInfo for all Windows 10.
   One thing more we could add - it is a cache in IsWindowsApplication() - no need to check the same application every time we run it (in a cycle).

If an user run a console application the user expects to get output from the application. This is why pwsh always creates a console (whether it is a console application or not) before run a console application.

So there are 4 scenarios here that I see:

• console powershell spawning console app
  • There will be a conhost associated with powershell and the console app inherits this
• console powershell spawning GUI app
  • There will be a conhost associated with powershell
  • I can't 100% remember if the GUI app still inherits this but it doesn't really matter here
• GUI powershell spawning console app
  • There is no conhost associated with powershell
  • When powershell starts the console app Windows will automatically assign a new conhost for it
• GUI powershell spawning GUI app
  • There is no conshost associated with powershell and the GUI app doesn't need one

The important one here is the 3rd scenario as currently PowerShell will allocate a new console to itself and have the subprocess inherit it. What confuses me is why do this at all when Windows will automatically create the conhost for a console app if there isn't one to inherit. This is how you can start a console process from a GUI app, say explorer.exe, and a new console appears.

When you start talking about redirection and having PowerShell capture (or send input) then this isn't related to the console itself. When a process is created there is a structure called STARTUPINFO which defines a few things about the new process but in particular there is the hStdInput, hStdOutput, hStdError fields. These fields are used by the parent process to override the stdin, stdout, stderr respectively of the new process allowing it to write/read data on the pipes specified. You can define this for any type of application and regardless of whether the child process is a GUI, console app, creates its own console, or inherits it, the stdio handles of that child will reflect what's defined here.

This is what allows PowerShell to capture the output of a process and is totally independent on the console that the child host is using. The only caveat here is if the child process is writing to the console directly using something like WriteConsoleOutput. This doesn't use the stdio pipes defined on the process but rather writes directly to the console buffer skipping the redirection. If the child process inherits the console from powershell it will appear in that console buffer, otherwise it will appear in the conhost that was spawned with the child. PowerShell can't capture/redirect this type of output right now, technically it could use the ConPTY APIs that Windows Terminal uses but that's orthogonal to this particular problem so let's ignore that. PowerShell spawning a new console here also doesn't matter because it will be hidden so the user still cannot see the output for scenario 3/4.

The only purpose I can see with the code to allocate a console is to hide it from the user but that's already achievable with the ProcessStartInfo.WindowStyle that I linked to before. It could technically be used to ensure the child process using an encoding known to PowerShell but this isn't done today and opens a whole can of worms.

In the end I might be missing something here but when looking at the code it sounds like we are creating the console for a scenario that Windows handles for us anyway. I personally believe this whole thing could be removed and @daxian-dbw point becomes moot based on my understanding but happy to be proven wrong.

When you start talking about redirection and having PowerShell capture (or send input) then this isn't related to the console itself.

It is true in common but we should consider how PowerShell code works and it mixes console and redirection :-) See AlwaysCaptureApplicationIO. It would be a breaking change for hosting apps if we removed this.

@jborean93 With that example, the focus is still on ISE after running [System.Diagnostics.Process]::Start($psi), so it looks like the focus will not change when the console window is hidden. Then that code does look unnecessary. The mysterious part is AlwaysCaptureApplicationIO, I just cannot make sense from it 😕

Rather run ping.exe in ISE :-)

I think that's the current code up for discussion, PowerShell will spawn the console and thus conhost leaving it run in the background rather than having it tied to the new process in general. At least when using your ping example through my snippet it acts exactly the same as the powershell.exe one. If you were to run it directly like ping.exe server then you can see the conhost survives the process due to PowerShell being the one that creates it. IMO this just adds more evidence that PowerShell creating the console if there isn't one isn't the right thing to do and thus is whole point moot.

The mysterious part is AlwaysCaptureApplicationIO, I just cannot make sense from it confused

Yea I cannot explain why it would be necessary, it seems to be set to true whenever the console PowerShell spawns is hidden but honestly that doesn't sound like a good reason. The only time I think PowerShell should be explicitly capturing data is:

• The output is being redirected for x reason
  • Stored in a var
  • Piped to another command
  • Redirected to another stream/var/null
• In a PSRemoting/server side session where the native app cannot be allowed to interact with the console directly
  • All output must come through the PS streams
  • This is required so that 1 the output isn't lost and 2 the output doesn't break the PSRemoting communication

hidden console ... This should be controlled by ProcessStartInfo.WindowStyle.

@jborean93 I understand this can be handled by ProcessStartInfo.WindowStyle, but how would PowerShell know when it should enforce WindowStyle.Hidden? It would still have to do a GetConsoleWindow call to see if there is a console already allocated, right?

If a call to GetConsoleWindow is needed for PowerShell to know that it's running in GUI application, then it might be just simpler to allocate the console and set SW_HIDE along with the GetConsoleWindow call.

Also, if leaving the console allocation to OS, will the console be reclaimed after the console app exits? When allocating in PowerShell, will the console be kept around while the process hosting PowerShell is still running? If yes to both questions, then allocating the console within PowerShell seems more efficient.

As for AlwaysCaptureApplicationIO, I looked a little more, and it seems to be used to cache the state that "PowerShell is running in a GUI application", and PowerShell should always redirect the stdout and stderr for a console application in such a case, because the new console is hidden, and the output of the application won't be visible if it's not redirected.

how would PowerShell know when it should enforce WindowStyle.Hidden? It would still have to do a GetConsoleWindow call to see if there is a console already allocated, right?

Ahh I think it finally clicked for me, yes you would only want to hide new console application windows and not GUI ones and thus need to know beforehand whether it should be hidden or not. Technically you could always hide it and just show the window after it's started once you know it's a GUI and not a console app but I'm unsure whether this is practical or whether there are other problems with this approach.

I looked a little more, and it seems to be used to cache the state that "PowerShell is running in a GUI application", and PowerShell should always redirect the stdout and stderr for a console application in such a case

I guess that makes some sense but I'm still unsure whether it's needed for PowerShell to explicitly create the console. If PowerShell knows it doesn't have a console and thus should always redirect and capture the output it doesn't actually need to spawn the console itself. It can just add the stdio redirection when starting the process and still check after it has started whether it needs to block the pipe and output the data like my.exe | Out-String.

Ultimately if what you said is that there will be a public API at some point then keeping the current behaviour and using that sounds like a better idea. I still think it would be a good idea to use the undocumented struct for older Windows versions (where you know it hasn't changed) and the new API on ones where it is available.

I'm still unsure whether it's needed for PowerShell to explicitly create the console. If PowerShell knows it doesn't have a console and thus should always redirect and capture the output it doesn't actually need to spawn the console itself.

Yes, it may not be needed for PowerShell to explicitly create the console, but the dev who wrote this code just decided to do it so setting SW_HIDE and foreground window can all be done at the same place. And maybe it has a slightly perf win given that this console will be present throughout the lifecycle of the process that hosts PowerShell so that the OS doesn't need to allocate/destroy a console for every console app that would be started by PowerShell? Don't know, but I'd rather to keep the code as is since we are able to make sense from it now 😄

Ultimately if what you said is that there will be a public API at some point then keeping the current behaviour and using that sounds like a better idea. I still think it would be a good idea to use the undocumented struct for older Windows versions (where you know it hasn't changed) and the new API on ones where it is available.

Yeah, I'm also leaning to waiting on the new public API, but don't have a strong opinion. We cannot use the undocumented struct in our code base because of compliance.

When _nativeProcess.HasExited == true, _isRunningInBackground is true even if it's really a console application.
However, when _isRunningInBackground == true, the code that set LastExitCode won't run. That code also includes some special handling to transcribing a native application.

It looks to me we still need to know whether an application is console app before starting it, in order to cover the cases where the child process has exit when code reaches here.

@iSazonov You may have missed this comment.

This is the zugzwang :-) Perhaps (I hope) we have made best what we can for starting native application but you ask about termination. The best we can do for this is to think about specific scenarios.
Difficults are (1) we can catch a race condition with process terminating, (2) even if we could always tell if an application is a console application, this does not guarantee the expected behavior (since any application can create/attach/detach/delete a console by itself).

So it makes sense only think about typical scenarios. Do you know such scenarios we should consider?

As side notice, we could use CREATE_SUSPENDED flag in the dwCreationFlags parameters of the CreateProcess() on Windows to address a race condition with process start..

The scenario is where a native console application exits very early. I don't have a working example to demonstrate that, but it surely could happen. I cannot approve this PR with this breaking change.

Using CreateProcess() means a fundamental refactoring of the native command processor, which I don't think will be approved. At this point, I think it's better we just wait for the new public API from the AppX team.

I removed _nativeProcess.HasExited from the condition - it is not critical here. Also I rebased to get latest changes.

As for refactoring, I feel this is inevitable. The new Appx API won't solve all the problems. It might make sense for PowerShell team to work with cmd.exe team to find a better way to run apps. As result, Process class could return the information whether running app is GUI or CUI.

@daxian-dbw Friendly ping...

                        catch (Win32Exception)
                        {
                            // We can not get MainModule, assume that the process has already been completed.
                            // The best thing we can do is to use default behavior like Windows PowerShell.
                            _isRunningInBackground = true;
                        }

This is the same effect as having _isRunningInBackground = ... || _nativeProcess.HasExited, isn't it? And thus, the issue I brought up above still exists.

I think we should just wait on the new API for detecting whether AppX app is CUI or GUI, instead of dealing with the subtle issues that could be caused by this change.

This is the same effect as having _isRunningInBackground = ... || _nativeProcess.HasExited, isn't it? And thus, the issue I brought up above still exists.

No, this is previous code (Windows PowerShell) default.

I have no strict desire to merge the PR so you can close if you want.

The _isRunningInBackground = true; is previous existing code, but it was not in a catch (Win32Exception) block previously . The previous related code was:

      _isRunningInBackground = true;
      if (!startInfo.UseShellExecute)
      {
          _isRunningInBackground = isWindowsApplication;
      }

I will close this PR then, but thank you for trying out this approach. I appreciate your time!

I appreciate your time!

This is my hobby :-) I'm more concerned about wasting your time, sorry.