Console Process

Or: How to catch output from a batch file safely and accurately.

Some time ago I needed to catch the output of a batch file in a string. No big deal: just run the batch file, redirect the output to a file, wait for the batch file to finish, read the output file, and finally delete it. Simple, but not very elegant – and not very efficient, either. Having to create a temporary output file means generating a unique file name, and supplying it to the batch file. And, of course, creating, writing, reading, and deleting temporary files is never as good as simply catching the output into a string directly.

First approach: System.Diagnostics.Process

As it turns out, the .NET built-in Process class in System.Diagnostics supports redirecting the standard output stream. Googling on Process and 'RedirectOutput' yielded many examples, all along the same lines:

Process.Create()
Process.RedirectOutput
Wait For Process To Finish
Read StandardOutput

So far, so good. A few wrapper methods later I was catching batch output in a string, without the hassle of temporary files.

Redirecting standard error

A few months later I revisited my code because I was writing a complex batch file to build an application. I needed to test it, have some idea of the length of operations involved, but most importantly I wanted to be able to spot error output at a glance. The batch file produced a lot of information on the screen, and an occasional error was easy to miss. The whole idea was to write the batch file in such a way that it would never produce error output if all was well. If output appeared on standard error, the process failed. If not, I could safely assume everything ran as intended.

Of course, the Process class supports redirecting standard output and standard error separately. So I added the option to redirect standard error in my code. However, the downside of that approach quickly surfaced: just by looking at standard error output there was no way to determine which line in the batch file caused the error. If you look a little closer at the way cmd.exe handles standard output and standard error, you’ll see that it simply mixes them on screen. Then, there is no way to separate error output from normal output. By explicitly separating them, I could quickly learn if any errors occurred, but I lost the ability to tell where they occurred! And I needed both.

So I tried getting the best of both worlds. I would try to write a shell application that – like cmd.exe – would mix standard output and standard error on screen, marking the error output in red. But it would also catch standard error separately, so it could tell me in the end if any errors occurred.

In .NET 2.0, the Process class has two events called OutputDataReceived and ErrorDataReceived that are raised when data is written to standard output and standard error, respectively. So my first go at it simply caught those, and sent them to the same rich text box, marking the standard error output in red. Alas, not a complete success.

Some more Googling set me on the track of the cause: the C standard library, which is used – on some way or another – in many console applications. Normally, this library will write data to both standard output and standard error in a non-buffered way, i.e.: if a character is ready to be written to either standard output or standard error, it’s written immediately. However, for efficiency reasons buffering is used if the library detects that the output stream has been redirected. This causes a very undesirable ‘batching’ of text lines in standard output and standard error, making errors show up much further down the output than intended.

The solution: using CreateProcess and pipes

Then I decided to bypass the Process class and use good-old CreateProcess to help me run the batch file. Redirecting is accomplished by creating pipes (there is a good example of that from Microsoft) but catching output from two streams at once is tricky. You need to read the right stream at the right moment to prevent a deadlock: if reading from the standard output pipe is blocked – because the batch file is not producing any standard output – it in turn can get blocked writing to standard error – because no-one is reading that pipe.

A bit more reading on the subject turned up that the way to do it, is to create two separate threads, one for each stream. So there you have it: set up a set of pipes to redirect the standard streams, then use CreateProcess() to set up a process that will run the batch file, letting the process inherit the pipes. Do not run the process yet: start threads that will read from the redirected output pipes and optionally a thread that will write to the redirected input pipe. Then start the process, and wait for it to finish. In the mean time, make the reading threads raise an event every time a complete line was read – much like the built-in Process class does.

Unfortunately, this didn’t completely solve my ‘delayed error output’ problem. On the positive side, reading the output streams with a buffer size of 1 did cause a remarkable improvement in the accuracy of the error output, when mixed with standard output. So I decided to accept it, since I have found no way of fooling the C standard library into thinking standard output was not redirected.

Using the new ConsoleProcess class, it was easy to write my batch file tester-and-runner. I called it Shell Runner. It runs a command more or less like cmd.exe would run it, mixing standard output with error output and marking the latter in red. (While I was at it, I added an option to prefix every line with a time stamp in blue.) In an optional second pane, the error output is shown separately, making it really easy to spot if there were any errors running the batch file – and allowing Shell Runner to exit automatically if no errors occurred. That was part of the purpose of this whole exercise: to have a batch file that runs and disappears off the screen if all goes according to plan, but that sticks on the screen if not – and displays enough information to find out where it all went awry.

Then, of course, I added some interface spice by adding a tool bar with buttons to start and stop the command, and to save the output to a file. But have a look at Shell Runner for the details.

Which brings me to another topic: how to kill a process tree in .NET. My batch file ran several other commands, some of which took a long time running (hence the need for timing information). When exiting Shell Runner, it makes sense to kill the process it started. But if that process started other processes, we need to in fact kill the process tree, like Task Manager does. But that is a different story.

32/64-bit issues

The first version of the source code was written for 32-bit Windows - and not very tidily, either. I've cleaned it up (replacing a lot of uints with IntPtrs, bools with uints, and using Marshal.SizeOf where appropriate). This eliminates a lot of ugly casting, too. I expected these changes to allow the code to run under 64-bit Windows, but it doesn't!

I feel I'm almost there, but somehow I still run into an issue with redirecting the standard handles. The code works fine under 64-bit Windows when I don't attempt to redirect stdin, stdout, and stderror, but as soon as I do the calls to CreatePipe return 0. I'm baffled as to why this is. If anybody can help me out, please do!

In the mean time, enjoy a version of ConsoleProcess.cs that looks a lot better and runs just fine on 64-bit Windows, albeit as a 32-bit process. It does allow you to start both 32-bit and 64-bit processes, though, so Shell Runner 1.0.4 (using this version of ConsoleProcess) now supports 64-bit Windows.

Source Code

View ConsoleProcess.cs