Parallel Compilation with OpenCL II

Continuing from the last post, we will now go over the communication code in Flam4OCL.

Starting with the master process, the CompilerService class is instantiated once for each compiler slave process. The primary constructor, new CompilerService(deviceName) starts a new complier slave process based on the device name string passed in. Once the main program exits, it closes CompilerService with Dispose() in which CompilerService sends the quit message "END" to the associated process, waits for that process to read the message, then closes the pipe.

Points of interest are the pipeName, which is dynamically generated with Guid.NewGuid() to ensure that nothing else on the system collides with it, and the readMessage function, which exhibits the pecularity of Async.RunSynchronously. While at first glance, running an asynchronous read in a synchronous fashion might make no sense at all, it serves a vital purpose: During a normal pipe.Read, the CPU spins until it's read the number of bytes requested. This means that if one were to use pipe.Read() to listen for the next message, the CPU will spend it's time eagerly awaiting an update from the pipe rather than doing something more important, such as running the compile service so that the update can be sent in the first place! By contrast, pipe.AsyncRead yields the CPU until there is enough data in the pipe to fulfill the read request. Hence, it can listen for a message without hogging system resources. Since the master program can't do much until it receives it's compiled binary, the RunSynchronously simply makes it (or rather, whichever thread in the program that is running the class) yield the CPU and do nothing until the compilation is finished.

The code for the slave process works very similarly to the master program code, the one big difference being that it uses NamedPipeClientStream instead of NamedPipeServerStream, since it didn't create the pipe, but is merely using the pipe provided by the master program. Since the compile process could easily be idling for a long time between compile requests, it's even more important that it's readMessage is asynchronous, so that it actually idles rather than running full throttle in neutral.

Once the compilation is complete, there is a "peculiarity" involved with getting it to actually be run. After creating the program with clCreateProgramWithBinary(), you simply call clCreateKernel(), it will fail! To fix this, you have to call clBuildProgram() after clCreateProgramWithBinary(). This seems an odd thing, since the whole point of a binary is that it's already built! It seems, however, that there is some other work involved, likely something to do with building up a symbol table for kernel entry points.

Parallel Compilation with OpenCL

If you've ever developed with OpenCL, you may have noticed something quite annoying - the runtime compiler is not thread-safe! In an age of computers with ever more cores and GPUs with two orders of magnitude more cores than that, this is really an embarrassment, especially since the entire point of OpenCL is to take advantage of this parallelism!

However, fear not, for there is a work around to this madness. While the runtime compiler may not be thread safe, it is at least process safe, at least for Nvidia's implementation. It may also be process safe for ATI's implementation as well, but I don't have a card available to test this on.

Since two OpenCL compilers can in fact run in parallel if they're created by different programs, all that needs to be done to exploit this is to simply run a compiler program for each CPU and Boom! Problem solved! Right?

Well, not so fast. As with anything else concerning programming, there are details...

First off, Process.Start() is known to be one of the most expensive calls in the Windows API, taking up to a second to execute! This is because there's all manner of overhead for starting a new process, from security concerns to disk access to read in the binary. Thus, the naive method of starting a new process each time we need to compile something is a bad idea.

Instead, we need persistent compile processes, which continue running throughout the lifespan of the master program. Ideally, the master program would start up the processes, then send them bits of source code as it needed things to be compiled. The slave compile processes would then build the source and send the binaries back to the master program, which would execute them. After some time of doing this, the master program would finish, at which point it would need to not only close itself, but close the slave processes as well. Since the standard kill process command doesn't give the process time to clean up, the master will have to send the slave processes a kill message, so that they can close themselves.

The key to all this is to find an API through which the processes can send messages back and forth asynchronously and without mangling binary data (this rules out stdin/stdout. In addition, it would be nice if the API also worked with MONO.NET, so that the program can be run on other OSes.

One such API is System.IO.Pipes. Here is the messaging system that flam4OCL uses:

Here's the messaging class used by the master process:

module CompilerDriver

open System
open System.IO
open System.IO.Pipes
open System.Text
open System.Diagnostics

type CompilerService (pipe:NamedPipeServerStream)=
  let pipe = pipe
  let readMessage() =
      async {
              let! length = (pipe:>Stream).AsyncRead(sizeof)
              let buf = Array.create(BitConverter.ToInt32 (length,0)) 0uy
              let! msg = (pipe:>Stream).AsyncRead(buf)
              return buf
          } |> Async.RunSynchronously
  let writeMessage (msg:byte[])=
      let lengthBuf = BitConverter.GetBytes (msg.Length)
      pipe.Write (Array.append lengthBuf msg,0,msg.Length+sizeof)

  new(deviceName) =
      //ensure a unique pipe name
      let pipeName = Guid.NewGuid().ToString()
      let pipe = new NamedPipeServerStream(pipeName,PipeDirection.InOut,1,PipeTransmissionMode.Message)
      let compilerProcess = new Process()
      let startInfo = new ProcessStartInfo()
      startInfo.FileName <- @".\Flam4CompileService.exe"
      startInfo.Arguments <- pipeName + " " + "\"" + deviceName + "\""
      startInfo.UseShellExecute <- false
      compilerProcess.StartInfo <- startInfo
      compilerProcess.Start() |> ignore
      pipe.WaitForConnection()
      new CompilerService(pipe)

  interface System.IDisposable with
      member s.Dispose() =
          writeMessage <| Encoding.UTF8.GetBytes("END")
          pipe.WaitForPipeDrain()
          pipe.Close()
          pipe.Dispose()

  member s.Compile (kernel:string) =
      let buf = Encoding.UTF8.GetBytes(kernel)
      writeMessage buf
      pipe.WaitForPipeDrain()
      readMessage()

And the slave compiler process:

module Program

open System
open System.IO
open System.IO.Pipes
open System.Text
open Cloo

let standardOCLCompilerOptions = @"-cl-single-precision-constant -cl-denorms-are-zero -cl-mad-enable -cl-no-signed-zeros -cl-fast-relaxed-math"

[<STAThread>]
[<EntryPoint>]
let main args =
  //Connect to the pipe
  use pipe = new NamedPipeClientStream(args.[0])
  pipe.Connect()
  pipe.ReadMode <- PipeTransmissionMode.Message
   try
       let readMessage (pipe:NamedPipeClientStream)=
          async {
              let! length = (pipe:>Stream).AsyncRead(sizeof)
              let buf = Array.create(BitConverter.ToInt32 (length,0)) 0uy
              let! msg = (pipe:>Stream).AsyncRead(buf)
              return buf
          } |> Async.RunSynchronously
      let writeMessage (pipe:NamedPipeClientStream) (msg:byte[])=
          let lengthBuf = BitConverter.GetBytes (msg.Length)
          pipe.Write (Array.append lengthBuf msg,0,msg.Length+sizeof)

      let msgToStr buffer =
          buffer |> Encoding.UTF8.GetString

  //set up the OpenCL compiler for the selected device
      let clPlatform = ComputePlatform.Platforms.Item(0)
      let clContextProps = new ComputeContextPropertyList(clPlatform)
      let clDevice = (clPlatform.Devices |>
                      (fun devs -> [for device in devs -> device]) |>
                      List.filter (fun device -> device.Name=args.[1])).[0]
      let devList = new System.Collections.Generic.List()
      devList.Add clDevice
      let dev = new System.Collections.ObjectModel.ReadOnlyCollection(devList)
      let clContext = new ComputeContext(dev,clContextProps,null,System.IntPtr 0)

  //Listen to pipe and compile recieved kernels until the message "END" is recieved
      let rec readLoop() =
          let msg = readMessage pipe |> msgToStr
          if (msg.Length>0)&&(msg<>"END") then
              let clProgram = new ComputeProgram(clContext,msg)
              try
                  clProgram.Build(dev,standardOCLCompilerOptions,null,System.IntPtr 0)
              with
              | :? Cloo.ComputeException ->
                  printfn "%s" (clProgram.GetBuildLog(clDevice))
              | _ ->
                  reraise()
              writeMessage pipe (clProgram.Binaries.Item(0))
              pipe.WaitForPipeDrain()
          if (msg<>"END") then readLoop() else () //Tailcall to simulate while loop

      readLoop()
  finally
      pipe.Close()
  0

Since Wordpress keeps messing with the code format and eating the HTML attributes I insert to preserve the code in a readable, not flowing off the page format, I think it's best that I stop this post here and continue with the description in the next post.

*EDIT* Seems that Wordpress now strips every attribute from HTML, including style="overflow-x:scroll". This makes it impossible to post code without it flowing off the page. I don't have time for this nonsense, so I've moved the blog over to Blogger instead.