Low Latency Rendering

Low Latency Rendering with Dataflow Architectures

Sebastian Friston's EngD research project 

This research is concerned with latency in virtual environments (VEs). Latency is the delay between a user's physical actions and the perceived response to these actions by a computer system. Latency is a product of the processing, transport and buffering delays present in any current computer system. While latency can be distracting, it is not critical for most applications. VEs on the other hand aim to facilitate direct interaction with a digitised world. This implies the formation of a sensorimotor loop which allows users to directly sense the digitised world and perceive it as a distinct physical place. This has many applications for knowledge transfer and efficient interaction. The complication is, the formation of the sensorimotor loop is highly dependent on latency.

Our main research questions are how can dataflow computing be used to improve the latency of visual stimuli, and what implications does this have for spatial fidelity. Secondarily, we ask what effects latency has on user behaviour, including high and low level functions at latencies previously unexplored. We create prototype renderers using dataflow platforms and integrate these into real systems. We compare these systems to traditional systems using objective measures of latency and image quality. We use our novel systems to perform user studies into the effects of latency.

The low latency apparatuses minimise the largest delays in traditional GPU rendering pipelines. Our 3D apparatus achieves lower latencies and higher fidelities than traditional systems, albeit with serious constraints. We do not foresee dataflow computing shouldering the bulk of the rendering workload in the future but rather augmenting the traditional pipeline with fast latency compensation techniques. This may be an image distortion stage or otherwise. Our experiments revealed that predictions of the effects of low latency should be re-evaluated, and experimenting in this range requires great care.