A Neural Frame Streaming Framework with a Simultaneous Prior for End-cloud Collaborative Rendering

Cloud rendering has gained much attention recently since it alleviates the client’s computing cost by distributing most of the rendering tasks on the cloud, and the rendered frames are compressed and streamed to a client device for display. However, the accessibility of cloud rendering is still limited by the network communication cost to stream high-resolution frames. By introducing neural networks into the video compression algorithms, SOTA methods have achieved a better compression ratio than existing video codecs like H.265. However, none of them are designed specifically for rendered content, in which features like motion vectors, albedo, and normal are readily available in nowadays rendering pipelines. We propose a new end-cloud collaborative rendering and streaming framework that generates these auxiliary features on both the end and cloud sides as a simultaneous prior, and utilizes this prior to guide both the encoding and decoding of frame compression. Our framework can empower low-end devices with high-end visual effects and reduce the cost of establishing high-speed network connections. Results show that our streaming pipeline is highly efficient and achieves higher compression ratio compared with standard video codecs and other neural compression methods.