WebGPU Rendering: Part 10 Rendering Videos and Images

Introduction

I have been reading through this online book on WebGPU. In this series of articles, I will be going through this book and implementing the lessons in a more structured typescript class approach and eventually we will build three types of WebGPU renderers: Gaussian Splatting, Ray tracing and Rasterization.

In this article we will talk about how to render what is shown on our canvas to either an image or a video. We will explore two ways to do this by either encapsulating the rendering output logic within our WebGPUContext OR directly rendering form our application using a separate reference to the canvas.

The following link is the commit in my Github repo that matches the code we will go over.

Screenshotting

To enable our renderer to take a screenshot, we are going to give our WebGPUContext object a state for if a screenshot should be taken (and create a button to set that state to true).

<button onClick={() => webGPUContextRef.current!.takeScreenshot = true}>Screenshot</button>

Then — inside our render loop — we will check for this state‘s value, and if it is true, we will render and download a screenshot.

 if (this._takeScreenshot) {
    this._takeScreenshot = false;
    this._canvas.toBlob((blob) => {
      if (blob === null) return;
      const a = document.createElement("a");
      a.href = URL.createObjectURL(blob);
      a.download = "screenshot.png";
      a.click();
    });
  }

Another option — which we will explore when generating the recording — would be to simply handle all render output actions directly via the canvas instead of adding additional state to our WebGPUContext.

Recording

Unlike screenshotting where we passed state directly into our renderer, our video recording will generate a video entirely through interactions with the canvas we are rendering to.

We introduce a react state variable for if we are recording or not and then create a button to allow us to toggle recording on or off.

Most of the work for managing the recording is done in a function called toggleRecording.

const toggleRecording = async () => {
    if (!recording) {
     const stream = canvasRef.current!.captureStream(30);
     mediaRecorderRef.current = new MediaRecorder(stream, { mimeType: "video/webm" });

     const recordedChunks: BlobPart[] = [];

     mediaRecorderRef.current!.ondataavailable = (e) => {
      if (e.data.size > 0) {
        recordedChunks.push(e.data);
      }
     }

     mediaRecorderRef.current!.onstop = () => {
      const blob = new Blob(recordedChunks, { type: "video/webm" });
      const url = URL.createObjectURL(blob);

      // Create a downloadable link
      const a = document.createElement("a");
      a.href = url;
      a.download = "recorded-video.webm";
      a.click();

      // Clean up
      URL.revokeObjectURL(url);
     }

     mediaRecorderRef.current!.start();
    } else {
      mediaRecorderRef.current!.stop();
    }

    setRecording(prev => !prev);
  }

To start recording, we grab a stream pointing to our canvas and then define a recorded chunk array and a callback to stream chunks into that array when they become available. We also create a callback for when the recording is stopped which will turn the chunks into a webm and download the final rendering to the client.

Conclusion

In this short article, we saw that in order to generate output from our render, we must interact with the canvas. We did this two ways: using our WebGPUContext to proxy our interaction with the canvas or directly interacting with the canvas ourselves.