Skip to main content
info

Hi!

Thanks for checking out the draft of my Remotion course. This is my first course and I'm excited to share the work in progress with interested early bird readers. By doing so I hope to get your feedback so we can improve the course together while it's being written.

If you'd like to know when I've updated this page please sign up to the mailing list on https://remotionkit.com.

All feedback is welcome. Just send me a DM on Twitter or reply to the email you got from the RemotionKit mailing list.

Cheers,
Marcus

caution

This is a draft (a work in progress) and at times things might not make sense. Sometimes it'll be notes for myself, or I haven't gotten to finishing a certain section, and other times it could be that I think I'm done but where my explanation isn't quite good enough yet.

Over time each section will reach a more finished and polished state. I'll try to mark sections with their status as I go.

FYI, no such markings are here yet... I'll get to it! 😌

If you find issues, please let me know. In the above information section I've listed the best ways to send feedback.

info
Changelog

2022-10-02

  • Draft: create render queue API, and split server-side rendering into three sections

2022-09-04

  • Draft first half of the server-rendering section
  • Draft of hooking up remotion composition to dynamic input on frontend
  • Improve code change highlighting in code blocks

2022-08-27

  • Initial outline
  • Drafted setup section and Remotion first steps section
Author's notes

Overview​

info

Section status: draft

In this course we'll build a video creator app with Next.js and Remotion. Along the way we'll use Node.js and related technologies to make a server-side rendering backend.

When you're finished you'll have a solid starting point for continuing whever you want to go with template editing, live preview, and server-side rendering.

Setup​

info

Section status: draft

Setup folder​

mkdir my-app && cd my-app

Bootstrap Next.js site​

Run command to create nextjs project

npx create-next-app web --use-npm --ts

Install TailwindCSS​

npm i -DE tailwindcss postcss-preset-env postcss autoprefixer
./web/postcss.config.js
module.exports = {
plugins: {
tailwindcss: {},
autoprefixer: {},
},
};
./web/tailwind.config.js
/** @type {import('tailwindcss').Config} */
module.exports = {
content: [
'./pages/**/*.{js,ts,jsx,tsx}',
'./components/**/*.{js,ts,jsx,tsx}',
],
theme: {
extend: {},
},
plugins: [],
};
./web/styles/globals.css
/* Add this to the top of globals.css */
@tailwind base;
@tailwind components;
@tailwind utilities;

If there's an error like Parsing error: Cannot find module 'next/babel', then do this.

./web/.eslintrc.json
{
"extends": "next/core-web-vitals",

// so babel knows to look in this dir
// https://www.google.com/search?q=cannot+find+module+next%2Fbabel&oq=cannot+find+module+next&aqs=chrome.0.0i512j69i57j0i20i263i512j0i512l7.4460j0j7&sourceid=chrome&ie=UTF-8
// https://github.com/sanity-io/sanity/issues/3320
"eslint.workingDirectories": ["./web"]
}

First steps with Remotion​

info

Section status: draft

Set up Remotion​

Install Remotion and the Remotion player in your Next.js app.

npm install --save remotion @remotion/player @remotion/cli

and also make sure the remotion packages are up to date

npx remotion upgrade

https://www.remotion.dev/docs/player/integration

Create a file for our first Remotion composition.

./web/remotion/MyComposition.tsx
import { useCurrentFrame } from 'remotion';

export const MyComposition = () => {
const frame = useCurrentFrame();
return (
<div
style={{
flex: 1,
textAlign: 'center',
fontSize: '7em',
backgroundColor: 'rebeccapurple',
}}
>
The current frame is {frame}.
</div>
);
};

Then create a file to hold our videos.

./web/remotion/Video.tsx
import { Composition } from 'remotion';
import { MyComposition } from './MyComposition';

export const RemotionVideo: React.FC = () => {
return (
<>
<Composition
id="MyComp"
component={MyComposition}
durationInFrames={60}
fps={30}
width={1280}
height={720}
/>
</>
);
};

The last file we need to create is a Remotion root file.

./web/remotion/index.tsx
import { registerRoot } from 'remotion';
import { RemotionVideo } from './Video';

registerRoot(RemotionVideo);

With this in place let's open up our simple Remotion project in the preview player.

./web
npx remotion preview remotion/index.tsx

The remotion preview will open. Our basic Remotion composition is displayed.

This is a really useful tool. Let's add it as an npm script so we don't have to remember the command.

./web/package.json
{
/* ... */
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start",
"lint": "next lint",
"remotion:preview": "remotion preview remotion/index.tsx"
}
/* ... */
}

Now we can open the preview by running an npm script.

./web
npm run remotion:preview

Create a basic video​

// TODO

Add the video to the website​

Install @remotion/player

npm install --save @remotion/player

Just to be safe, let's make sure all our remotion packages are in sync.

npx remotion upgrade

Now we can add the player to the Next.js site.

./web/pages/index.tsx
import { Player } from '@remotion/player';
import type { NextPage } from 'next';
import Head from 'next/head';
import Image from 'next/image';
import { MyComposition } from '../remotion/MyComposition';
import styles from '../styles/Home.module.css';

const Home: NextPage = () => {
return (
<div className={styles.container}>
<Head>
<title>Create Next App</title>
<meta name="description" content="Generated by create next app" />
<link rel="icon" href="/favicon.ico" />
</Head>

<main className={styles.main}>
<h1 className={styles.title}>
<span className="text-red-500">Welcome</span> to{' '}
<a href="https://nextjs.org">Next.js</a>
<a href="https://remotion.dev">Remotion</a>
</h1>
<p className={styles.description}>
Get started by editing{' '}
<code className={styles.code}>pages/index.tsx</code>
</p>
{/* I've added a dashed red border so we can see the edges of the video. */}
<div className="border-2 border-dashed border-red-500">
<Player
style={{
// The remotion player will resize to fit inside the player.
// We set a fixed width and the height will be automatically adjusted for you.
width: 640,
}}
component={MyComposition}
durationInFrames={120}
compositionWidth={1920}
compositionHeight={1080}
fps={30}
loop
autoPlay
controls
/>
</div>
</main>

<footer className={styles.footer}>
<a
href="https://vercel.com?utm_source=create-next-app&utm_medium=default-template&utm_campaign=create-next-app"
target="_blank"
rel="noopener noreferrer"
>
Powered by{' '}
<span className={styles.logo}>
<Image src="/vercel.svg" alt="Vercel Logo" width={72} height={16} />
</span>
</a>
</footer>
</div>
);
};

export default Home;

Isolate styles with iframe​

https://www.remotion.dev/docs/miscellaneous/snippets/player-in-iframe

Now that the Remotion composition is added to our Next.js page there's a problem: it doesn't look the same as when we opened the composition in the preview tool.

The reason is that the styles in our Next.js app are being applied to the Remotion composition too.

Let's fix this by placing the Remotion player inside of an iframe.

Create a file for out Player <iframe/> wrapper component. We're not going to go in to the details of what's happening in the <IframePlayer /> component but in short it places the Remotion player inside of an <iframe/>. Other than that you use it exactly the same as the <Player /> component from @remotion/player.

./web/components/IframePlayer.tsx
import { Player, PlayerRef, PlayerProps } from '@remotion/player';
import React, {
forwardRef,
MutableRefObject,
useEffect,
useRef,
useState,
} from 'react';
import ReactDOM from 'react-dom';

const className = '__player';
const borderNone: React.CSSProperties = {
border: 'none',
};

const IframePlayerWithoutRef = <T,>(
props: PlayerProps<T>,
ref: MutableRefObject<PlayerRef>
) => {
const [contentRef, setContentRef] = useState<HTMLIFrameElement | null>(null);
const resizeObserverRef = useRef<ResizeObserver | null>(null);

const mountNode = contentRef?.contentDocument?.body;

useEffect(() => {
if (!contentRef || !contentRef.contentDocument) return;

// Remove margin and padding so player fits snugly
contentRef.contentDocument.body.style.margin = '0';
contentRef.contentDocument.body.style.padding = '0';

// When player div is resized also resize iframe
resizeObserverRef.current = new ResizeObserver(([playerEntry]) => {
const playerRect = playerEntry.contentRect;
contentRef.width = String(playerRect.width);
contentRef.height = String(playerRect.height);
});

// The remotion player element
const playerElement = contentRef.contentDocument.querySelector(
'.' + className
);
if (!playerElement) {
throw new Error(
'Player element not found. Add a "' +
className +
'" class to the <Player>.'
);
}
// Watch the player element for size changes
resizeObserverRef.current.observe(playerElement as Element);
return () => {
// ContentRef changed: unobserve!
(resizeObserverRef.current as ResizeObserver).unobserve(
playerElement as Element
);
};
}, [contentRef]);

const combinedClassName = `${className} ${props.className ?? ''}`.trim();

return (
// eslint-disable-next-line @remotion/warn-native-media-tag
<iframe ref={setContentRef} style={borderNone}>
{mountNode &&
ReactDOM.createPortal(
// @ts-expect-error PlayerProps are incorrectly typed
<Player<T> {...props} ref={ref} className={combinedClassName} />,
mountNode
)}
</iframe>
);
};

export const IframePlayer = forwardRef(IframePlayerWithoutRef);

Switch out the <Player/> with the new <IframePlayer />.

./web/pages/index.tsx
import { Player } from '@remotion/player';
import type { NextPage } from 'next';
import Head from 'next/head';
import Image from 'next/image';
import { IframePlayer } from '../components/IframePlayer';
import { MyComposition } from '../remotion/MyComposition';
import styles from '../styles/Home.module.css';

const Home: NextPage = () => {
return (
<div className={styles.container}>
<Head>
<title>Create Next App</title>
<meta name="description" content="Generated by create next app" />
<link rel="icon" href="/favicon.ico" />
</Head>

<main className={styles.main}>
<h1 className={styles.title}>
<span className="text-red-500">Welcome</span> to{' '}
<a href="https://remotion.dev">Remotion</a>
</h1>
<p className={styles.description}>
Get started by editing{' '}
<code className={styles.code}>pages/index.tsx</code>
</p>

{/* I've added a dashed red border so we can see the edges of the video. */}
<div className="border-2 border-dashed border-red-500">
<Player
<IframePlayer
style={{
// The remotion player will resize to fit inside the player.
// We set a fixed width and the height will be automatically adjusted for you.
width: 640,
}}
component={MyComposition}
durationInFrames={120}
compositionWidth={1920}
compositionHeight={1080}
fps={30}
loop
autoPlay
controls
/>
</div>
</main>

<footer className={styles.footer}>
<a
href="https://vercel.com?utm_source=create-next-app&utm_medium=default-template&utm_campaign=create-next-app"
target="_blank"
rel="noopener noreferrer"
>
Powered by{' '}
<span className={styles.logo}>
<Image src="/vercel.svg" alt="Vercel Logo" width={72} height={16} />
</span>
</a>
</footer>
</div>
);
};

export default Home;

Adding interactivity to the template​

info

Section status: draft

We've added a Remotion video to the Next.js page. At the moment there's no way to change what is in the video. Remotion is built on top of React. That means we can use basically any React tool to make the video customizable.

Let's add an input box to the page and then hook it up to the Remotion template.

./web/pages/index.tsx
import type { NextPage } from 'next';
import Head from 'next/head';
import Image from 'next/image';
import { useState } from 'react';
import { IframePlayer } from '../components/IframePlayer';
import { MyComposition } from '../remotion/MyComposition';
import styles from '../styles/Home.module.css';

const Home: NextPage = () => {
const [message, setMessage] = useState('Hello!');

return (
<div className={styles.container}>
<Head>
<title>Create Next App</title>
<meta name="description" content="Generated by create next app" />
<link rel="icon" href="/favicon.ico" />
</Head>

<main className={styles.main}>
<h1 className={styles.title}>
<span className="text-red-500">Welcome</span> to
<a href="https://remotion.dev">Remotion</a>
</h1>
<p className={styles.description}>
Get started by editing{' '}
<code className={styles.code}>pages/index.tsx</code>
</p>
<div className="my-4">
<input
className="px-2 py-1"
value={message}
onChange={(e) => setMessage(e.target.value)}
/>
</div>

{/* I've added a dashed red border so we can see the edges of the video. */}
<div className="border-2 border-dashed border-red-500">
<IframePlayer
inputProps={{ message }}
style={{
// The remotion player will resize to fit inside the player.
// We set a fixed width and the height will be automatically adjusted for you.
width: 640,
}}
component={MyComposition}
durationInFrames={120}
compositionWidth={1920}
compositionHeight={1080}
fps={30}
loop
autoPlay
controls
/>
</div>
</main>

<footer className={styles.footer}>
<a
href="https://vercel.com?utm_source=create-next-app&utm_medium=default-template&utm_campaign=create-next-app"
target="_blank"
rel="noopener noreferrer"
>
Powered by{' '}
<span className={styles.logo}>
<Image src="/vercel.svg" alt="Vercel Logo" width={72} height={16} />
</span>
</a>
</footer>
</div>
);
};

export default Home;

Send dynamic input to the Remotion video​

The input box is added. Now we'll pass the message as a prop to the Remotion player. The player will pass the props to the Remotion composition its displaying.

./web/pages/index.tsx
<div className="border-2 border-dashed border-red-500">
<IframePlayer
inputProps={{ message }}
style={{
// The remotion player will resize to fit inside the player.
// We set a fixed width and the height will be automatically adjusted for you.
width: 640,
}}
component={MyComposition}
durationInFrames={120}
compositionWidth={1920}
compositionHeight={1080}
fps={30}
loop
autoPlay
controls
/>
</div>

Nothing happens yet and that's because our composition doesn't do anything with the input props yet.

Use input props in a Remotion composition​

Now that the player is hooked up to receive the message from the input box on the page we are going to make the composition display the message in the video.

./web/remotion/MyComposition.tsx
import { useCurrentFrame } from 'remotion';

export const MyComposition = () => {
export const MyComposition: React.FC<{ message: string }> = ({ message }) => {
const frame = useCurrentFrame();
return (
<div
style={{
flex: 1,
display: 'flex',
flexDirection: 'column',
justifyContent: 'center',
gap: '0.5em',
textAlign: 'center',
fontSize: '7em',
backgroundColor: 'rebeccapurple',
}}
>
The current frame is {frame}.
<div>The current frame is {frame}.</div>
<div>
The message is:
<br /> {message}
</div>
</div>
);
};

Intro to Server-side Rendering (SSR)​

info

Section status: draft

  • Initialize backend service project

Initialize a Node.js TypeScript project​

Let's create a folder to hold the project files for our backend service.

# in the project's root folder (not in the "web" folder)
mkdir server && cd server

First we'll initialize the project

npm init -y

This create a package.json file for us.

./server/package.json
{
"name": "server",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC"
}

Node.js doesn't support TypeScript out of the box, so we'll use ts-node to run our backend TypeScript code. We'll first install ts-node and typescript.

npm i --save typescript ts-node

Next up let's initialize TypeScript.

npx tsc --init

It'll create a tsconfig.json file that contains this project's TypeScript configuration.

We want our source code, the code in ./server/src, to be processed by TypeScript but we don't want it process third party code, the code in node_modules. We make this happen by using the include and exclude properties in tsconfig.json.

./server/tsconfig.json
{
"include": ["src/**/*"],
"exclude": ["node_modules"],
"compilerOptions": {
/* ... the rest of the config ... */
}
}

Let's create a server.ts file that echoes a message to the console. Later we'll replace the contents of this file with server code.

./server/src/server.ts
const message: string = 'Hello!';
console.log(message);

Running the command npx ts-node src/server.ts in the server folder will print the message to the console.

> npx ts-node src/server.ts
Hello!

Set up a basic express server​

Express.js is a web framework for Node.js. We'll use it to write our server.

Install the express dependency

npm i --save express

and the types for express

npm i --save-dev @types/express

Now replace the contents of server.ts with a basic Express server

./src/server.ts
const message: string = 'Hello!';
console.log(message);
import express from 'express';

const port = 3000;

const app = express();

app.get('/', (req, res) => {
res.json('Hello from server!');
});

app.listen(port, () => {
console.log(`Server listening on port: ${port}`);
});

Now we can start the server

npx ts-node src/server.ts

If everything starts alright you'll be able to go to localhost:3000 in a browser and see the message

Hello from server!

but if you had the Next.js server running at the same you probably got a EADDRINUSE error. Let's fix that.

Use a different port via environment variable​

We want to be able to running both Next.js and our express server at the same time. Both Next.js and our server will by default want to listen to port 3000. That work work so we need to choose another port for one of them. We could hard-code a different port in the source code, but that's not very flexible. Instead we're going to make it possible to choose the port when starting the server.

For that we'll read the environment variables and only default to port 3000 if there's no environment variable for the port.

./server/src/server.ts
import express from 'express';

const port = 3000;
const port = process.env.PORT ?? 3000;

const app = express();

app.get('/', (req, res) => {
res.json('Hello from server!');
});

app.listen(port, () => {
console.log(`Server listening on port: ${port}`);
});

Now let's run the server and assign a port through an environment variable

PORT=3001 npx ts-node src/server.ts

The console should print that it's listening the port you chose instead of port 3000

> PORT=3001 npx ts-node src/server.ts
Server listening on port: 3001

and if you go to localhost:3001 in a browser you should see the message

"Hello from server!"

before we continue let's add a script to package.json that'll start the server on port 3001

./server/package.json
  "scripts": {
"start": "PORT=3001 ts-node src/server.ts",
"test": "echo \"Error: no test specified\" && exit 1"
},

and now we can start the server simply by running

npm run start

Let's move on to actually building the server.

Render on the server​

Author's notes

We've done a ton of work without seeing anything fun for a while, so let's jump a few steps ahead and render a video on the server.

First off we'll need @remotion/renderer in order to render on the server so let's install it. We'll also install @remotion/bundler since the Remotion component code needs to be bundled up before it can be used on the server.

npm i --save @remotion/renderer @remotion/bundler

Then we'll add an endpoint POST /render that'll trigger a render of our Remotion composition. This is quite a bit of code, but don't worry, we'll go through and build all of this from scratch later.

./server/src/server.ts
import { bundle } from '@remotion/bundler';
import { getCompositions, renderMedia } from '@remotion/renderer';
import express from 'express';
import path from 'path';

const port = process.env.PORT ?? 3000;

const app = express();
app.use(express.json());

app.get('/', (req, res) => {
res.json('Hello from server!');
});

app.post('/render', async (req, res) => {
// this sets timeout to 60 seconds
req.setTimeout(60 * 1000);

// The composition you want to render
const compositionId = 'MyComp';

// TODO: better bundling
const entry = '../web/remotion/index';

console.log('Creating a Webpack bundle of the video');
const bundleLocation = await bundle(path.resolve(entry), () => undefined, {
// If you have a Webpack override, make sure to add it here
webpackOverride: (config) => config,
});

// Parametrize the video by passing arbitrary props to your component.
const inputProps = req.body;

// Extract all the compositions you have defined in your project
// from the webpack bundle.
const comps = await getCompositions(bundleLocation, {
// You can pass custom input props that you can retrieve using getInputProps()
// in the composition list. Use this if you want to dynamically set the duration or
// dimensions of the video.
inputProps,
});

// Select the composition you want to render.
const composition = comps.find((c) => c.id === compositionId);

// Ensure the composition exists
if (!composition) {
throw new Error(`No composition with the ID ${compositionId} found.
Review "${entry}" for the correct ID.`);
}

const fileName = `${compositionId}-${Date.now()}.mp4`;
const outputLocation = path.resolve('./tmp', fileName);

console.log('Attempting to render:', outputLocation);
await renderMedia({
composition,
serveUrl: bundleLocation,
codec: 'h264',
outputLocation,
inputProps,

onBrowserLog: (log) => {
// `type` is the console.* method: `log`, `warn`, `error`, etc.
console.log(`[${log.type}]\n${log.text}\nat ${log.stackTrace}`);
},

onStart: (props) => {
console.log(`Beginning to render ${props.frameCount}.`);
},

onProgress: ({
renderedFrames,
encodedFrames,
encodedDoneIn,
renderedDoneIn,
stitchStage,
}) => {
if (stitchStage === 'encoding') {
// First pass, parallel rendering of frames and encoding into video
console.log('Encoding...');
} else if (stitchStage === 'muxing') {
// Second pass, adding audio to the video
console.log('Muxing audio...');
}
// Amount of frames rendered into images
console.log(`${renderedFrames} rendered`);
// Amount of frame encoded into a video
console.log(`${encodedFrames} encoded`);
// Time to create images of all frames
if (renderedDoneIn !== null) {
console.log(`Rendered in ${renderedDoneIn}ms`);
}
// Time to encode video from images
if (encodedDoneIn !== null) {
console.log(`Encoded in ${encodedDoneIn}ms`);
}
},

onDownload: (src) => {
console.log(`Downloading ${src}...`);
return ({ percent, downloaded, totalSize }) => {
// percent and totalSize can be `null` if the downloaded resource
// does not have a `Content-Length` header
if (percent === null) {
console.log(`${downloaded} bytes downloaded`);
} else {
console.log(`${Math.round(percent * 100)}% done)`);
}
};
},
});
console.log('Render done!');

res.json({ message: 'success' });
});

app.listen(port, () => {
console.log(`Server listening on port: ${port}`);
});

Oof. That's a lot of code. Let's try it out and then dig into it. Start the server

npm run start

Then open up a new terminal and perform a POST request to the server. Keep an eye on your server console when you run the following command.

curl -X POST localhost:3001/render -H 'Content-Type: application/json' -d '{"message": "SSR FTW!"}'

It will take a bit of time, but you should see your server first bundling the Remotion project and then moving on to render a video.

If everything worked out you'll find a freshly server-rendered video in ./server/tmp/.

OK, now let's build it from scratch... but better!

Problem: Rendering blocks everything!​

Right now there's a big problem with our server-side rendering. Each render blocks everyone else from making renders until the current in-progress render is done.

That's unacceptable!

We can avoid this problem by using a job queue.

Server-side rendering with an in-memory queue​

info

Section status: draft

Author's notes
Use an in-memory queue (we'll do a persistent queue later)

We want our server to be accept requests while rendering already accepted requests in the background. We'll do that by implementing a system similar to what happens in a restaurant.

When someone orders something that order is written down on a paper slip and hung up where the cooks can see the orders. When a cook sees one it's picked up and the dish is cooked. While cooking the serving staff can continue accepting orders. When a dish is ready it's picked up by serving staff who brings it to the customer.

This is called a job queue, and that's what we'll build next.

Let's remove the rendering code before we start

./server/src/server.ts
import { bundle } from '@remotion/bundler';
import { getCompositions, renderMedia } from '@remotion/renderer';
import express from 'express';
import path from 'path';

const port = process.env.PORT ?? 3000;

const app = express();
app.use(express.json());

app.get('/', (req, res) => {
res.json('Hello from server!');
});

app.post('/render', async (req, res) => {
// this sets timeout to 60 seconds
req.setTimeout(60 * 1000);

// The composition you want to render
const compositionId = 'MyComp';

// TODO: better bundling
const entry = '../web/remotion/index';

console.log('Creating a Webpack bundle of the video');
const bundleLocation = await bundle(path.resolve(entry), () => undefined, {
// If you have a Webpack override, make sure to add it here
webpackOverride: (config) => config,
});

// Parametrize the video by passing arbitrary props to your component.
const inputProps = req.body;

// Extract all the compositions you have defined in your project
// from the webpack bundle.
const comps = await getCompositions(bundleLocation, {
// You can pass custom input props that you can retrieve using getInputProps()
// in the composition list. Use this if you want to dynamically set the duration or
// dimensions of the video.
inputProps,
});

// Select the composition you want to render.
const composition = comps.find((c) => c.id === compositionId);

// Ensure the composition exists
if (!composition) {
throw new Error(`No composition with the ID ${compositionId} found.
Review "${entry}" for the correct ID.`);
}

const fileName = `${compositionId}-${Date.now()}.mp4`;
const outputLocation = path.resolve('./tmp', fileName);

console.log('Attempting to render:', outputLocation);
await renderMedia({
composition,
serveUrl: bundleLocation,
codec: 'h264',
outputLocation,
inputProps,

onBrowserLog: (log) => {
// `type` is the console.* method: `log`, `warn`, `error`, etc.
console.log(`[${log.type}]\n${log.text}\nat ${log.stackTrace}`);
},

onStart: (props) => {
console.log(`Beginning to render ${props.frameCount}.`);
},

onProgress: ({
renderedFrames,
encodedFrames,
encodedDoneIn,
renderedDoneIn,
stitchStage,
}) => {
if (stitchStage === 'encoding') {
// First pass, parallel rendering of frames and encoding into video
console.log('Encoding...');
} else if (stitchStage === 'muxing') {
// Second pass, adding audio to the video
console.log('Muxing audio...');
}
// Amount of frames rendered into images
console.log(`${renderedFrames} rendered`);
// Amount of frame encoded into a video
console.log(`${encodedFrames} encoded`);
// Time to create images of all frames
if (renderedDoneIn !== null) {
console.log(`Rendered in ${renderedDoneIn}ms`);
}
// Time to encode video from images
if (encodedDoneIn !== null) {
console.log(`Encoded in ${encodedDoneIn}ms`);
}
},

onDownload: (src) => {
console.log(`Downloading ${src}...`);
return ({ percent, downloaded, totalSize }) => {
// percent and totalSize can be `null` if the downloaded resource
// does not have a `Content-Length` header
if (percent === null) {
console.log(`${downloaded} bytes downloaded`);
} else {
console.log(`${Math.round(percent * 100)}% done)`);
}
};
},
});
console.log('Render done!');

res.json({ message: 'success' });
});

app.listen(port, () => {
console.log(`Server listening on port: ${port}`);
});

A simple queue containing render jobs​

In the basic sense a queue is just a list of things in order, so let's create the most basic version of a queue.

./server/src/server.ts
import express from 'express';

const port = process.env.PORT ?? 3000;

let queue: string[] = [];

const app = express();
app.use(express.json());

app.get('/', (req, res) => {
res.json('Hello from server!');
});

app.listen(port, () => {
console.log(`Server listening on port: ${port}`);
});

This list of strings will be our render processing queue. Each string item in the array will correspond to a "render job". Let's have a closer look at the render job.

./server/src/server.ts
import express from 'express';

const port = process.env.PORT ?? 3000;

type RenderJobId = string;

type RenderJob = {
id: RenderJobId;
};

let queue: string[] = [];
let queue: RenderJobId[] = [];
let renderJobs = new Map<RenderJobId, RenderJob>();

const app = express();
app.use(express.json());

app.get('/', (req, res) => {
res.json('Hello from server!');
});

app.listen(port, () => {
console.log(`Server listening on port: ${port}`);
});

The array queue is a list of RenderJobIds, and we have a new Map object that stores RenderJobs by their RenderJobId. This structure is a good start for us to create the restaurant-style order system. Let's create the endpoints that allow us to talk to the backend.

Render queue API-endpoints​

Before we take a look at actually rendering the jobs we need a way to create, list and check the status of our renders from the frontend. So let's do that now.

First remove the root endpoint, and add an endpoint that returns all jobs in the render queue.

./server/src/server.ts
import express from 'express';

const port = process.env.PORT ?? 3000;

type RenderJobId = string;

type RenderJob = {
id: RenderJobId;
};

let queue: RenderJobId[] = [];
let renderJobs = new Map<RenderJobId, RenderJob>();

const app = express();
app.use(express.json());

app.get('/', (req, res) => {
res.json('Hello from server!');
});
app.get('/render-jobs', (req, res) => {
res.json({
items: [...renderJobs.values()],
});
});

app.listen(port, () => {
console.log(`Server listening on port: ${port}`);
});

Run the server and make a request to the new endpoint

curl -X GET localhost:3001/render-jobs -H 'Content-Type: application/json'

And you should get a response like this

{ "items": [] }

which is quite uninteresting so far.

We need a way to create jobs, so let's add an endpoint for creating render jobs.

./server/src/server.ts
import express from 'express';

function makeFingerprint() {
// returns a semi-random string like this: '16ff118259e6a'
return Math.random().toString(16).slice(2);
}

const port = process.env.PORT ?? 3000;

type RenderJobId = string;

type RenderJob = {
id: RenderJobId;
};

let queue: RenderJobId[] = [];
let renderJobs = new Map<RenderJobId, RenderJob>();

const app = express();
app.use(express.json());

app.get('/render-jobs', (req, res) => {
res.json({
items: [...renderJobs.values()],
});
});

app.post('/render-jobs', (req, res) => {
// create a render job
const renderJob: RenderJob = {
id: makeFingerprint(),
};

// add the render job to the render job store
renderJobs.set(renderJob.id, renderJob);

// also add the render job ID to the queue
queue.push(renderJob.id);

// respond with the render job JSON
res.json(renderJob);
});

app.listen(port, () => {
console.log(`Server listening on port: ${port}`);
});
What's up with the `makeFingerprint()` function?
The `makeFingerprint()` function creates a semi-random alphanumeric string we can use as render job IDs (like this: '16ff118259e6a'). It works for now, but later on we probably want to use something more robust, like the `nanoid` package.

Now run the server and make a post request to create a render job.

curl -X POST localhost:3001/render -H 'Content-Type: application/json'

Which should return the new job

{ "id": "16ff118259e6a" }

and requesting all jobs should now include the newly created job

curl -X GET localhost:3001/render -H 'Content-Type: application/json'
{
"items": [{ "id": "16ff118259e6a" }]
}

Finally, in order to round things off let's add a final backend endpoint where we can get a specific render job.

./server/src/server.ts
import express from 'express';

function makeFingerprint() {
// returns a semi-random string like this: '16ff118259e6a'
return Math.random().toString(16).slice(2);
}

const port = process.env.PORT ?? 3000;

type RenderJobId = string;

type RenderJob = {
id: RenderJobId;
};

let queue: RenderJobId[] = [];
let renderJobs = new Map<RenderJobId, RenderJob>();

const app = express();
app.use(express.json());

app.get('/render-jobs', (req, res) => {
res.json({
items: [...renderJobs.values()],
});
});

app.post('/render-jobs', (req, res) => {
// create a render job
const renderJob: RenderJob = {
id: makeFingerprint(),
};

// add the render job to the render job store
renderJobs.set(renderJob.id, renderJob);

// also add the render job ID to the queue
queue.push(renderJob.id);

// respond with the render job JSON
res.json(renderJob);
});

app.get('/render-jobs/:id', (req, res) => {
const renderJob = renderJobs.get(id);

if (!renderJob) {
return res.status(404).json({ message: 'Not found' });
}

return res.json(renderJob);
});

app.listen(port, () => {
console.log(`Server listening on port: ${port}`);
});

Now that we have an API to for creating and getting render jobs it's time to make the backend render things again.

Processing the render queue​

// TODO: add back rendering, but skip accepting props from the frontend for now

Let's render!​

// TODO

renderMedia() from @remotion/renderer

Update the job with intermediate progress​

// TODO

The on* callbacks from renderMedia()

Problems with an in-memory queue​

// TODO

  • All jobs are forgotten when the server restarts/crashes
  • It would be nice to separate the queue from the processing server; more processing servers = more renders in parallell

Server-side rendering with a persistent queue and worker nodes​

// TODO

Redis + Bull?

Backend TODO​

Build a remotion bundle we can use on the server​

// TODO

Deploy to production​

// TODO

Render.com? Some other PaaS?

Later​

Queue priority​

// TODO

Some jobs are more important than others... like premium customers! Add priorities!

Dockerize​

// TODO

Dockerize Next.js app​

./web/Dockerfile
FROM node:14-alpine as build

RUN apk update
RUN apk --no-cache --virtual build-dependencies add \
jpeg-dev \
cairo-dev \
giflib-dev \
pango-dev \
python3 \
make \
g++

WORKDIR /app

# Add the source code to app
COPY . /app

# Install all the dependencies
RUN npm install

# Generate the build of the application
RUN npm run build

# Production image, copy all the files and run next
FROM node:14-alpine AS runner
WORKDIR /app

# Copy the build output to replace the default nginx contents.
# COPY --from=build /app/next.config.js ./
COPY --from=build /app/public ./public
COPY --from=build /app/.next ./.next
COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/package.json ./package.json

USER root

EXPOSE 3000

CMD ["npm", "run", "dev"]

Dockerize queue​

// TODO

Dockerize backend​

// TODO

Dockerize render workers​

// TODO

Run everything with Docker Compose​

// TODO