VideoEncoder API with H.264 high profile config produces choppy video

jozefchutka

jozefchutka

I have observed there is something wrong with VideoEncoder API esp. in combination with H.264 high profile producing choppy video when played in browser.

I am using opera
Version: 93.0.4585.11
Update stream: Stable
System: Windows 10 64-bit
Chromium version: 107.0.5304.88

Steps to reproduce:

run attached HTML code (creates h264 stream from simple animation on canvas)
save generated blob.h264
run generated ffmpeg command to mux into blob.h264.mp4 (without re-encoding)
play blob.h264.mp4 in browser to observe choppy video

<body>
<style>
canvas, a, code {display:block}
</style>
<script>
(async () => {
// avc1.640016  (H.264 high, 2.2) - choppy
// avc1.420028 (H.264 baseline, 4.0) - smooth
const codec = "avc1.640016";
const width = 640;
const height = 360;
const framerate = 20;
const config = {codec, width, height, framerate,
	avc:{format:"annexb"}, hardwareAcceleration:"prefer-software"};
console.log(await VideoEncoder.isConfigSupported(config));

const chunks = [];

const encoder = new VideoEncoder({
	output:chunk => {
		const buffer = new ArrayBuffer(chunk.byteLength);
		chunk.copyTo(buffer);
		chunks.push(buffer);
	},
	error:error => console.log(error)
});
encoder.configure(config);

const canvas = document.createElement("canvas");
canvas.width = width;
canvas.height = height;
document.body.append(canvas);

const frames = framerate * 5;
const ctx = canvas.getContext("2d");
for(let i = 0; i < frames; i++) {
	ctx.clearRect(0, 0, width, height);
	ctx.beginPath();
	ctx.rect(i / frames * (width * .8), height / 2.5, width / 5, height / 5);
	ctx.fillStyle = "red";
	ctx.fill();
	
	const frame = new VideoFrame(canvas, {
		format:"RGBA",
		codedWidth:width,
		codedHeight:height,
		timestamp:Math.round(i / framerate * 1000000),
		duration:Math.round(1 / framerate * 1000000)});
	
	const keyFrame = !(i % framerate);
	encoder.encode(frame, {keyFrame});
	frame.close();
}

await encoder.flush();
encoder.close();

const blob = new Blob(chunks);
const name = "blob.h264";

const link = document.createElement("a");
link.href = URL.createObjectURL(blob);
link.download = name;
link.innerHTML = `download ${name} (${blob.size} bytes)`;

const code = document.createElement("code");
code.textContent = `ffmpeg -r ${framerate} -i ${name} -vcodec copy ${name}.mp4`;

document.body.append(link, code);
})()
</script>

Observations:

h264 generated in Opera using avc1.640016 encoder plays choppy in Opera
h264 generated in Chrome using avc1.640016 encoder plays smooth in Opera
h264 generated in Opera using avc1.640016 encoder plays smooth in VLC
h264 generated in Opera using avc1.420028 encoder (baseline profile) plays smooth in Opera and VLC

burnout426

burnout426 Volunteer

If you mess with the Angle flag, does the resulting mp4 still play choppy?

jozefchutka

jozefchutka @burnout426

@burnout426 tried OpenGL and D3D11, but no difference

burnout426

burnout426 Volunteer

A bug (DNA-103558) was filed internally for this to investigate, when there's time. I'll post if there are any updates.

burnout426

burnout426 Volunteer

See https://bugs.chromium.org/p/chromium/issues/detail?id=1383643#c5 and https://bugs.chromium.org/p/chromium/issues/detail?id=1383643#c7, which might have something to do with this.

jozefchutka

jozefchutka

Thanks for keeping an eye on this. Potentially related, but hard to say for sure.

My issue is only reproducible on opera using hardwareAcceleration:"prefer-software", while such configuration it not supported in my chrome or edge.