Time Out David

You may have been wondering where I've been all these years... easy... I've been tenting my fingers Future updates at @_earthchat

moved back to <video> without webgl :( splitting to frames didn't yield any better performance. today: add scrubbable timeline

plan is to use ffmpeg.js to split into frames; unfortunately, kagamihi/ffmpeg.js was not built with any still image encoders. trying to make a new build with a still image encoder (+ gif decoder)...

Today: thinking about next steps in light of the issues posted above. one last test I want to do is to split videos into uncompressed frames and see how that goes. otherwise, probably back to <video>, but that route has a lot of issues as well :(

faster version seems to be using a concrete URL (e.g. `/path/to/file.mp4`), while the slower one was using an object URL (from `URL.createObjectURL`), since it's coming from a file upload. I can fix by passing the Blob from the file upload to Video.srcObject, but only in Safari

this is still confusing to me. here’s a test where I’m loading 11 videos from URL » <video> » THREE.VideoTexture – all load in ~3s. using almost the same code as above.

Loading <video> metadata appears to be an order of magnitude slower on iOS Safari as opposed to desktop browsers (i.e. time between `load()` -> `loadedmetadata`): on desktop, a 1mb video takes ~60ms; on iOS, it's ~900ms

tbc, I value rendering to WebGL for easy, wysiwyg export-to-video. rendering to DOM with <video> seems much more performant, so I think I'll probably fall back to that. easy export seems super important, though

3. Convert base videos to uncompressed image sequences. Since videos are synced, could combine multiple videos onto a single "spritesheet" per frame (then cut the sheet up and composite on render). seems like this would be ~10x the storage if you start from an mp4 source video 😓

2. Composite flat videos "just in time" via ffmpeg.js. This would be ideal – could reduce use to ~2 <video>s onscreen at any time. `overlay` via ffmpeg.js is slow on my iPhone (~10s). maybe this could work with a cache, but cache would become huge with infinite zoom

tried a couple of approaches to make multiple composited videos with infinite zoom performant on mobile browser via WebGL, but not feeling optimistic about any of the approaches:

Today: Make multiple composited <video>s -> three.js fast on iPhone

<video> textures are also not great performance, but that seems more solvable (read: I am probably doing something stupid)

pretty bad performance on iPhone. gifs can barely play via react-gifs -> THREE.CanvasTexture: kind of expected, but still a bummer. if I can't make react-gifs -> THREE fast, fallbacks are (1) convert via ffmpeg.js (idk how i'll do transparency), or (2) write WebGL gif player

got videos playing through THREE.Texture on iPhone – key issue was that my <video>'s `canplaythrough` event was not firing on mobile Safari, which was fixed by removing the `src` attribute from JSX and instead setting `src` in a `useEffect`. ¯\_(ツ)_/¯