When trying to load videos onto a Three.js texture, I noticed that the Chrome is loading a video in ~2Mb chunks and playing the video once the chunk loads:
CHROME:
I am comparing this with firefox where a single GET request is shown. This request actually never fully resolves but the video loads smoothly on my texture.
FIREFOX:
The reason this is of interest to me is I am trying to map a video onto a Threejs texture and how the video loads effects when I can update my model. The fact that google only loads quite large chunks means that if the time of the video is changed the video cannot load->play in time.
One can see these results in the server I am working on:
https://blackfynnpythonlink.ml/hoverGraph/MPB/simple_heart/
Corresponding repo:
https://github.com/Tehsurfer/MPB
Related
I have a MP4 video of a product which shows a 360 of the product (so it looks like it's spinning) - I have it autoplaying on the page and looks great.
I want the user to be able to click and drag on the video to basically rewind or fastforward the video, so they can kind of scroll back and forth and spin the product around as they wish.
We have used the js-cloudimage-360-view library and used images, but because we wanted this as smooth as possible each colour had 360 images and loading times weren't great, we have MP4s which are small in size and load fast but it's just the srubbing I can't get right.
I'd had a look at this https://codepen.io/webandapp/pen/xEjjOJ too but it's supper jumpy, essentially we want if the user drags to the right it fast forwards as they drag, when they stop it should play from that point (and rewinds if dragging to the left) - this should be as smooth as possible.
The video is just a HTML video element like so:
<video id="three-sixty-video" preload="auto" controls loop playsinline autoplay muted>
<source src="/video/video-1.mp4" type="video/mp4;">
</video>
Scrubbing on a streamed video is limited by the speed at which the video can be downloaded, decoded and displayed.
For a large video on most devices and connections it is not practical/possible to do all this as quickly as a user can scroll along the video timeline.
This is why video containers (mp4 etc) often include a track of thumbnail images which are displayed when the user scrubs along the line.
In other words, it is not actually the video you see in the small image in the timeline, but the appropriate thumbnail for that time in the video.
When the user stop scrubbing, the player then requests that section of the video, decodes and displays it. This is (generally) not immediate.
If your video is small enough and is downloaded completely then you may find you can scrub as quickly as you want, but even then it may be device/client dependent - a quick local test with a number of videos seem to suggest it should scrub ok when available locally:
Using a relatively small 30MB, 50 second local mp4 video, available locally on he computer hard drive (a MAC), scrubbing is relatively smooth using the Quicktime client player.
The same video on the same device being viewed by the Safari browser rather than directly by the QuickTime client appears to be similar, although it is hard to be sure. The playback may well use very similar paths under the skin.
Using a much larger 2G, 4K, 2 minute 30 second video, also locally on the hard drive, scrubbing is again relatively smooth both in Safari and QuickTime.
As a side note, if you do decide the solution is to have the video fully downloaded before you play, that is also an interesting challenge. Most effort is usually focused on the opposite, allowing a video to start playing before it fully downloads. You may find that, providing the video is not too big, having the header info at the end (the MooV atom for mp4) will force the browser to download it fully. This is the opposite of the mp4 faststart you will see discussed some places. I'm not sure how reliable this will be as I think some browser may be clever enough to 'search' in the video by making different requests for bytes at the start and end to find the metadata.
The video in the top is the stream from navigator.mediaDevices.getUserMedia() which is set as the source of a the first video element. The bottom video is the video from MediaRecorder API. The video is played from a Virtual Web Camera (OBS VirtualCam specifically)
The problem I'm having with this
When the video starts to stream for the web camera (once allowed) the media seems to be pushed to the MediaSource but the playback took a while to actually play (about 2-3 seconds). What kind of behaviour is this?
Another problem is the playback in the bottom video is delayed, I have been testing and I am finding that the problem is because of the mediaRecorder.ondataavailable is not triggered fast enough, thus the chucks are delayed. In what way possible that the MediaRecorder API or similar API can be used to record in much lower latency than this.
UPDATE:
What is happening here? 1. Video from camera is play directly in the video element, that element video stream is recorded through MediaRecorder API, recorded video is sent to server (websocket). The video (bottom) is playing the video from websocket. My tests shows that the latency is not from the Websocket but from the either the MediaRecorder or the playback is delayed.
I'm developing a browser based application/tool in Javascript that runs WebGL to animate images loaded by the user. The animation stuff is working good. Now I want the user to be able to save/export the animation. I'm currently using the HTML MediaRecorder API to record the browser screen on the fly/in real time, generating a WEBM file that saves to their PC.
Overall this works well enough. However if a very large image is being animated, ie 2048x2048 pixels, when the recording process starts the browser FPS drops to about 12. Obviously then the recorded animation looks crap. Now what I can do is detect the FPS, and if it drops below say 24fps I can slow down the animation accordingly. eg if fps drops to 12, I set the animation to half speed while recording. Afterwards I can then run the output video in VLC player, increase playback speed to 200%, and the video looks smooth.
So, my question... Can I somehow set the encoding information of the WEBM file so that it will play smoothly in the player? Tell the player what the FPS is? (ie 12 in the above example). Or tell it what the playback speed is?
How do I know what the default recording FPS is with HTML MediaRecorder? I've tried using the FFMPEG tools on the output files but not really getting much useful information from it (possibly user error here).
(Video encoding noob here, so I'm probably screwing up terminology, but hopefully my problem/intention is explained clearly enough)
EDIT: Is it possible to get the MediaRecorder API to record in step with the animation frame? It seems like it runs independent of the actual browser FPS.
So i'm trying to stream video in segments without using the MediaSource extension. (Because not all browsers support MSE). Now i'm trying to do this by loading two video elements and play the next one at the right moment. But this has a very tiny delay between switching. I tried to keep checking the currenTime of the video and after a tiny fraction play the next video element. But this doesn't really work that well (audio overlaps, or delay)
Mind you that the video are preloaded and loaded from Blob storage. So the loading shouldn't delay the playback.
How can i make this (Or another solution without flash) play smoothly without using MediaSource extensions?
Your best bet is to use two video elements positioned one on top of the other. The first one should be playing the current part (or chunk) of the video. The second video element should be loaded with the blob that contains the next chunk and be paused. It should also be hidden (you can set display:'none' or z-index:-99999). And then when the first video element ends (the end event is dispatched), call the play() method of the second video element, show it, and hide the first one. Rinse and repeat.
This is what the LifemirrorPlayer does.
If the chunks of the stream are perfectly encoded and cut then this technique works. Often it doesn't. Common problems are:
Synchronizing the audio stream is the probably the hardest. After twenty or thirty chunks are played, the audio usually loses sync with the video. This is very annoying for the viewer and hard to detect and fix (i don't know of any solution actually).
The video element doesn't end. This is usually caused by an encoder that has put too much (or too little) content into a chunk. It can also be caused by improperly cut chunks, by a buggy encoder or decoder.
Flickering when the two video elements are swapped. This depends on the browser. However most browsers deal very well with this and the swap is quick and smooth.
I am drawing many frames on a canvas using KineticJs library.
I am creating a new javascript Image object with the relevant image source for every frame and setting the new image on the canvas after the image loaded.
I see that it takes about 22ms on chrome and about 600ms on ipad safari ... that's a big difference, has anyone experienced such a behavior and does anyone have any insight on how we can overcome this issue? For example maybe I can force the ipad to use its gpu.
Any response will be highly appreciated.
This page says that iPad cuts off loading images at 6.5mb. I'm not sure if this is still true but I've definitely noticed throttling before 6.5 mb.
Since I'm using a photo scroller, what I've done is load the image names in a hidden html field and then load the next image in a hidden image tag 2.5 seconds before the scroller displays the next image. Of course, you could also load the image names in a JavaScript array. I can provide code snippet if still interested...