Description
With the fast approaching removal of synchronous XMLHttpRequest (i.e.: Chrome 88 is removing this), I am looking for the next optimal alternative method to precache a video.
"Sychronous XMLHttpRequest is a horrible idea" -
said no one ever
Yes, you're right for most scenarios but this is different.
Previously
On android and ios, the company I work for has an SDK that opens a webview in the background, injects HTML into it and waits for the onload event to fire. This notifies the SDK when the webview is ready to be shown to the user.
It is imperative that when a video plays there is NO buffering whatsoever for the best possible experience.
This is why when the webview is loading in the background, we precache the video synchronously with XMLHttpRequest (which by consequence, delays the onload event from being fired).
Possible solutions
We've thought about some different solutions, and they each have their pros and cons; here are a few:
Preload content via <link rel="preload" ... />
Embed the video within the index.html page in base64 (if the video weights 2-3Mo, it'll weigh 30% more after converting to base64)
(1) is the cleanest method, but requires some heavy changes on our backend for various reasons. There is also no guarantee that the video will be fully cached by the time the browser/webview appears. There is no guarantee that the priority of the precaching will be the same across webviews and mobile browsers. Users can deactivate the precaching features, for example, in their Chrome configuration. It also does not work when the connection is 4G or lower (sigh).
(2) is a hacky and unoptimized method but is relatively simple to implement compared to (1)
Question
What is the next best method to precache a video in the background of a webview/mobile browser that:
Guarantees (or closely guarantees) no buffering when the video is played
Is done within webview/browser
Is (preferably) cross mobile browser/webview compatible
(preferably and not required) delays the onload event from being triggered
Note: not all users may have a 4g or wifi connection.
Note2: tag is in autoplay
To guarantee that there is no buffering when the video is played, the video can first be downloaded as a blob using the native fetch API and then converted to objectURL using window.URL.createObjectURL. This URL can then be used as the source for video element
This is a native API built-in the webview/browser and the compatibility report can be found here Can I use fetch
And instead of listening for onload event, listen for some other custom made event which you can manually control. This will give better flexibility in future also.
<script>
fetch('http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4').then(response => {
//Video Download Started
return response.blob();
}).then(data => {
//Video Downloaded"
let objectUrl = window.URL.createObjectURL(data);
document.querySelector("#myvideoPlayer").src = objectUrl;
//Trigger a custom event here
}).catch(error => {
// Log Error
})
</script>
<video controls id="myvideoPlayer"></video>
The new solution is by using the Cache API
caches = window.caches;
caches.open("app-assets").then((cache) => {
cache.add(linkToFileToBeCached).then(() => {
// Now the file is cached. Start rendering the app!
});
});
If you are loading an html page - use preload="auto" in the video tag. This tells the browser to download the entire video on page load (the default is preload = "metadata" which downloads 3-5% of the video).
You can then look at the mediaEvent canPlayThrough to fire to know when the video is ready to play(MDN reference).
Have you thought about streaming the video? A properly configured stream should begin playback immediately, and have little to no buffering as the adaptive bitrate algorithm can change the video delivered based on the device screen AND network throughput. api.video has a great service (with SDKs for iOS, Android and several web backends)
Related
I'm trying to display a continuous video stream (live-stream) in a browser.
Description:
My client reported a video stream doesn't work in the Chrome browser. I thought it will be an easy case, I even tried to write a demo, to prove streaming should be available with just HTML5 native video tag:
https://github.com/mishaszu/streaming-video-demo
No problems with random video but:
the particular video stream on client-side doesn't work.
With html code:
<video id="video-block" width="320" height="200" autoplay>
<source src="url/to/my/video" type="video/mp4">
</video>
it shows loader for a while and dies.
What I know about the stream:
1. Codec used: H264-MPEG-4 AVC (part 10) (avc1)
2. It's a live stream, not a file, so I can't use command like MP4Box from a terminal with it
3. Because it's live stream it probably doesn't have "end of file"
4. I know it's not broken because VLC is able to display it
5. I tried native HTML 5 video tag with all Media type strings (just in case to check all codecs for mp4)
As I mentioned trying different mime types didn't help, I also tried to use MediaSource but I am really not sure how to use it with a live stream, as all information I found made assumptions:
a) waiting for resolve promise and then appends buffer
b) adding the event listener for updateend to appends buffer
I think in the case of a live stream it won't work.
Conclusion:
I found a lot of information about how a streamed file might contain metadata (at the beginning of the file or at the end)... and I ended up with a conclusion that maybe I do not fully understand what's going on.
Questions:
What's the proper way to handle the mp4 live stream?
If a native HTML video tag should support the live stream, how to debug it?
I thought that maybe I should look for something like HLS but for mp4 format?
I've went through the same - I needed to mux an incoming live stream from rtsp to HTML5 video, and sadly this may become non-trivial.
So, for a live stream you need a fragmented mp4 (check this SO question if you do not know what that is:). The is the isobmff specification, which sets rules on what boxes should be present in the stream. From my experience though browsers have their own quirks (had to debug chrome/firefox to find them) when it comes to a live stream. Chrome has chrome://media-internals/ tab, which shows the errors for all loaded players - this can help debugging as well.
So my shortlist to solve this would be:
1) If you say that VLC plays the stream, open the Messages window in VLC ( Tools -> Messages ), set severity to debug, and you should see the mp4 box info in there as the stream comes in, verify that moof boxes are present
2a) Load the stream in chrome, open chrome://media-internals/ in a new tab and inspect errors
2b) Chrome uses ffmpeg underneath, so you could try playing the stream with ffplay as well and check for any errors.
2c) You are actually incorrect about mp4box - you could simply load a number of starting bytes from the stream, save to a file and use mp4box or other tools on that (at worst it should complain about some corrupted boxes at the end if you cut a box short)
If none of 2a/2b/2c provide any relevant error info that you can fix yourself, update the question with the outputs from these, so that others have more info.
I am using screenshare with getUserMedia for a web application I am developing.
I am currently using Chrome with a Chrome extension using the API below to achieve the functionality, but the functionality is also possible in Firefox and may become natively available(I believe it was possible in the past) via getDisplayMedia or other means in the future.
https://developer.chrome.com/extensions/desktopCapture
During screenshare, a native dialog is shown at the bottom of the browser to stop the screenshare. I would like for a way to handle this event, but I am unaware of a common, standard way to achieve this. Is there an appropriate way to do this(preferably in all APIs), or should I achieve this by track events?
I am not quite sure as to how Chrome unspecified implementation behaves, but yes, you would normally have to listen to your VideoTrack's onended event.
This snippet should work only in FF, which are the only one allowing non-chrome pages to access this feature.
navigator.mediaDevices.getUserMedia({video: {mediaSource: 'screen'}}).then(stream => {
document.getElementById('vid').srcObject = stream;
const v_track = stream.getVideoTracks()[0];
v_track.onended = e => console.log('stopped');
}).catch(console.error);
<video id="vid" autoplay></video>
And in case it doesn't even work (because of StackSnippets© heavy security rules), here is a fiddle.
I'm using the MediaRecorder API along with the Canvas captureStream method to encode a VP8 video stream of a canvas in browser. This data is sent to FFmpeg via binary web socket.
var outputCaptureStream = $('canvas')[0].captureStream(30);
var mediaRecoder = new MediaRecoder(outputCaptureStream, {
mimeType: 'video/webm'
});
mediaRecorder.ondataavailable = function (e) {
ffmpegStdin.write(e.data);
}
mediaRecoder.start(1000);
For some reason, the stream seems to be randomly switching to a lower resolution mid-stream. FFmpeg isn't happy about this:
Input stream #0:0 frame changed from size:1280x720 fmt:yuv420p to size:1024x576 fmt:yuv420p
[vp8 # 0x2a02c00] Upscaling is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[vp8 # 0x2a02c00] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel#ffmpeg.org)
I suspect that it has something to do with excessive CPU usage and that Firefox is trying to be helpful by scaling down the video. My questions:
Does Firefox scale down the video on the fly?
If so, what conditions cause this to happen? (CPU load? Stream backpressure?)
Is it possible to prevent Firefox from doing this?
Is there a different explanation for this behavior that I'm missing?
Firefox will rescale (downscale) WebRTC/getUserMedia video if it detects the system's CPU is being overloaded. There are a few prefs in about:config that control this behavior, but it's not controllable via JS.
You can disable the feature by setting
media.navigator.load_adapt=false
You can look at the other media.navigator.load_adapt.* flags for some control over the behavior. By default you will get downscaling if the CPU gets pegged more than 90% for 3 seconds.
In app I can use http://developer.android.com/reference/android/os/PowerManager.WakeLock.html
but is there a way to keep webpage running and prevent from going to sleep?
It would be nice if it runs at least on android.
You can use: https://github.com/richtr/NoSleep.js
Prevent display sleep and enable wake lock in any Android or iOS web browser.
Note that the library has some reliability/performance issues on some platforms/browsers. Users have found solutions that are listed in the issue comments and pull requests, but they have not been added since the repo owner appears not to be active currently.
It's recommended that you check those pull requests (and/or issues) for potential improvements before using in production.
You can use the Wake Lock web API (check support)
https://web.dev/wakelock/
In an app there are a couple of ways you can do it, but I guess you mean just in a mobile web page, viewed in any browser via Android. With normal HTML/Javascript/etc., I really, really doubt it.
It actually may be possible using Flash (on flash-enabled phones with plugins enabled), though, at least in specific circumstances. I say this because, in a test app without the WAKE_LOCK permission, loading this swf file into a WebView caused the following exception on some devices:
java.lang.SecurityException: Neither
user ##### nor current process has
android.permission.WAKE_LOCK
Even if this did work, however, it would run the risk of crashing apps or browsers that did not have the WAKE_LOCK permission. It may be possible due to bad code in the Adobe Flash Player plugin, rather than any intentional functionality.
Play fake looped VIDEO or AUDIO on your page
You can use this a quick example to add a looped video with fake data to your page and prevent mobile device from sleep:
// Create the root video element
var video = document.createElement('video');
video.setAttribute('loop', '');
// Add some styles if needed
video.setAttribute('style', 'position: fixed;');
// A helper to add sources to video
function addSourceToVideo(element, type, dataURI) {
var source = document.createElement('source');
source.src = dataURI;
source.type = 'video/' + type;
element.appendChild(source);
}
// A helper to concat base64
var base64 = function(mimeType, base64) {
return 'data:' + mimeType + ';base64,' + base64;
};
// Add Fake sourced
addSourceToVideo(video,'webm', base64('video/webm', 'GkXfo0AgQoaBAUL3gQFC8oEEQvOBCEKCQAR3ZWJtQoeBAkKFgQIYU4BnQI0VSalmQCgq17FAAw9CQE2AQAZ3aGFtbXlXQUAGd2hhbW15RIlACECPQAAAAAAAFlSua0AxrkAu14EBY8WBAZyBACK1nEADdW5khkAFVl9WUDglhohAA1ZQOIOBAeBABrCBCLqBCB9DtnVAIueBAKNAHIEAAIAwAQCdASoIAAgAAUAmJaQAA3AA/vz0AAA='));
addSourceToVideo(video, 'mp4', base64('video/mp4', 'AAAAHGZ0eXBpc29tAAACAGlzb21pc28ybXA0MQAAAAhmcmVlAAAAG21kYXQAAAGzABAHAAABthADAowdbb9/AAAC6W1vb3YAAABsbXZoZAAAAAB8JbCAfCWwgAAAA+gAAAAAAAEAAAEAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAIVdHJhawAAAFx0a2hkAAAAD3wlsIB8JbCAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAAIAAAACAAAAAABsW1kaWEAAAAgbWRoZAAAAAB8JbCAfCWwgAAAA+gAAAAAVcQAAAAAAC1oZGxyAAAAAAAAAAB2aWRlAAAAAAAAAAAAAAAAVmlkZW9IYW5kbGVyAAAAAVxtaW5mAAAAFHZtaGQAAAABAAAAAAAAAAAAAAAkZGluZgAAABxkcmVmAAAAAAAAAAEAAAAMdXJsIAAAAAEAAAEcc3RibAAAALhzdHNkAAAAAAAAAAEAAACobXA0dgAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAIAAgASAAAAEgAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABj//wAAAFJlc2RzAAAAAANEAAEABDwgEQAAAAADDUAAAAAABS0AAAGwAQAAAbWJEwAAAQAAAAEgAMSNiB9FAEQBFGMAAAGyTGF2YzUyLjg3LjQGAQIAAAAYc3R0cwAAAAAAAAABAAAAAQAAAAAAAAAcc3RzYwAAAAAAAAABAAAAAQAAAAEAAAABAAAAFHN0c3oAAAAAAAAAEwAAAAEAAAAUc3RjbwAAAAAAAAABAAAALAAAAGB1ZHRhAAAAWG1ldGEAAAAAAAAAIWhkbHIAAAAAAAAAAG1kaXJhcHBsAAAAAAAAAAAAAAAAK2lsc3QAAAAjqXRvbwAAABtkYXRhAAAAAQAAAABMYXZmNTIuNzguMw=='));
// Append the video to where ever you need
document.body.appendChild(video);
// Start playing video after any user interaction.
// NOTE: Running video.play() handler without a user action may be blocked by browser.
var playFn = function() {
video.play();
document.body.removeEventListener('touchend', playFn);
};
document.body.addEventListener('touchend', playFn);
If you build a WebViewGold/WebView app on Android (while having the actual webpage/web app in such a wrapper), all these mentioned approaches here will not work. But then you can also do set
PREVENT_SLEEP = TRUE;
in Config.java which should do the trick.
On iOS devices, just refreshing the page in Javascript every few seconds will keep the screen awake. This seems to be the correct strategy, hopefully Android will adopt this in a future version.
Ok, my new website has just gone live, delivered through Google Apps. On a lark, I decided to include a javascript / HTML5 Lunar Lander clone (Martian Lander) which I wrote as an exercise a while back. The game works fine when I open it locally, but when it's delivered through GAE, the sounds don't seem to load on every system. In mobile safari, none of them load. In safari on the desktop, they all load reliably on my computer, but not on some other computers. In Chrome (on the desktop) it seems to work, but in Chrome in iOS, only one sound loads. On the desktop, it always seems to be the same sound which fails to load (explode1.mp3), which is the smallest of the sounds I'm loading. As you can see, if you click that link, the sound downloads fine from the server...
At first the problem seemed to be related to case sensitivity, so I switched the case in the filename, but that fix didn't keep working. This is a problem, as my loading bar is directly tied to how many resources have loaded, so it just sits there waiting for a GET request with no reply... Has anyone experienced anything like this, where a GET receives no reply on a specific resource, but loading the resource directly works fine?
I should say that I'm very new to most of these technologies, so it seems quite likely to me that I just made some novice mistake. Unfortunately, I'm not sure what those novice mistakes would be, seeing as I'm a novice!
Here's the code I use to load the sounds:
function loadSound(soundName) {
var newElement = document.createElement("audio");
newElement.addEventListener("canplaythrough", assetLoaded, false);
document.body.appendChild(newElement);
var audioType = supportedAudioFormat(newElement);
if (audioType == "") {
alert("no audio support");
return;
}
newElement.setAttribute("src", "lander/sounds/" + soundName + "." + audioType);
console.log("loading sound " + newElement.src + "...");
return newElement;
}
and...
function assetLoaded() {
var assetName = this.src;
numAssetsLoaded++;
console.log("loaded asset " + numAssetsLoaded + " (" + assetName + ")");
if (numAssetsLoaded >= numAssetsToLoad) {
shipSpriteSheet.removeEventListener("load", assetLoaded, false);
pointImage.removeEventListener("load", assetLoaded, false);
thrustAudioElement.removeEventListener("canplaythrough", assetLoaded, false);
explosionAudioElement.removeEventListener("canplaythrough", assetLoaded, false);
victoryAudioElement.removeEventListener("canplaythrough", assetLoaded, false);
musicTrackElement.removeEventListener("canplaythrough", assetLoaded, false);
gameState = GAME_STATE_INIT;
}
}
If you take a look at the console output, you'll see that all of the sounds begin loading (particularly explode1.mp3) but don't necessarily finish and call assetLoaded...
UPDATE:
It seems to be the consensus is that I should not be using mp3 (incidentally, I'm already using mp3, AAC, AND ogg, but defaulting to mp3), and also that I should use the Web Audio API. These are both welcome pieces of input, and I will make the necessary changes. However, I still don't have an answer to the original question, which is, "Why does one particular sound not load reliably on desktop while the others load with no problem?" Anybody wanna take a crack at that one? Or is the answer going to be something like, "These things are highly unpredictable, and there's no way to fix it except by switching to a more dependable methodology, like Web Audio API"?
UDATE:
Here's an excerpt from my app.yaml file, which, I gather, helps GAE setup the server.
- url: /(.*\.(mp3|ogg|wav))
static_files: \1
upload: (.*\.(mp3|ogg|wav))
Some things to be aware of:
You shouldn't use MP3 for HTML5 games.
You will need to dual-encode all your sounds to both AAC (.m4a) and Ogg Vorbis (.ogg) to ensure they can be played everywhere, since there is no one format which can be played everywhere.
You must ensure your server has the correct MIME types for the audio files. Some browsers will happily play audio if the server says it has the wrong MIME type; others will fail silently. For AAC and Ogg Vorbis the types are audio/mp4 and audio/ogg respectively.
Most mobile devices can only play one sound at a time, and iOS generally doesn't let you play audio unless it's in a user-initiated input event (such as touchstart).
You'll probably want to use the Web Audio API where supported (Chrome and iOS 6+) since playback is more reliable and polyphonic even on iOS - but note iOS still mutes the Web Audio API until a user input event.
This is not a direct answer to your question why sound is not being played, but more like what you should do with your game sound effects.
For game sound effects I suggest you use HTML5 Web Audio API which gives more control over how sounds are played (pitch of the sound effect, less delay in playback, etc):
http://www.html5rocks.com/en/tutorials/webaudio/intro/
iOS 6+ supports Web Audio https://developer.apple.com/technologies/ios6/
Web audio is not supported in FF yet, but the support is coming