I am playing around with the requestAnimationFrame but I get very jerky animations in any other browser than Chrome.
I create an object like this:
var object = function() {
var lastrender = (new Date()).getTime();
var delta = 0;
return {
update: function() {
//do updates using delta value in calculations.
},
loop: function() {
var looptimestamp = (new Date()).getTime();
delta = looptimestamp - lastrender;
lastrender = looptimestamp;
this.update();
window.requestAnimationFrame(this.loop.bind(this));
}
};
}
Right now I am just drawing a single rectangle on a canvas element and moving it around. It is a very lightweight operation on the processor. This is running pretty smoothly in Chrome, and when I console log the delta value, it is almost consistant around ~17. However, if I do the same in Firefox or Safari I get the following delta values:
17-3-17-2-32-34-32-31-19-31-17-3-0-14-3-14-35-31-16-3-17-2 ... and so on
It looks as if the browser is not syncing with the display very nicely, and in all other cases than Chrome, one would get smoother animations using the old setTimeout method with 16ms as the target timeout.
Does anyone know, if it is possible to get smoother animations using requestAnimationFrame in browsers other than Chrome? Has anyone succeded in getting more stable delta values than the ones posted above in Firefox?
The reason the smooth framerate of your animation decreases is because of the memory of your browser with regards to the canvas. I don't know the real details of the performance in browsers but firefox almost immediately has a framerate drop and in chrome the drop occurs some time later.
The real problem of the framerate drop is because of the memory occupied by the canvas element. Each time you draw something to the canvas that operation is saved to the path of the canvas. This path occupies more memory each time you draw something on the canvas. If you don't empty the path of the canvas you will have framerate drops. The canvas path can't be emptied by clearing the canvas with context.clearRect(x, y, w, h);, instead you have to reset the canvas path by beginning a new path with context.beginPath();. For example:
// function that draws you content or animation
function draw(){
// start a new path/empty canvas path
canvas.beginPath();
// actual drawing of content or animation here
}
You might get smoother animation if you skip updates when delta < threshold, for example:
if (delta > 5) this.update();
Related
I would like to use requestAnimationFrame to play an HTML <video> element. This is useful because it offers greater control over the playback (e.g. can play certain sections, control the speed, etc). However, I'm running into an issue with the following approach:
function playAnimation() {
window.cancelAnimationFrame(animationFrame);
var duration = video.seekable.end(0);
var start = null;
var step = function(timestamp) {
if (!start) start = timestamp;
const progress = timestamp - start;
const time = progress / 1000;
video.currentTime = time;
console.log(video.currentTime);
if (time > duration) {
start = null;
}
animationFrame = window.requestAnimationFrame(step);
}
animationFrame = window.requestAnimationFrame(step);
}
In Google Chrome, the video plays a little bit but then freezes. In Firefox it freezes even more. The console shows that the video's currentTime is being updated as expected, but it's not rendering the new time. Additionally, in the instances when the video is frozen, the ontimeupdate event does not fire, even though the currentTime is being updated.
A simple demo can be found here: https://codepen.io/TGordon18/pen/bGVQaXM
Any idea what's breaking?
Update:
Interestingly, controlling/throttling the animationFrame actually helps in Firefox.
setTimeout(() => {
animationFrame = window.requestAnimationFrame(step);
}, 1000 / FPS);
This doesn't seem like the right approach though
The seeking of the video is usually slower than one frame of requestAnimationFrame. One ideal frame of requestAnimationFrame is about 16.6ms (60 FPS), but the duration of the seek depends on how the video is encoded and where in the video you want to seek. When in step function you set video.currentTime and then do the same thing in the next frame, the previous seek operation most likely has not finished yet. As you continue calling video.currentTime over and over again, browser still tries to execute old tasks until the point it starts freezing because it is overwhelmed with the number of tasks. It might also influence how it fires the events like timeupdate.
The solution might be to explicitly wait for the seek to finish and only after that asking for the next animation frame.
video.onseeked = () => {
window.requestAnimationFrame(step);
}
https://codepen.io/mradionov/pen/vYNvyym?editors=0010
Nevertheless you most likely won't be able to achieve the same smooth real-time playback like in the video tag, because of how the seeking operation works. Unless you are willing to drop some current frames when the previous frame is still not ready.
Basically storing an entire image for each video frame is very expensive. One of the core video compression techniques is to store full video frame only in some intervals, like every 1 second (they are called key-frames of I-frames). The rest of the frames in between will store the difference from the previous frame (P-frames), which is pretty small compared to entire image. When video plays as usual, it already has previous frame in buffer, the only thing it needs to do is apply the difference for the next frame. But when you make a seek operation, there is no previous frame to calculate the difference from. Video decoder has to find the nearest key-frame with the full image and then apply the difference for all of the following frames up until the point it finally reaches the frame you wanted to seek to.
If you use my suggestion to wait for previous seek operation to complete before requesting for the next seek, you will see that video starts smooth, but when it gets closer to 2.5 seconds it will stutter more in more, until it reaches 2.5s+ and becomes smooth again. But then again it will start stuttering up to the point of 5s, and become smooth again after 5s+. That's because key-frame interval for this video is 2.5 seconds and the farther the timestamp you want to seek to from the key-frame, the longer it will take, because more frames need to be decoded.
Chrome/Firefox are deciding to fire requestAnimationFrame at 30FPS. The profiler shows most of the time is spent idling, so its not cpu cycles being burned up.
I tried writing a delta timing to "catch up" / stay synced. The FPS is correct according to my FPS meter, but the result is very choppy animating. I also tried Drawing immediately after each Update, still choppy.
I am running a very vanilla rAF paradigm and notice the window size affects the FPS, but the profiler doesn't seem to explain why the FPS drops at full screen
let start = null;
let step = (timestamp)=>{
if(thisTimerID !== scope._timerID){
return;
}
start = start || timestamp + delay;
let current = timestamp;
let delta = current - start;
if(delta >= delay) {
this.Update();
this.Draw();
start = timestamp;
}
requestAnimationFrame(step);
};
thisTimerID = this._timerID = requestAnimationFrame(step);
Should I try requestAnimationFrame, and if I detect a low FPS when initializing, fall back to setInterval? Should I just use requestAnimationFrame anyway? Maybe there's a flaw in my delta timing logic? I know theres a huge flaw because if you switch tabs then come back, it will try to catch up and might think it missed thousands of frames!
Thank you
EDIT:
requestAnimationFrame IS firing at 60fps under minimal load. It is the Draw method that is causing the frame rate to drop. But why? I have 2 profiles showing a smaller resolution runs at 60fps, full screen does not.
60FPS 350x321 Resolution:
38FPS 1680x921 Resolution:
What's really strange to me, is the full screen version is actually doing LESS work according to the profiler.
I have been playing around with canvas and animation, with HTML5 games in mind specifically and quickly learnt the limitations of just using requestAnimationFrame (rFA) and have moved to time-based animations.
I want to maintain constant gameplay regardless of monitor refresh rate or FPS but am unsure how best to handle the animations. I have read through all sorts of implementations but have not found any best practice so to speak. Should I be using a combination of the two?
So far I have considered several options:
rFa only (changes results when fps changes):
var animate = function() {
draw();
requestAnimationFrame(animate);
}
time-based only (not always consistent):
var animate = function() {
now = Date.now();
delta = now - last;
last = now;
draw(delta);
window.setTimeout(animate, 1000/60)
}
set FPS on rFA with setInterval (not always consistent):
setInterval(function () {
draw();
requestAnimationFrame();
}, 1000/fps);
rFA trying to force fps (does not seem very robust, variable delta would work better):
var delta = 1000 / fps;
var animate = function() {
now = Date.now();
if (now - last >= delta) {
last = now;
}
draw(delta);
requestAnimationFrame(animate);
}
time-based rFA (some strange results):
var animate = function () {
now = Date.now();
delta = now - last;
last = now;
draw(delta);
requestAnimationFrame(animate);
}
Ignore the lack of browser support and the use of Date.now(), I just want to demonstrate my flow of thinking. I think that the last option is preferable, but the last two can run into problems with updating too far and missing collisions etc as well as updates taking too long that the animation looses all control.
Also when a user tabs out using rFA only the animation will pause, using a time based function to call rFA means that the game/animation will continue to run in the background which is not ideal.
What would be the best way to handle animations trying to keep consistent results regardless of fps, all of the above might be bad and my apologies for the long post (it is just what I have tried so far and am still pretty lost)? even better with with the above issues in mind?
If you have requestAnimationFrame available, I wouldn't go against it and only call draw() from its callbacks. Of course, you should always use delta timing.
Here's a sophisticated variation of raF with a fallback to setTimeout for the game logic updates in case the frame rate is too low:
var maximalUpdateDelay = 25; // ms
var updateTimeout, now;
function animate() {
updateTimeout = setTimeout(animate, maximalUpdateDelay);
var delta = -now + (now = Date.now());
update(now, delta);
}
function main() {
clearTimeout(updateTimeout);
animate(); // update the scene
draw(); // render the scene
requestAnimationFrame(main);
}
main();
I'ld recommend taking a look at the HTML 5 - Game Development course on Udacity. I don't remember the implementation of this problem from the course (but there definitely was one), but my opinion from a gameplay perspective is that just using rAF (like your first bullet) is the most fun, even if there is game slow down due to too much processing needed on slower computers.
I think you're on the right track with the last one because it should give you the most consistency across devices running at different frame rates, but you definitely want to force your delta value down if it gets too high to avoid big jumps:
var animate = function () {
now = Date.now();
delta = now - last;
last = now;
if(delta > 20) {
delta = 20;
}
draw(delta);
requestAnimationFrame(animate);
};
Whenever I have a new buffer that come into my client I want to redraw that instance of audio onto my canvas. I took the sample code from http://webaudioapi.com/samples/visualizer/ and tried to alter it to fit my needs in a live environment. I seem to have something working because I do see the canvas updating when I call .draw() but it's not nearly as fast as it should be. I'm probably seeing about 1 fps as it is. How do I speed up my fps and still call draw for each instance of a new buffer?
Entire code:
https://github.com/grkblood13/web-audio-stream/tree/master/visualizer
Here's the portion calling .draw() for every buffer:
function playBuffer(audio, sampleRate) {
var source = context.createBufferSource();
var audioBuffer = context.createBuffer(1, audio.length , sampleRate);
source.buffer = audioBuffer;
audioBuffer.getChannelData(0).set(audio);
source.connect(analyser);
var visualizer = new Visualizer(analyser);
visualizer.analyser.connect(context.destination);
visualizer.draw(); // Draw new canvas for every new set of data
if (nextTime == 0) {
nextTime = context.currentTime + 0.05; /// add 50ms latency to work well across systems - tune this if you like
}
source.start(nextTime);
nextTime+=source.buffer.duration; // Make the next buffer wait the length of the last buffer before being played
}
And here's the .draw() method:
Visualizer.prototype.draw = function() {
function myDraw() {
this.analyser.smoothingTimeConstant = SMOOTHING;
this.analyser.fftSize = FFT_SIZE;
// Get the frequency data from the currently playing music
this.analyser.getByteFrequencyData(this.freqs);
this.analyser.getByteTimeDomainData(this.times);
var width = Math.floor(1/this.freqs.length, 10);
// Draw the time domain chart.
this.drawContext.fillStyle = 'black';
this.drawContext.fillRect(0, 0, WIDTH, HEIGHT);
for (var i = 0; i < this.analyser.frequencyBinCount; i++) {
var value = this.times[i];
var percent = value / 256;
var height = HEIGHT * percent;
var offset = HEIGHT - height - 1;
var barWidth = WIDTH/this.analyser.frequencyBinCount;
this.drawContext.fillStyle = 'green';
this.drawContext.fillRect(i * barWidth, offset, 1, 2);
}
}
requestAnimFrame(myDraw.bind(this));
}
Do you have a working demo? as you can easily debug this using the Timeline in chrome. You can find out what process takes long. Also please take unnecessary math out. Most of your code doesn't need to be executed every frame. Also, how many times is the draw function called from playBuffer? When you call play, on the end of that function it requests a new animation frame. If you call play every time you get a buffer, you get much more cycles of math->drawing->request frame. This also makes it very slow. If you are already using the requestanimationframe, you shall only call the play function once.
To fix up the multi frame issue:
window.animframe = requestAnimFrame(myDraw.bind(this));
And on your playBuffer:
if(!animframe) visualizer.draw();
This makes sure it only executes the play function when there is no request.
Do you have a live example demo? I'd like to run it through some profiling. You're trying to get updates for the playing audio, not just once per chunk, right?
I see a number of inefficiencies:
you're copying the data at least once more than necessary - you should have your scheduleBuffers() method create an AudioBuffer of the appropriate length, rather than an array that then needs to be converted.
If I understand your code logic, it's going to create a new Visualizer for every incoming chunk, although they use the same Analyser. I'm not sure you really want a new Visualizer every time - in fact, I think you probably don't.
-You're using a pretty big fftSize, which might be desirable for a frequency analysis, but at 2048/44100 you're sampling more than you need. Minor point, though.
-I'm not sure why you're doing a getByteFrequencyData at all.
-I think the extra closure may be causing memory leakage. This is one of the reasons I'd like to run it through the dev tools.
-You should move the barWidth definition outside of the loop, along with the snippet length:
var snippetLength = this.analyser.frequencyBinCount;
var barWidth = WIDTH/snippetLength;
if you can post a live demo that shows the 1fps behavior, or send an URL to me privately (cwilso at google or gmail), I'd be happy to take a look.
I am using the Web Audio API to display a visualization of the audio being played. I have an <audio> element that is controlling the playback, I then hook it up to the Web Audio API with by creating a MediaElementSource node from the <audio> element. That is then connected to a GainNode and an AnalyserNode. The AnalyserNode's smoothingTimeConstant is set to 0.6. The GainNode is then connected to the AudioContext.destination.
I then call my audio processing function: onAudioProcess(). That function will continually call itself using:
audioAnimation = requestAnimationFrame(onAudioProcess);
The function uses the AnalyserNode to getByteFrequencyData from the audio, then loops through the (now populated) Uint8Array and draws each frequency magnitude on the <canvas> element's 2d context. This all works fine.
My issue is that when you pause the <audio> element, my onAudioProcess function continues to loop (by requesting animation frames on itself) which is needlessly eating up CPU cycles. I can cancelAnimationFrame(audioAnimation) but that leaves the last-drawn frequencies on the canvas. I can resolve that by also calling clearRect on the canvas's 2d context, but it looks very odd compared to just letting the audio processing loop continue (which slowly lowers each bar to the bottom of the canvas because of the smoothingTimeConstant).
So what I ended up doing was setting a timeout when the <audio> is paused, prior to canceling the animation frame. Doing this I was able to save CPU cycles when no audio was playing AND I was still able to maintain the smooth lowering of the frequency bars drawn on the <canvas>.
MY QUESTION: How do I accurately calculate the number of milliseconds it takes for a frequency magnitude of 255 to hit 0 (the range is 0-255) based on the AnalyserNode's smoothingTimeConstant value so that I can properly set the timeout to cancel the animation frame?
Based on my reading of the spec, I'd think you'd figure it out like this:
var val = 255
, smooth = 0.6
, sampl = 48000
, i = 0
, ms;
for ( ; val > 0.001; i++ ){
val = ( val + val * smooth ) / 2;
}
ms = ( i / sampl * 1000 );
The problem is that with this kind of averaging, you never really get all the way down to zero - so the loop condition is kind of arbitrary. You can make that number smaller and as you'd expect, the value for ms gets larger.
Anyway, I could be completely off-base here. But a quick look through the actual Chromium source code seems to sort of confirm that this is how it works. Although I'll be the first to admit my C++ is pretty bad.