There seems to be some differing opinions about this going from blog to blog, video to video, and forum to forum. This is specifically for the 2D context for the canvas tag, not WebGL. I know that WebGL would give me better performance but my goal is to understand how the canvas tag works in its 2D context.
I hear that pre-rendering your objects in a "virtual" offscreen canvas is probably the best for performance which makes sense seeing the browser wouldn't literally be drawing it. People then say to grab the data from that canvas using "getImageData" which from my understanding returns a base64 code that you can apply to a canvas that is added to the DOM using "putImageData". Wouldn't this be a huge performance hit?
Should I render the entire seen for the game on this virtual canvas and then put it to the visible one using this method inside of the loop?
Yes you can. I recommend using requestAnimationFrame() and inside the draw() part of your game loop, draw all separate objects to an offscreen canavs, and then do one get/put image to the visible canvas. While this may seem to be a performance hit, but when you have many objects being drawn, this overhead of get/put image is actually negligible.
The Canvas is just an image, and from my experience there is no difference between using a canvas or an image in terms of performance.
Offscreen or on the canvas is no different, they will both take advantage of the GPU wherever possible.
Using context.getImageData(), context.createImageData(), and context.putImageData() should be avoided for realtime rendering, It does not take advantage of the GPU and any processing you do to it will be done in main memory by Javascript. Though the data is stored in a typed array Uint8ClampedArray and can be converted to any type of typed array, such as a Uint32Array allowing you to handle a single pixel with one variable, rather than 4. There are also many native functions for typed arrays that provide much quicker array manipulation than the standard Javascript array.
The limiting factor for images (including canvas as image) is the amount of GPU RAM available, when you exceed the amount of RAM available the browser will start swapping images into GPU RAM as they are needed, when it does so this blocks the GPU's ability to render, and the transfer from Main RAM to the GPU RAM is slow in comparison to normal RAM access. When this happens you will instantly see a loss in frame rate. As there are a huge variety of platforms that the browsers can run on and no way to know the machines capabilities you should be careful when you publish realtime applications.
If you have written a game with high resolution images for a high end desktop machine, it will not perform very well on tablets and low end laptops. To mitigate this problem downsample the images to match the screen resolution. Using a hires background image on a device that is 1/8th the resolution is putting undue strain on the hardware. Devices are made to handle the resolution of their screens, going over this resolution will have a major unnecessary performance hit. This is where you can use an offscreen canvas to render the image at the native resolution of the device and then dump the original hires image. There will be no loss of quality, but a huge gain in performance, turning something that is unplayable into playable. This applies to all graphic resources. Never store and use images at a resolution higher then the device you are using can display.
Because there is such a variety of things you can do with the canvas the best way to find out what runs best is to experiment. Monitor the frame rate and try different approaches to the problem at hand. If the frame rate improves you have found a better way of doing thing, if the frame rate drops then you have used the wrong method.
Related
I am trying to create a water ripple effect in a video embedded in HTML5 default web player.
I am doing it fine with using images and and a overlay canvas on top of it, but what I am trying to do now is to get single frames from a video and output it to a canvas every 1-5ms using this tutorial.
And I am stuck at this point, I can output frame into another canvas using
canvas.toDataURL() function.
I have seen advanced web-based video players that allow for applying Processing.js sketches on top of videos, would that be a good solution?
My question is: what would be the best and most reliable solution for applying visual effects (water ripples in this case) using JavaScript to a video playing in HTML5 media player.
My question is: what would be the best and most reliable solution for applying visual effects (water ripples in this case) using JavaScript to a video playing in HTML5 media player.
In my opinion the best approach would be to use WebGL to create the effects, using the video as input texture and a simple flat geometry that is manipulated using a animated bump-map - or - directly manipulating the vertices - or - a perhaps a shader program, and then output the result to a canvas.
I would not place the video in DOM at all but create control buttons and display those together with the output [webgl-]canvas in DOM.
The obvious drawback would be besides from slightly more complex code, when used on computers which doesn't have a GPU (but that would be a drawback in any case and more so if you used a regular 2D canvas and pixel manipulation).
This is of course very broad in terms of code example. But I assume you get the general idea.
A couple of notes:
[...] what I am trying to do now is to get single frames from a video and output it to a canvas every 1-5ms [...]
This is sort of pointless in regards to the time-budget, as the typical monitor can only refresh the image every ~16.7ms which means you're wasting at least 3-4 frames that are never displayed.
Also, a typical video source never runs faster than 30 FPS (in the US, 25 FPS in Europe) which means there is only a new frame every 33.3ms (there are of course special case videos such as VR/AR, stereoscopic, games and "Mac" recorded video etc. that may use higher fps - but for the most part anything > 30 fps is usually wasted cycles) which allows for a higher time-budget for processing per frame.
I can output frame into another canvas using canvas.toDataURL() function [...]
Outch! :) This comes which a huge overhead. The browser would have to use line filtering (in case of PNG - well, some browsers skip this and use filter 0), compression, base-64 encoding, then apply the source to an image element, decode the base-64, decompress, defilter...
You can instead simply use the source canvas for drawImage() directly and only have to deal with a in-memory bitmap (super fast!).
All that being said: if simplicity is important code-wise, you can of course do all this using a 2D canvas. You can use drawImage() to displace pixels/blocks and therefor work on the GPU (but not necessarily faster than working on the bitmap directly depending on how you apply the actual displacement).
But there are still many caveats such as video source and destination resolution which has a exponential impact on performance, limited use of the GPU as you still would have to do multiple serial operations versus parallel operations with webgl/gpu and so forth. In essence, the performance will suffer compared to a webgl solution.
If you want to get high performance you can use WebGL. Following is a github reop for Water ripple project.
https://github.com/sirxemic/jquery.ripples/
Following is a ruining example for jQuery WebGL Ripples
http://sirxemic.github.io/jquery.ripples/
Think this might help
You can do it in several ways. You can write core javascript code with canvas.
But I think it is best to use Jquery plugins. There are several plugins available for water ripple.
You may check the following links:
https://github.com/virtyaluk/paper-ripple
https://github.com/sirxemic/jquery.ripples
https://github.com/andyvr/water-ripple
I'm coding a JavaScript game. This game obviously needs to be constantly rendering a screen, which, using canvas, must be an Uint8Array(width * height * 4) (as used by Canvas's ImageData). In order to test the expected FPS, I've tried filling that array with white. Much to my surprise, the performance was mediocre. I could barely fill a 1000x1000 bitmap with white in a high-end computer at 150 fps. Considering this is the best performance, without any game logic running, the end result will probably be much lower. My question is: why is the performance so low and what can I do to improve it?
Figuring out how many times you can fill a 1000x1000 canvas using putImageData will not give you any kind of realistic results. The reason is the graphics are pipelined. In a normal app you'd only call putImageData once a frame. If you call it more than once a frame at some point you'll fill the pipeline and stall it. In a real app though you'd be manipulating your data most the frame and only uploading it once not stalling the pipeline.
If you want to see how much work you can do, make a 1000x1000 canvas, call getImageData on it to get an ImageData, manipulate that image data a certain amount, call putImageData, then call requestAnimationFrame and do it again. Slowly increase the amount of manipulation you do until it starts running slower than 60fps. That will tell you how much work you can do realistically.
Here's a fiddle that tries this out
http://jsfiddle.net/greggman/TVA34/
Also, using putImageData has a few issues. One is that Canvas requires pre-multiplied alpha but putImageData supplies un-premultiplied alpha which means some process has to convert your data to pre-multiplied alpha. Then, now-a-days, most Canvas implementations are GPU accelerated. That's great for nearly every feature of the Canvas API but it sucks for putImageData since that data has to be transferred from the CPU to the GPU. It's even worse for getImageData since copying data from the GPU back to the CPU generally stalls the GPU. The stories is even more bad on an HD-DPI machine. getImageData and putImageData have to convert from CSS pixels to the resolution that's actually being used.
If you use WebGL you can at least skip the pre-multiplied alpha conversion step.
Here's a WebGL version
http://jsfiddle.net/greggman/XLgs6/
Interestingly on my 2012 Macbook Pro Retina I find the canvas version is faster on Chrome and Safari. I'm curious why since I would not expect that having worked on them.
/*
Canvas WebGL
Chrome 32 : 710k 650k numbers are in 'operations per frame`
Firefox 26 : 80k 190k
Safari 7.0.1 : 150k 120k
*/
My test also might not be valid. Manipulating only 710k pixels (of 1000k pixels) seems pretty slow. Maybe one of the functions like Math.random or Math.floor is particularly slow, especially given they're using doubles.
See these 2 tests
Only doing loop and assign to the same address
http://jsperf.com/variable-assign
Good old array expansion (and also another popular for loop trick that suppose to be faster)
http://jsperf.com/fill-an-type-array-expand/3
First test shows the address look up is taking about 3/4 of the time.
Second test shows the for loop is taking more than 30% of the time.
I think what typed array is really in lack of is some native code that does the block copy that would really be used in a game development (rather than set pixel by pixel in the tests). WebGL might worth considering.
I am in process of making a game where the health bar (animated) and some other info represented visually like some icons showing the number of bombs the player has etc. Now, this can be done both in canvas (by making another canvas for info that sits over the main canvas, or it can be done using many divs and spans with absolute positioning. This is my first time in making a browser based game so if any experienced people view this, tell me what you recommend. I would like to know that which method would be faster.
The game will also be running on mobile devices. Thanks!
There is no straighforward answer and I suggest you do FPS testing with different browser how it plays out for your use case. If you do not wish to go such in-depth I suggest you simply draw the elements inside canvas and if you need to hide them then leave out drawHUD() call from your rendering loop.
For HTML HUD overlay on <canvas> the following factors should be considered
Can the web browser compositor do hardware accelerated <canvas> properly if there are DOM elements upon the canvas
HTML / DOM manipulation will be always slower than <canvas> operations due to inherited complexity dealing with DOM elements
<canvas> pixel space stays inside <canvas> and it might be difficult to have pixel-perfect aligment if you try to draw elements on <canvas> outside the canvas itself
HTML offers much more formatting options for text than canvas drawString() - is HTML formatting necessary
Use the canvas. Use two canvases if you want, one overlaid over the other, but use the canvas.
Touching the DOM at all is slow. Making the document redo its layout because the size of DOM elements moved is very slow. Dealing with the canceling (or not) of even more events because there are DOM items physically on top of the canvas can be a pain and why bother dealing with that?
If your HUD does not update very often then the fastest thing to do would be drawing it to an in-memory canvas when it changes, and then always drawing that canvas to the main canvas when you update the frame. In that way your drawHud method will look exactly like this:
function drawHUD() {
// This is what gets called every frame
// one call to drawImage = simple and fast
ctx.drawImage(inMemoryCanvas, 0, 0);
}
and of course updating the HUD information would be like:
function updateHUD() {
// This is only called if information in the HUD changes
inMemCtx.clearRect(0, 0, width, height);
inMemCtx.fillRect(blah);
inMemCtx.drawImage(SomeHudImage, x, y);
var textToDraw = "Actually text is really slow and if there's" +
"often repeated lines of text in your game you should be" +
"caching them to images instead";
inMemCtx.fillText(textToDraw, x, y);
}
Since HUDs often contain text I really do urge caching it if you're using any. More on text performance here.
As others have said, there is no universally best approach, as it depends on the specifics of what you need to render, how often, and possibly what messaging needs to happen between graphical components.
While it is true the DOM reflows are expensive, this blanket warning is not always applicable. For instance, using position:fixed; elements avoids triggering reflows for the page (not necessarily within the element if there are non-fixed children). Repaint is (correct me if this is wrong) expensive because it is pixel pushing, and so is not intrinsically slower than pushing the same number of pixels to a canvas. It can be faster for some things. What's more, each has certain operations that have performance advantages over the other.
Here are some points to consider:
It's increasingly possible to use WebGL-accelerated canvas elements on many A-grade browsers. This works fine for 2D, with the advantage that drawing operations are sent to the GPU, which is MUCH faster than the 2D context. However this may not be available on some target platforms (e.g., at the time of this writing, it is available in iOS Safari but not in the iOS UIWebView used if you target hybrid mobile applications.) Using a library to wrap canvas can abstract this and use WebGL if its available. Take a look at pixi.js.
Conversely, the DOM has CSS3 animations/transitions which are typically hardware-accelerated by the GPU automatically (with no reliance on WebGL). Depending on the type of animation, you can often get much faster results this way than with canvas, and often with simpler code.
Ultimately, as a rule in software performance, understanding the algorithms used is critical. That is, regardless of which approach used, how are you scheduling animation frames? Have you looked in a profiler to see what things take the most time? This practice is excellent for understanding what is impacting performance.
I've been working on an app with multiple animations, and have implemented each component both as DOM and canvas. I was initially surprised that the DOM version was higher performant than the canvas (wrapped with KineticJS) version, though I know see that this was because all the animated elements were position:fixed and using CSS (under the hood via jQuery UI), thereby getting GPU performance. However the code to manage these elements felt clunky (in my case, ymmv). Using a canvas approach allows more pixel-perfect rendering, but then it loses the ability to style with CSS (which technically allows pixel-perfect rendering as well but may be more or less complex to achieve).
I achieved a big speed up by throttling the most complex animation to a lower framerate, which for my case is indistinguishable from the 60fps version but runs smooth as butter on an older iPad 2. Throttling required using requestAnimationFrame and clamping calls to be no more often than the desired framerate. This would be hard to do with CSS animations on the DOM (though again, these are intrinsically faster for many things). The next thing I'm looking at is syncing multiple canvas-based components to the same requestAnimationFrame loop (possibly independently throttled, or a round-robin approach where each component gets a set fraction of the framerate, which may work okay for 2-3 elements. (Incidentally, I have some GUI controls like sliders that are not locked to any framerate as they are should be as close to 60fps as possible and are small/simple enough that I haven't seen performance issues with them).
I also achieved a huge speed boost by profiling and seeing that one class in my code that had nothing to do with the GUI was having a specific method called very often to calculate a property. The class in question was immutable, so I changed the method to memoize the value and saw the CPU usage drop in half. Thanks Chrome DevTools and the flame chart! Always profile.
Most of the time, the number of pixels being updated will tend to be the biggest bottleneck, though if you can do it on the GPU you have effectively regained all the CPU for your code. DOM reflows should be avoided, but this does not mean avoid the DOM. Some elements are far simpler to render using the DOM (e.g. text!) and may be optimized by the browser's (or OS's) native code more than canvas. Finally, if you can get acceptable performance for a given component using either approach (DOM or canvas), use the one that makes the code simplest for managing that type of component.
Best advice is to try isolated portions in the different approaches, run with a profiler, use techniques to over-draw or otherwise push the limits to see which approach can run fastest, and do NOT optimize before you have to. The caveat to this rule is the question you are asking: how do I know in advance which technical approach is going to allow the best performance? If you pick one based on assuming the answer, you are basically prematurely optimizing and will live with the arbitrary pain this causes. If instead you are picking by rapid prototyping or (even better) controlled experiments that focus on the needs of your application, you are doing R&D :)
Browserquest displays their HUD using HTML elements, which has the benefit that you don't have to worry about redrawing etc. (and the performance will be pretty good, given that the entire browser engine is optimized to render the DOM pretty fast.
They (browserquest) also use several layered canvas elements for different game elements. I don't know the exact structure, but I guess that on which canvas an element is displayed depends on how often it needs to be redrawn.
I am developing a game for html5 canvas while mainly targeting mobile devices. The canvas is resized to the biggest available resolution, so that it almost makes a fullscreen game.
On an ipad that would be a 1024x786 canvas; at a resolution like this I notice a significant drop in framerate. On smaller resolution like 480x320 on iphone the game runs smooth! I guess this is because the device is fillrate limited.
Anyhow I would like to optimize as much as possible. I would be very grateful if you could post any general performance tips that you have for html5 canvas development.
Also see this canvas performance optimization article on html5rocks, which contains jsperf data which includes mobile browsers.
You can read Making an iPad HTML5 App & making it really fast by Thomas Fuchs
The key points he make:
Images slow things down immensely– get rid of them
Avoid text-shadow & box-shadow
Hardware-acceleration is quite new… and buggy (concurrent animations is limited)
Avoid opacity if possible (sometimes interferes with hardware-accelerated rendering)
Use translate3d, not translate (the latter is not hard-accelerated)
Some other points that can improve performance significantly:
use getImageData as infrequently as possible (major slowdown) [2]
combine more than one canvas in order to repaint smaller parts that are changing more frequently
You can also benchmark your app with Chrome/Firebug Profile can show you which functions are slow.
[2] http://ajaxian.com/archives/canvas-image-data-optimization-tip
A couple of more links:
HTML5 Games 0.2: Integers are Your Friends
HTML5 Canvas Performance in deviantART muro
Game Engine Listing. See esp. "Other" for extra links.
It's hard to give any specific tips as I'm not familiar enough with your game. You might want to split up rendering to multiple layers, though. This makes sense particularly if you have some static elements there. This way you can avoid some clearing and end up with nicer code overall.
I'm interested in making a game using html and javascript. I was wondering if it really is that much faster drawing in html5 and javascript than it is with images and div's in html and javascript.
Example of a game using html and javascript that works nicely:
http://scrabb.ly/
Example of a game using html5 and javascript that works nicely:
http://htmlchess.sourceforge.net/demo/example.html
I've run a bunch of numbers on HTML-made drawing versus Canvas-made drawing. I could make a huge post about the benefits of each, but I will give some of the relevant results of my tests to consider for your specific application:
I made Canvas and HTML test pages, both had movable "nodes." Canvas nodes were objects I created and kept track of in Javascript. HTML nodes were <div>s, though they could be <image> or <video> too.
I added 100,000 nodes to each of my two tests. They performed quite differently:
The HTML test tab took forever to load (timed at slightly under 5 minutes, chrome asked to kill the page the first time). Chrome's task manager says that tab is taking up 168MB. It takes up 12-13% CPU time when I am looking at it, 0% when I am not looking.
The Canvas tab loaded in one second and takes up 30MB. It also takes up 13% of CPU time all of the time, regardless of whether or not one is looking at it.
Dragging on the HTML page is smoother, which I suppose is expected, since the current setup is to redraw EVERYTHING every 30 milliseconds in the Canvas test. There are plenty of optimizations to be had for Canvas for this. (canvas invalidation being the easiest, also clipping regions, selective redrawing, etc.. just depends on how much you feel like implementing)
Video on the HTML page, while I am not moving objects, is actually perfectly smooth.
On canvas the video is always slow, since I am redrawing constantly because I turned off my drawing canvas invalidation. There is of course plenty of room for improvement.
Drawing/loading alone is far faster in Canvas and has far more room for optimizations, too (ie, excluding things that are off-screen is very easy).
Fast as in faster rendering or faster development? I would say the answer to both is HTML5 canvas. Although it is a fairly new technology, and not even supported by all mainstream browsers yet, it already has much more functionality than you would have using DIVs with normal HTML. I've done drawing with divs before and it was incredibly frustrating just getting something to work. With canvas you already have a framework in place to do most basic drawing. Furthermore, html5 is new. Even if it is relatively slower than drawing with divs right now (which it probably isn't), that performance will increase as development and adoption increases. I can't say the same for drawing with divs.
Pros to using HTML5 Canvas:
Similar to other drawing frameworks (OpenGL, DirectX)
Will continue to increase in performance and functionality
May become hardware accelerated in the future
Possible 3D framework in the future
Neither of those games requires HTML 5. scrabb.ly does everything with rectangular objects, which divs handle just fine, and the chess game doesn't even use animation. If that's the kind of game you're thinking of building, then what you use should be decided on the grounds of familiarity and compatibility rather than performance.