I am trying to create a water ripple effect in a video embedded in HTML5 default web player.
I am doing it fine with using images and and a overlay canvas on top of it, but what I am trying to do now is to get single frames from a video and output it to a canvas every 1-5ms using this tutorial.
And I am stuck at this point, I can output frame into another canvas using
canvas.toDataURL() function.
I have seen advanced web-based video players that allow for applying Processing.js sketches on top of videos, would that be a good solution?
My question is: what would be the best and most reliable solution for applying visual effects (water ripples in this case) using JavaScript to a video playing in HTML5 media player.
My question is: what would be the best and most reliable solution for applying visual effects (water ripples in this case) using JavaScript to a video playing in HTML5 media player.
In my opinion the best approach would be to use WebGL to create the effects, using the video as input texture and a simple flat geometry that is manipulated using a animated bump-map - or - directly manipulating the vertices - or - a perhaps a shader program, and then output the result to a canvas.
I would not place the video in DOM at all but create control buttons and display those together with the output [webgl-]canvas in DOM.
The obvious drawback would be besides from slightly more complex code, when used on computers which doesn't have a GPU (but that would be a drawback in any case and more so if you used a regular 2D canvas and pixel manipulation).
This is of course very broad in terms of code example. But I assume you get the general idea.
A couple of notes:
[...] what I am trying to do now is to get single frames from a video and output it to a canvas every 1-5ms [...]
This is sort of pointless in regards to the time-budget, as the typical monitor can only refresh the image every ~16.7ms which means you're wasting at least 3-4 frames that are never displayed.
Also, a typical video source never runs faster than 30 FPS (in the US, 25 FPS in Europe) which means there is only a new frame every 33.3ms (there are of course special case videos such as VR/AR, stereoscopic, games and "Mac" recorded video etc. that may use higher fps - but for the most part anything > 30 fps is usually wasted cycles) which allows for a higher time-budget for processing per frame.
I can output frame into another canvas using canvas.toDataURL() function [...]
Outch! :) This comes which a huge overhead. The browser would have to use line filtering (in case of PNG - well, some browsers skip this and use filter 0), compression, base-64 encoding, then apply the source to an image element, decode the base-64, decompress, defilter...
You can instead simply use the source canvas for drawImage() directly and only have to deal with a in-memory bitmap (super fast!).
All that being said: if simplicity is important code-wise, you can of course do all this using a 2D canvas. You can use drawImage() to displace pixels/blocks and therefor work on the GPU (but not necessarily faster than working on the bitmap directly depending on how you apply the actual displacement).
But there are still many caveats such as video source and destination resolution which has a exponential impact on performance, limited use of the GPU as you still would have to do multiple serial operations versus parallel operations with webgl/gpu and so forth. In essence, the performance will suffer compared to a webgl solution.
If you want to get high performance you can use WebGL. Following is a github reop for Water ripple project.
https://github.com/sirxemic/jquery.ripples/
Following is a ruining example for jQuery WebGL Ripples
http://sirxemic.github.io/jquery.ripples/
Think this might help
You can do it in several ways. You can write core javascript code with canvas.
But I think it is best to use Jquery plugins. There are several plugins available for water ripple.
You may check the following links:
https://github.com/virtyaluk/paper-ripple
https://github.com/sirxemic/jquery.ripples
https://github.com/andyvr/water-ripple
Related
There seems to be some differing opinions about this going from blog to blog, video to video, and forum to forum. This is specifically for the 2D context for the canvas tag, not WebGL. I know that WebGL would give me better performance but my goal is to understand how the canvas tag works in its 2D context.
I hear that pre-rendering your objects in a "virtual" offscreen canvas is probably the best for performance which makes sense seeing the browser wouldn't literally be drawing it. People then say to grab the data from that canvas using "getImageData" which from my understanding returns a base64 code that you can apply to a canvas that is added to the DOM using "putImageData". Wouldn't this be a huge performance hit?
Should I render the entire seen for the game on this virtual canvas and then put it to the visible one using this method inside of the loop?
Yes you can. I recommend using requestAnimationFrame() and inside the draw() part of your game loop, draw all separate objects to an offscreen canavs, and then do one get/put image to the visible canvas. While this may seem to be a performance hit, but when you have many objects being drawn, this overhead of get/put image is actually negligible.
The Canvas is just an image, and from my experience there is no difference between using a canvas or an image in terms of performance.
Offscreen or on the canvas is no different, they will both take advantage of the GPU wherever possible.
Using context.getImageData(), context.createImageData(), and context.putImageData() should be avoided for realtime rendering, It does not take advantage of the GPU and any processing you do to it will be done in main memory by Javascript. Though the data is stored in a typed array Uint8ClampedArray and can be converted to any type of typed array, such as a Uint32Array allowing you to handle a single pixel with one variable, rather than 4. There are also many native functions for typed arrays that provide much quicker array manipulation than the standard Javascript array.
The limiting factor for images (including canvas as image) is the amount of GPU RAM available, when you exceed the amount of RAM available the browser will start swapping images into GPU RAM as they are needed, when it does so this blocks the GPU's ability to render, and the transfer from Main RAM to the GPU RAM is slow in comparison to normal RAM access. When this happens you will instantly see a loss in frame rate. As there are a huge variety of platforms that the browsers can run on and no way to know the machines capabilities you should be careful when you publish realtime applications.
If you have written a game with high resolution images for a high end desktop machine, it will not perform very well on tablets and low end laptops. To mitigate this problem downsample the images to match the screen resolution. Using a hires background image on a device that is 1/8th the resolution is putting undue strain on the hardware. Devices are made to handle the resolution of their screens, going over this resolution will have a major unnecessary performance hit. This is where you can use an offscreen canvas to render the image at the native resolution of the device and then dump the original hires image. There will be no loss of quality, but a huge gain in performance, turning something that is unplayable into playable. This applies to all graphic resources. Never store and use images at a resolution higher then the device you are using can display.
Because there is such a variety of things you can do with the canvas the best way to find out what runs best is to experiment. Monitor the frame rate and try different approaches to the problem at hand. If the frame rate improves you have found a better way of doing thing, if the frame rate drops then you have used the wrong method.
What I'm looking is a flash alternative for mobile phones using html5.
I was looking into SVG and it seems the only way to get hardware acceleration is to use CSS transforms on it. But CSS transforms aren't enough, I want to animate the actual nodes that make up a vector (ie, points on a path) so I could get more sophisticated character animation. To do this I was looking at some gui based editors.
I checked what adobe has been up to and they seem to have killed Edge Animate and rebranded Flash as "Animate CC" for 2016.
http://blogs.adobe.com/creativecloud/update-about-edge-tools-and-services/
https://blogs.adobe.com/flashpro/welcome-adobe-animate-cc-a-new-era-for-flash-professional/
But reading up on "Animate CC" I see that it exports vector animations to either Canvas or WebGL. Which I think is due to them not getting hardware acceleration with native SVG via SMIL or using javascript.
https://css-tricks.com/guide-svg-animations-smil/
Another one is http://www.animatron.com which converts everything to canvas as well.
So my question is, in order to do keyframe animations on nodes within a vector path, a vector needs to be converted to either WebGL or Canvas for it to be hardware accelerated on mobile?
p.s I prefer using SVG as it's loaded in the DOM and I can manipulate things with jquery. This is for a mobile game that uses vectors (svg) as its base but I'd like to incorporate animations too - beyond the basic css transforms. I wish there was a way to have a .svg file that not only contains the vector information but also the animation info. so I could load this .svg file. and then in javascript go:
character1.play('animation1') or something. If SMIL worked fast I'm sure editors like adobe would make it as simple as that.
EDIT: I just read that Chrome 45 killed SMIL in favor of "web animations" and css.
https://developer.mozilla.org/en-US/docs/Web/SVG/SVG_animation_with_SMIL
And as Kaiido mentioned in the comments IE never supported smil so maybe that's why adobe never exported to it (?).
http://caniuse.com/#feat=svg-smil
also I never saw any examples online that show hardware accelerated path animation with smil, if any of you guys find a link pls let me know.
EDIT #2: I'm thinking of giving up my wishful thinking and instead looking at vector to canvas exporters like animatron.com. However, it doesn't seem like canvas is even hardware accelerated or fast like css3 transforms. I loaded some animations from animatron in my old iPhone 4s/iOS 8 and it's jittery and slow for example:
https://www.animatron.com/project/1953f3526e5b2ec4eef429c8
whereas css3 transform animations always run very smooth...
I still haven't tested vector to webgl.. but I think that's why adobe eventually chose to use it for their vector animations since canvas is slow and svg is limited.
EDIT #3: sure enough it seems like webgl is the way to go (unless someone finds a way to do this with native svg) http://www.yeahbutisitflash.com/?p=7231 .. this works fast in my iphone 4s/ios8.. I currently think this is the only way to do what I want: hardware accelerated vector based animation (however the graphics don't look as crisp as I'd want them.. webgl kinda messed with that I think).
but this is why I think Edge Animate got killed cause they were trying to create a tool that took advantage of css3 transforms, but ppl want to animate vector nodes so they went back to Flash and rebranded it. (another note: the above webgl anim doesn't work so well on my galaxy S4/kitkat android phone.. so this is mainly for newer devices/OSs)
EDIT #4: come to think of it. it'd be a pain to have multiple webgl contexts running in my program. so if I had 10 animated characters I'd have to have 10 webgl contexts which would be intense for a mobile device.. unless I chose to do the whole game in flash, and then I'd have one big webgl context after I export it. but I prefer to work in the dom. oh well css3 transforms for the meantime.. :/
EDIT #5 - Dec 2016: I'm now using svg/javascript with snap.svg. modern phones seem fast enough.
Other Useful Links I Found:
http://www.crmarsh.com/svg-performance/
Canvas is (as far as i know) software accelerated. So it's rendered by the processor (CPU). The processor (cause of them pixels) ain't that good at graphical stuff but for simple things it would be enough. And it runs everywhere, where there is a processor.
If you want to have better performance on hardware accelerated devices which most modern smartphones are, you need webgl. But you can export your stuff in webgl from adobe CC. Older smartphones are not very optimized on hardware acceleration so please check with your target group what devices they have and try to run your app on one of the slowest devices.
I would not use SVG. SVG is even worse than DOM. You can be faster manipulating HTML in javascript than SVG. I don't know why but it is damn slow. SVG is just an alternative if you want to have scalable graphics or charts and that's what it is made for. To animate in SVG is a pain. Don't do it. It is not optimized for animation.
CSS-Transform is a prototype-like and will not help you with keyframe animation. But it has potential to have an eye on it.
Does this help you?
This library/plugin might be exactly what you are looking for: Greensock SVG Morph Plugin. It seems to perform pretty well on mobile, not sure how well it performs on devices mentioned by you.
It's also not GPU accelerated but Greensock Animation Platform seems to perform extremely well even on mobile devices.
I think its a difficult question to answer as there's so many issues with different browsers. Some don't support SMIL transforms well (or being deprecated, but as mentioned there are fills for it), some don't support CSS3 transforms on SVG elements at all, so most of the 'solutions' out there, have some issue that may need to be compromised on. I think one browser doesn't support d attribute morph properly, but I can't recall which (so test this early on with required browsers if you do go down this route).
Canvas does typically seem to perform better on mobile for most animation I've seen or played with, certainly as the number of objects increases on page.
Other alternatives to webGL that was mentioned..
One option that kinda springs to mind is fabric.js, which is a canvas approach to SVG. It will take the same commands, elements as SVG, but display it on an HTML5 canvas. Turn off drag/freetransform within it (as I think its slowed down a bit with this if it needs extra checks), and I think it will be a bit faster, but its a while since I've played. This could be an interesting approach if you don't need specific DOM element access, but would be fine if you are ok with a similar object based setup.
If you insist on using SVG, then I would look at integrating it with another library (or even 2). Snap or SVG.js are both decent, but can be a bit slow. However, I would experiment with using Velocity.js or React.js with that svg library, as they have some methods that can squeeze out a bit more performance. Also GSAP may be worth a look.
There is no general answer to this, it depends on the browser implementation and your hardware, but ... for the long term solution I would bet on WebGL because it uses the GPU and will ultimately dominate.
WebGL is not well supported right now (2016), and it potentially comes with security issues.
Also see: https://css-tricks.com/rendering-svg-paths-in-webgl/
Canvas is better at working on raw 2d pixel buffers (+ alpha channel) and is slow for higher resolution+animation.
You could use some dual lib like Pixi though.
If you don't mind the k-weight increasing a bit and are comfortable using flash, the easiest and the fastest solution would be to use Animate CC (It's exactly Flash, without the symbol filters and gives canvas output instead of swf).
I want to write an animated game which needs a lot of animations running all the time, something like rolling the cards/images in slot machine. I am wondering if I should mainly use just pure GWT/Javascript to run the animation, or use HTML5 Canvas or HTML5 animation to achieve this.
I know I might be able to make more fancy animated motion or graphics if I use HTML5 canvas, but I also cares about the performance more because there are several "animations" running in the same time just like rolling so many images in 5 columns slot machine. In this case, I think performance is most important.
I heard that drawing things in Canvas is really expensive. So I am not sure if I keep rolling images in Canvas is expensive as well.
What do you think which technique I should use to write a game like slot machine? HTML5-Canvas, pure JS/GWT-animation, or HTML5-animation?
I need to use some audio for the games, but I am pretty new to this, I do not know what libraries or technique I can use for audio. Please give me some advice.
Thanks.
if you have a lot of graphical animations, Canvas 2D will be a good choice. If you are dealing with vector shapes, simple animations SVG might be an option. I would not go for GWT as the abstraction layer of Java might hinder your ability to use full power of HTML/CSS/JS.
Ensure you use a basic game library if you choose plain JS and Canvas. I used jawsjs which is ok for simple games. See and observe the performance of below game which is based on Canvas 2D if you need a reference. Let me know if you need the code of it as a reference.
http://99challenge.com/_index.html
On audio, there are few challenges. Firefox behaves different to all other browsers, and has quite instable audio support. You might need to look for Flash support in getting audio to work properly on FF.
I'm trying to create moving lights with trails for an HTML5 website/app targeted at iPad 2.
I wonder what the best way to do this is and whether using HTML5 is viable at all. I chose HTML5 because it's easier and cheaper to develop and deploy than native iOS apps with Objective C. Of course if it turns out that HTML5 simply doesn't offer enough performance I might have to swallow the bitter pill.
Anyway to give you an impression what I'm talking about, this is what I got so far:
screenshot http://devdali.no-ip.org/mathias/test-lights/screenshots/1.jpg
Or you can see it in action here (only works in webkit based browsers).
At first I tried using HTML5 canvas and drawing radial gradients as particles in similar manner you see above. It worked but the framerate was horrible even on my desktop computer!
So after a bit of reading I found out that CSS3 transforms may be hardware accelerated, so I build the version you see above. Every "particle" is a 64x64 png image. For each light there is the "head" light (one img) followed by a trail consisting of 115 img elements. Each img element is transformed using "translate3d" (as well as scale and rotation). Also the opacity of each element is adjusted dynamically.
Doing it this way provided much better framerates on my computer, but I doubt the iPad 2 will handle it.
I'd be grateful if anyone could give me some hints on how to improve the performance of this in general and considering the target platform.
Thanks for any help in advance!
If you accept small changes to the effect, some other procedure may work fast:
Instead of drawing the light's trails by the means of many particles, just draw the lights in their current positions in a Canvas element.
You can then darken the whole image at the beginning of a frame by filling a black rectangle with a very low opacity on top. This way the trails fade into dark, but would not alter their color like they do now.
The amount of drawing operations however will reduce vastly. The most costly operation would be filling the fading rectangle for every frame.
This should be built in the canvas. Check out EaselJS and this demo.
http://easeljs.com/
http://easeljs.com/demos/MusicVisualizer/index.html
You could optimize performances a LOT by using WebGL(, which is supported on the iPad2.)... which is not supported for basic html pages on ios safari as stated Nison Maël...
For the time being you only have canvas as a solution. Which will still give you better performances...
(You can check this blog for more info:
http://learningwebgl.com/blog/
With a little faith and time you'll be amazed!)
I'm interested in making a game using html and javascript. I was wondering if it really is that much faster drawing in html5 and javascript than it is with images and div's in html and javascript.
Example of a game using html and javascript that works nicely:
http://scrabb.ly/
Example of a game using html5 and javascript that works nicely:
http://htmlchess.sourceforge.net/demo/example.html
I've run a bunch of numbers on HTML-made drawing versus Canvas-made drawing. I could make a huge post about the benefits of each, but I will give some of the relevant results of my tests to consider for your specific application:
I made Canvas and HTML test pages, both had movable "nodes." Canvas nodes were objects I created and kept track of in Javascript. HTML nodes were <div>s, though they could be <image> or <video> too.
I added 100,000 nodes to each of my two tests. They performed quite differently:
The HTML test tab took forever to load (timed at slightly under 5 minutes, chrome asked to kill the page the first time). Chrome's task manager says that tab is taking up 168MB. It takes up 12-13% CPU time when I am looking at it, 0% when I am not looking.
The Canvas tab loaded in one second and takes up 30MB. It also takes up 13% of CPU time all of the time, regardless of whether or not one is looking at it.
Dragging on the HTML page is smoother, which I suppose is expected, since the current setup is to redraw EVERYTHING every 30 milliseconds in the Canvas test. There are plenty of optimizations to be had for Canvas for this. (canvas invalidation being the easiest, also clipping regions, selective redrawing, etc.. just depends on how much you feel like implementing)
Video on the HTML page, while I am not moving objects, is actually perfectly smooth.
On canvas the video is always slow, since I am redrawing constantly because I turned off my drawing canvas invalidation. There is of course plenty of room for improvement.
Drawing/loading alone is far faster in Canvas and has far more room for optimizations, too (ie, excluding things that are off-screen is very easy).
Fast as in faster rendering or faster development? I would say the answer to both is HTML5 canvas. Although it is a fairly new technology, and not even supported by all mainstream browsers yet, it already has much more functionality than you would have using DIVs with normal HTML. I've done drawing with divs before and it was incredibly frustrating just getting something to work. With canvas you already have a framework in place to do most basic drawing. Furthermore, html5 is new. Even if it is relatively slower than drawing with divs right now (which it probably isn't), that performance will increase as development and adoption increases. I can't say the same for drawing with divs.
Pros to using HTML5 Canvas:
Similar to other drawing frameworks (OpenGL, DirectX)
Will continue to increase in performance and functionality
May become hardware accelerated in the future
Possible 3D framework in the future
Neither of those games requires HTML 5. scrabb.ly does everything with rectangular objects, which divs handle just fine, and the chess game doesn't even use animation. If that's the kind of game you're thinking of building, then what you use should be decided on the grounds of familiarity and compatibility rather than performance.