I'm noob at WebGL. I read in several posts of ND-Buffers and G-Buffers as if it were a strategic choice for WebGL development.
How are ND-Buffers and G-Buffers related to rendering pipelines? Are ND-Buffers used only in forward-rendering and G-Buffers only in deferred-rendering?
A JavaScript code example how to implement both would be useful for me to understand the difference.
G-Buffers are just a set of buffers generally used in deferred rendering.
Wikipedia gives a good example of the kind of data often found in a g-buffer
Diffuse color info
World space or screen space normals
Depth buffer / Z-Buffer
The combination of those 3 buffers is referred to as a "g-buffer"
Generating those 3 buffers from geometry and material data you can then run a shader to combine them to generate the final image.
What actually goes into a g-buffer is up to the particular engine/renderer. For example one of Unity3D's deferred renders contains diffuse color, occlusion, specular color, roughness, normal, depth, stencil, emission, lighting, lightmap, reflection probs.
An ND buffer just stands for "normal depth buffer" which makes it a subset of what's usually found in a typical g-buffer.
As for a sample that's arguably too big for SO but there's an article about deferred rendering in WebGL on MDN
Choosing a rendering path is a major architectural decision for a 3D renderer, no matter what API does it use. That choice's heavily depends upon the set of features the renderer has to support and it's performance requirements.
A substantial set of said features consists of so-called screen-space effects. If means that we render some crucial data about each pixel of the screen to a set of renderbuffers and then using that data (not the geometry) to compute some new data needed for a frame. Ambient Occlusion is a great example of such an effect. Based on some spacial values of pixels we compute a "mask" which we can later use to properly shade each pixel.
Moreover, there is a rendering pass which almost exclusively relies on screen-space computations. And it is indeed Deferred Shading. And that's where G-buffer come in. All data needed to compute colour of a pixel are rendered to a G-buffer: a set of renderbuffers storing that data. The data it self (and hence meanings of G-buffer's renderbuffers) can be different: diffuse component, specular component, shininess, normal, position, depth, etc. And as part of rendering of a frame contemporary deferred shading engines use screen-space ambient occlusion (SSAO), which use data from several G-buffer's renderbuffers (usually, they are position, normal and depth).
About ND-buffers. It seems to me that it's not a widely used term (Google failed to find any relevant info on them besides this question). I believe that ND stands for Normal-Depth. They're just a specific case of a G-buffer for a particular algorithm and effect (in the thesis it's SSAO).
So using G-buffers (and ND-buffers as a subset of G-buffers) and exadepends upon shading algorithms and effects you're implementing. But all screen-space computation will require some form of G-buffer.
P.S. The thesis you've link contains an inaccuracy. Author lists an ability to implement to ND-buffers on GLES 2.0 as an advantage to the method. However it's not actually possible since GLES 2.0 doesn't have depth textures (they've been added in OES_depth_texture extension).
I would like to add some more informations to previvous answers.
I read in several posts of ND-Buffers and G-Buffers as if it were a
strategic choice for WebGL development.
One of the most important part of deferred rendering is, if given platform supports MRT (multiple render targets). If it doesn't, you are not able to share partial calculations in shaders between each rendering and it also forces you to run rendnering as many times as you have "layers" (in case of unity 3D, it might be up to 11 times?). This could slow down your program a lot.
Read more in this question
Is deferred rendering/shading possible with OpenGL ES 2.0 ?
Webgl doesn't support MRT, but it has extension:
https://www.khronos.org/registry/webgl/extensions/WEBGL_draw_buffers/
Also there is an extension for depth textures:
https://www.khronos.org/registry/webgl/extensions/WEBGL_depth_texture/
So it should be possible to use deferred rendering technique, but its speed is hard to guess.
Related
So far I seen so many discussion on this topic and using different approaches to achieve this (https://github.com/tensorflow/models/issues/1809) but I want to know if anyone managed to successfully use Tensorflowjs to achieve this.
I know some also achieved this using transfer learning but it is not same as being able to add my own new class.
The short answer: No, not yet, though technically possible, I have not seen an implementation of this in the wild.
The longer answer - why:
Given that "transfer learning" essentially means reusing the existing knowledge in a trained model to help you then classify things of a similar nature without having to redo all the prior learning there are actually 2 ways to do that:
1) This is the easier route but may not be possible for some use cases: Use one of the high level layers of the frozen model that you have access to (eg the models that are released by TF.js are frozen models I believe - the ones on GitHub). This allows you to reuse some of its lower layers (or final output) which may already be good at picking out certain features that are useful for the use case you need eg object detection in a general sense, which you can then feed into your own unfrozen layers that sit on top of that output you are sampling from (which is where the new training would happen). This is faster as you are only updating weights etc for the new layers you have added, however because the original model is frozen, it means you would have to replicate in TF.js the layers you were bypassing to ensure you have the same resulting model architecture for COCO-SSD in this case if you wanted the architecture. This may not be trivial to do.
2) Retraining the original model - can think of tuning the original model - but this is only possible if you have access to the original unfrozen model and the data used to train that. This would take longer as you are essentially retraining the whole model on all the data + your new data. If you do not have the original unfrozen model, then the only way to do this would be to implement the said model in TF.js yourself using the layers / ops APIs as needed and then use that to train on your own data.
What?!
So an easier to visualize example of this is if we consider PoseNet - the one that estimates where human joints/skeletons are.
Now in this Posenet example imagine you wanted to make a new ML model that could detect when a person is in a certain position - eg waving a hand.
In this example you could use method 1 to simply take the output of existing posenet predictions for all the joints it has detected and feed that into a new layer - something simple like a multi layered perceptron - that could then very quickly learn from example data when a hand was in a waving position for example. In this case we are simply adding to the existing architecture to achieve a new result - gesture prediction vs the raw x-y point predictions for the joints themselves.
Now consider case 2 for PoseNet - you want to be able to recognise a new part of the body that it currently does not. For that to happen you would need to retrain the original model so that it could learn to predict that new body part as part of its output.
This is much harder as you would need to retrain the base model to do this, which means you need to have access to the unfrozen model to do that. If you didn't have access to the unfrozen model then you would have no choice but attempt to recreate PoseNet architecture entirely yourself and then train that with your own data. As you can see this 2nd use case is much harder and more involved to do.
I have somewhere between 2M and 10M static objects which I would like overlay on Google Maps. I've previously tried HeatmapLayer successfully on much smaller sets. Due to the shear volume I'm a bit concerned, and I must to lump the objects together to avoid performance problems. The target platform is Chrome on a standard desktop.
What is the best way to space partition and merge objects in close proximity? Should I try some type of loose quad tree to lump the objects together, and then display each node with its respective weight using the HeatmapLayer? Or should I try to dynamically build some type of triangle mesh where vertices can be dynamically merged and triangles gain weight as more objects are added to them and then display the triangles on top of Google Maps? HeatmapLayer is pretty fast (looks like it's implemented in GL shaders), but I doubt Polygon is.
I've tried searching for open source loose quad tree JavaScript implementations and other fast space partition JavaScript implementations but found nothing. Is my best bet to port some C++ implementation? Any answers/comments from someone who built something similar would be helpful!
I settled on preprocessing my data in the backend using a space partitioning implementation. I recommend it for anybody who has the luxury of doing so.
I'm trying to implement a layer which displays raster data sent by a server.
The data sent by the server protocol does not have a built-in support in the widely-used browser (this is a jpeg2000 data).
Thus I'm decoding the data by myself and let it to Cesium to show.
What makes it a little complicated:
The server is stateful, thus both client and server should maintain a channel. The channel is associated with a single region of interest. The region may change over time, but in every time point there is only single region for which the server sends data on the channel.
I can use some channels in the session, but the server does not perform well with more than very small amount of channels.
The region of interest is of uniform resolution (thus problematic for 3D).
The server supports progressive sending of data gradually enhance the quality ("quality layers" in jpeg2000), a property which I want to use due to very low network resource available.
The decoding is heavy in terms of CPU time.
As a first stage I've implemented an ImageryProvider which just creates a channel for each tile requested by the rendering engine. It worked but created too many connections and I didn't enjoy the progressive rendering. In addition the performance was poor, a problem which was almost resolved by implementing a priority mechanism which first decoded tiles in the view area of the Cesium viewer.
Next I implemented a self-rendered raster "layer" which change the region of interest of the channel according to the view area. Then the multiple channels problem was resolved and I enjoyed progressive rendering. However I encountered the following problems:
a. The method I used to show the decoded pixels was to implement an imagery provider which shows a single Canvas with the decoded pixels. Each time the image was updated (repositioned or progressively-decoded) I had to remove the old imagery provider and replace it with a new one. I guess that's not the correct way to do such things, and it may cause some bad behavior like wrong z-ordering when replacing the old provider with a new one, etc. Some of these issues may be resovled by using primitives with Image material, but then I have to use the data URL form of images. Doing that degrades performance, because it will cause a lot of conversions from canvas into data URLs.
b. I had to write special code to understand the view area in order to send it to the server (using pickEllipsoid and similar functionality). I guess this code is a duplication of something that is done within Cesium engine. In addition I saw in some discussions that pickEllipsoid is not supported in 2D. Generally I was very happy to have a function which calcualtes the view area for me, rather than implementing that code by myself.
c. The way I implemented it raises an API issue: As opposed to the nice API of Cesium to add and remove imagery provider (addImageryProvider() method and removeLayer() ), in my implementation the user need to use only the methods I expose to him (for example a method add() which accepts the Viewer as argument).
d. In 3D mode, when the resolution is not uniform the image is not sharp in the close region. I know that's an inherent problem because of the way that my server works, just point it out.
I think what I'm really missing here is a way to implement a plugin which is more powerful than the interface of ImageryProvider: Implementing a self-rendered raster layer, which receives view area change events from the render engine and can decide when and how to refresh its tiles.
Another alternative (which is even better for me, but is less reusable by others I guess), is to expose a list of the tiles in the view area to the ImageryProvider implementation.
What is the right way to cope with this scenario?
I'm currently writing an application that displays a lot, and I mean, a lot of 2D paths (made of hundreds, thousands of tiny segments) on an HTML5 canvas. Typically, a few million points. These points are downloaded from a server into a binary ArrayBuffer.
I probably won't be using that many points in the real world, but I'm kinda interested in how I could improve the performance. You can call it curiosity if you want ;)
Anyway, I've tested the following solutions :
Using gl.LINES or gl.LINE_STRIP with WebGL, and compute everything in shaders on the GPU. Currently the fastest, can display up to 10M segments without flinching on my Macbook Air. But there are very strict constraints for the binary format if you want to avoid processing things in JavaScript, which is slow.
Using Canvas2D, draw a huge path with all the segments in one stroke() call. When I'm getting past 100k points, the page freezes for a few seconds before the canvas is updated. So, not working here.
Using Canvas2D, but draw each path with its own stroke() call. Despite what others have been saying on the internet, this is much faster than drawing everything in one call, but still a lot slower than WebGL. Things start to get bad when I reach about 500k segments.
The two Canvas2D solutions require looping through all the points of all the paths in JavaScript, so this is quite slow. Do you know of any method(s) that could improve JavaScript's iteration speed in an ArrayBuffer, or processing speed in general?
But, what's strange is, the screen isn't updated immediately after all the canvas draw calls have finished. When I start getting to the performance limit, there is a noticeable delay between the end of the draw calls and the update of the canvas. Do you have any idea where that comes from, and is there a way to reduce it?
First, WebGL was a nice and hype idea, but the amount of processing required to decode and display the binary data simply doesn't work in shaders, so I ruled it out.
Here are the main bottlenecks I've encountered. Some of them are quite common in general programming, but it's a good idea to remember them :
It's best to use multiple, small for loops
Create variables and closures at the highest level possible, don't create them inside the for loops
Render your data in chunks, and use setTimeout to schedule the next chunk after a few milliseconds : that way, the user will still be able to use the UI
JavaScript objects and arrays are fast and cheap, use them. It's best to read/write them in sequential order, from the beginning to the end.
If you don't write data sequentially in an array, use objects (because non-sequential read-writes are cheap for objects) and push the indexes into an index array. I used a SortedList implementation to keep the indexes sorted, which I found here. Overhead was minimal (about 10-20% of the rendering time), and in the end it was well worth it.
That's about everything I remember. If I do find something else, I'll update this answer!
I want to draw a 3D cat(with animation) which is no more than a bunch of 3D objects - ellipsoids, pyramids, spheres, etc.
And I have 2 questions:
1) Are there any ways to define your own complex geometrical 3D objects rather than standard Three.js objects such as Sphere, Cube...
2) When animation the whole cat should I define an animation function for each object? Is there any way to combine some objects together?
For question one I'd recommend reading up on parameter driven modelling, this will allow you to make consistent complex objects without reinventing the wheel every time you create one. As for creating the custom objects, much like in the way polylines are are effectively a collection of lines with iterative implementations of the standard line methods (as well as object-specific methods) you'd create a javascript object which contains a collection of the objects necessary to create your custom shape. Here's a good webgl cheat sheet to help you out a bit.
Question two is somewhat similar to the way we've described complex objects above in that while you'll write a Cat object render / animate function, you'll handle the animation on a per object basis (with the exception full object static movement, imagine a cat on an escalator). Once again constraint, or parameter driven design will be your saviour here since the fact that two or more objects are partially superposed in no way means that the objects are explicitly linked.
As an end note I'd recommend looking into clojurescript. It might not be necessary for this type of work but lisp is very popular in the CAD scripting world and you'd definitely be doin' yourself a favour in the long run by at least familiarising yourself with the coding conventions - a lot of the questions you're goin' to have whilst working on this project will be answered in a variety of programming languages but you'll likely find that many of the answers that were written by folk working on both sides of the fence (cad/programming) will be written in lisp. Here's a final general CAD forum that's a great resource for all things CAD.