I'm trying to implement a layer which displays raster data sent by a server.
The data sent by the server protocol does not have a built-in support in the widely-used browser (this is a jpeg2000 data).
Thus I'm decoding the data by myself and let it to Cesium to show.
What makes it a little complicated:
The server is stateful, thus both client and server should maintain a channel. The channel is associated with a single region of interest. The region may change over time, but in every time point there is only single region for which the server sends data on the channel.
I can use some channels in the session, but the server does not perform well with more than very small amount of channels.
The region of interest is of uniform resolution (thus problematic for 3D).
The server supports progressive sending of data gradually enhance the quality ("quality layers" in jpeg2000), a property which I want to use due to very low network resource available.
The decoding is heavy in terms of CPU time.
As a first stage I've implemented an ImageryProvider which just creates a channel for each tile requested by the rendering engine. It worked but created too many connections and I didn't enjoy the progressive rendering. In addition the performance was poor, a problem which was almost resolved by implementing a priority mechanism which first decoded tiles in the view area of the Cesium viewer.
Next I implemented a self-rendered raster "layer" which change the region of interest of the channel according to the view area. Then the multiple channels problem was resolved and I enjoyed progressive rendering. However I encountered the following problems:
a. The method I used to show the decoded pixels was to implement an imagery provider which shows a single Canvas with the decoded pixels. Each time the image was updated (repositioned or progressively-decoded) I had to remove the old imagery provider and replace it with a new one. I guess that's not the correct way to do such things, and it may cause some bad behavior like wrong z-ordering when replacing the old provider with a new one, etc. Some of these issues may be resovled by using primitives with Image material, but then I have to use the data URL form of images. Doing that degrades performance, because it will cause a lot of conversions from canvas into data URLs.
b. I had to write special code to understand the view area in order to send it to the server (using pickEllipsoid and similar functionality). I guess this code is a duplication of something that is done within Cesium engine. In addition I saw in some discussions that pickEllipsoid is not supported in 2D. Generally I was very happy to have a function which calcualtes the view area for me, rather than implementing that code by myself.
c. The way I implemented it raises an API issue: As opposed to the nice API of Cesium to add and remove imagery provider (addImageryProvider() method and removeLayer() ), in my implementation the user need to use only the methods I expose to him (for example a method add() which accepts the Viewer as argument).
d. In 3D mode, when the resolution is not uniform the image is not sharp in the close region. I know that's an inherent problem because of the way that my server works, just point it out.
I think what I'm really missing here is a way to implement a plugin which is more powerful than the interface of ImageryProvider: Implementing a self-rendered raster layer, which receives view area change events from the render engine and can decide when and how to refresh its tiles.
Another alternative (which is even better for me, but is less reusable by others I guess), is to expose a list of the tiles in the view area to the ImageryProvider implementation.
What is the right way to cope with this scenario?
Related
I am a bit confused about Mapbox MVT. As I understood, a tile is a little piece of map, as in a jigsaw puzzle.
Not completely sure about the working of MVT.
https://docs.mapbox.com/data/tilesets/guides/vector-tiles-introduction/#benefits-of-vector-tiles
Here, it says Vector tiles are really small, enabling global high resolution maps, fast map loads, and efficient caching.
So the thing is I am trying to get all the coordinates from db which can go up to more than 10K and currently getting the data from postgis using:
with connection.cursor() as cursor:
cursor.execute(query)
rows = cursor.fetchall()
mvt = bytes(rows[-1][-1])
return Response(
mvt, content_type="application/vnd.mapbox-vector-tile", status=200
)
Now I am wondering about the performance issues, as everytime a user will visit it will put stress on db.
And another problem I am having is when using vector tiles as a source, it calls the source url and (hitting the db) everytime I move the map.
type: 'vector',
tiles: [
'http://url/{z}/{x}/{y}.mvt'
]
Is it possible to call the source url at a specific zoom level and until then all the points remains on the map?
for eg.
Mapbox should call source url (only one time from zoom level 1-7) at zoom level 1 and draw points on to the map and when zoom level reach 7 then mapbox should call the source url(only one time from zoom level 7-22) and update the map.
Really be grateful if anyone can help.
When it comes to tiling data (vector, raster, whatever format), you will almost always want a service that has a caching strategy, especially if the tiles are being created in real time from data in a database. Calling directly into the DB for when every tile is needed should only be done during development/testing, or for simple demos. Vector tiles alone are not the solution, you need an end-to-end architecture for serving the data. Here are a couple of examples of caching strategies:
If your data is constantly changing by the second or faster, you often don't need to show the update that quickly on a map. If you have a lot of users a 15 second cache expiry on tiles can drastically reduce the number of times a tile has to be created. Azure Maps does this for their creator platform which is designed to show the near real time sensor readings in a digital twin scenario. It is capable of supporting billions of sensors updating the map every 15 seconds. If you truly need to see updates as they happen, then vector tiles likely isn't the right approach, instead consider using a stream service like SignalR, and limiting the data sent to the map based on bounding box and zoom level.
In most applications the data doesn't update that quickly, so a longer cache header can be used.
In some cases, the data changes so infrequently (or not at all) that it makes more sense to pre-generate the tiles once, host those, and serve those directly. You would still have a caching strategy so that users aren't constantly requesting the same tiles from your server when they could simply pull it from their browser cache. In this case, it is useful to put the tiles into a MBTile file which is basically a Sqlite DB with a standardized structure. The benifit of putting the tiles in an MBTile file is you only have to move one file around, instead of thousands of tiles (moving this many files creates a ton of IO read/writes that can make deploying extremely slow).
It's also important to note that you should optimize the data in your tiles. For example, if you are working with high resolution polygons, for tiles that are zoomed out, you will likely find there are a lot of coordinates that fit inside the same pixel, so when you request the polygon from the database, reduce its resolution to match the resolution of the tile. This will drastically reduce the size of the tile, and the amount of data the database is outputting.
You can find a lot of tools and articles for vector tiles here: https://github.com/mapbox/awesome-vector-tiles
There are also lots of blogs out there on how to serve vector tiles and create pipelines.
So far I seen so many discussion on this topic and using different approaches to achieve this (https://github.com/tensorflow/models/issues/1809) but I want to know if anyone managed to successfully use Tensorflowjs to achieve this.
I know some also achieved this using transfer learning but it is not same as being able to add my own new class.
The short answer: No, not yet, though technically possible, I have not seen an implementation of this in the wild.
The longer answer - why:
Given that "transfer learning" essentially means reusing the existing knowledge in a trained model to help you then classify things of a similar nature without having to redo all the prior learning there are actually 2 ways to do that:
1) This is the easier route but may not be possible for some use cases: Use one of the high level layers of the frozen model that you have access to (eg the models that are released by TF.js are frozen models I believe - the ones on GitHub). This allows you to reuse some of its lower layers (or final output) which may already be good at picking out certain features that are useful for the use case you need eg object detection in a general sense, which you can then feed into your own unfrozen layers that sit on top of that output you are sampling from (which is where the new training would happen). This is faster as you are only updating weights etc for the new layers you have added, however because the original model is frozen, it means you would have to replicate in TF.js the layers you were bypassing to ensure you have the same resulting model architecture for COCO-SSD in this case if you wanted the architecture. This may not be trivial to do.
2) Retraining the original model - can think of tuning the original model - but this is only possible if you have access to the original unfrozen model and the data used to train that. This would take longer as you are essentially retraining the whole model on all the data + your new data. If you do not have the original unfrozen model, then the only way to do this would be to implement the said model in TF.js yourself using the layers / ops APIs as needed and then use that to train on your own data.
What?!
So an easier to visualize example of this is if we consider PoseNet - the one that estimates where human joints/skeletons are.
Now in this Posenet example imagine you wanted to make a new ML model that could detect when a person is in a certain position - eg waving a hand.
In this example you could use method 1 to simply take the output of existing posenet predictions for all the joints it has detected and feed that into a new layer - something simple like a multi layered perceptron - that could then very quickly learn from example data when a hand was in a waving position for example. In this case we are simply adding to the existing architecture to achieve a new result - gesture prediction vs the raw x-y point predictions for the joints themselves.
Now consider case 2 for PoseNet - you want to be able to recognise a new part of the body that it currently does not. For that to happen you would need to retrain the original model so that it could learn to predict that new body part as part of its output.
This is much harder as you would need to retrain the base model to do this, which means you need to have access to the unfrozen model to do that. If you didn't have access to the unfrozen model then you would have no choice but attempt to recreate PoseNet architecture entirely yourself and then train that with your own data. As you can see this 2nd use case is much harder and more involved to do.
I am currently looking at an efficient way to visualise a lot of data in javascript. The data is geospatial and I have approximately 2 million data points.
Now I know that I cannot give that many datapoint to the browser directly otherwise it would just crash most of the time (or the response time will be very slow anyway).
I was thinking of having a javascript window communicating with a python which would do all the operations on the data and stream json data back to the javascript app.
My idea was to have the javascript window send in real time the bounding box of the map (lat and lng of north east and south west Point) so that the python script could go through all the entries before sending the json of only viewable objects.
I just did a very simple script that could do that which basically
Reads the whole CSV and store data in a list with lat, lng, and other attributes (2 or 3)
A naive implementation to check whether points are within the bounding box sent by the javascript.
Currently, going through all the datapoints takes approximately 15 seconds... Which is way too long, since I also have to then transform them into a geojson object before streaming them to my javascript application.
Now of course, I could first of all sort my points in ascending order of lat and lng so that the function checking if a point is within the javascript sent bounding box would be an order of magnitude faster. However, the processing time would still be too slow.
But even admitting that it is not, I still have the problem that at very low zoom levels, I would get too many points. Constraining the min_zoom_level is not really an option for me. So I was thinking that I should probably try and cluster data points.
My question is therefore do you think that this approach is the right one? If so, how does one compute the clusters... It seems to me that I would have to generate a lot of possible clusters (different zoom levels, different places on the map...) and I am not sure if this is an efficient and smart way to do that.
I would very much like to have your input on that, with possible adjustments or completely different solutions if you have some.
This is almost language agnostic, but I will tag as python since currently my server is running python script and I believe that python is quite efficient for large datasets.
Final note:
I know that it is possible to pre-compute tiles that I could just feed my javascript visualization but as I want to have interactive control over what is being displayed, this is not really an option for me.
Edit:
I know that, for instance, mapbox provides the clustering of data point to facilitate displaying something like a million data point.
However, I think (and this is related to an open question here
) while I can easily display clusters of points, I cannot possibly make a data-driven style for my cluster.
For instance, if we take the now famous example of ethnicity maps, if I use mapbox to cluster data points and a cluster is giving me 50 people per cluster, I cannot make the cluster the color of the most represented ethnicity in the sample of 50 people that it gathers.
Edit 2:
Also learned about supercluster, but I am quite unsure whether this tool could support multiple million data points without crashing either.
I'm noob at WebGL. I read in several posts of ND-Buffers and G-Buffers as if it were a strategic choice for WebGL development.
How are ND-Buffers and G-Buffers related to rendering pipelines? Are ND-Buffers used only in forward-rendering and G-Buffers only in deferred-rendering?
A JavaScript code example how to implement both would be useful for me to understand the difference.
G-Buffers are just a set of buffers generally used in deferred rendering.
Wikipedia gives a good example of the kind of data often found in a g-buffer
Diffuse color info
World space or screen space normals
Depth buffer / Z-Buffer
The combination of those 3 buffers is referred to as a "g-buffer"
Generating those 3 buffers from geometry and material data you can then run a shader to combine them to generate the final image.
What actually goes into a g-buffer is up to the particular engine/renderer. For example one of Unity3D's deferred renders contains diffuse color, occlusion, specular color, roughness, normal, depth, stencil, emission, lighting, lightmap, reflection probs.
An ND buffer just stands for "normal depth buffer" which makes it a subset of what's usually found in a typical g-buffer.
As for a sample that's arguably too big for SO but there's an article about deferred rendering in WebGL on MDN
Choosing a rendering path is a major architectural decision for a 3D renderer, no matter what API does it use. That choice's heavily depends upon the set of features the renderer has to support and it's performance requirements.
A substantial set of said features consists of so-called screen-space effects. If means that we render some crucial data about each pixel of the screen to a set of renderbuffers and then using that data (not the geometry) to compute some new data needed for a frame. Ambient Occlusion is a great example of such an effect. Based on some spacial values of pixels we compute a "mask" which we can later use to properly shade each pixel.
Moreover, there is a rendering pass which almost exclusively relies on screen-space computations. And it is indeed Deferred Shading. And that's where G-buffer come in. All data needed to compute colour of a pixel are rendered to a G-buffer: a set of renderbuffers storing that data. The data it self (and hence meanings of G-buffer's renderbuffers) can be different: diffuse component, specular component, shininess, normal, position, depth, etc. And as part of rendering of a frame contemporary deferred shading engines use screen-space ambient occlusion (SSAO), which use data from several G-buffer's renderbuffers (usually, they are position, normal and depth).
About ND-buffers. It seems to me that it's not a widely used term (Google failed to find any relevant info on them besides this question). I believe that ND stands for Normal-Depth. They're just a specific case of a G-buffer for a particular algorithm and effect (in the thesis it's SSAO).
So using G-buffers (and ND-buffers as a subset of G-buffers) and exadepends upon shading algorithms and effects you're implementing. But all screen-space computation will require some form of G-buffer.
P.S. The thesis you've link contains an inaccuracy. Author lists an ability to implement to ND-buffers on GLES 2.0 as an advantage to the method. However it's not actually possible since GLES 2.0 doesn't have depth textures (they've been added in OES_depth_texture extension).
I would like to add some more informations to previvous answers.
I read in several posts of ND-Buffers and G-Buffers as if it were a
strategic choice for WebGL development.
One of the most important part of deferred rendering is, if given platform supports MRT (multiple render targets). If it doesn't, you are not able to share partial calculations in shaders between each rendering and it also forces you to run rendnering as many times as you have "layers" (in case of unity 3D, it might be up to 11 times?). This could slow down your program a lot.
Read more in this question
Is deferred rendering/shading possible with OpenGL ES 2.0 ?
Webgl doesn't support MRT, but it has extension:
https://www.khronos.org/registry/webgl/extensions/WEBGL_draw_buffers/
Also there is an extension for depth textures:
https://www.khronos.org/registry/webgl/extensions/WEBGL_depth_texture/
So it should be possible to use deferred rendering technique, but its speed is hard to guess.
I am developing a web app based on the Google App Engine.
It has some hundreds of places (name, latitude, longitude) stored in the Data Store.
My aim is to show them on google map.
Since they are many I have registered a javascript function to the idle event of the map and, when executed, it posts the map boundaries (minLat,maxLat,minLng,maxLng) to a request handler which should retrieve from the data store only the places in the specified boundaries.
The problem is that it doesn't allow me to execute more than one inequality in the query (i.e. Place.latminLat, Place.lntminLng).
How should I do that? (trying also to minimize the number of required queries)
You could divide the map into regions, make an algorithm to translate the current position into a region, and then get the places by an equality query. In this case you would need overlapping regions, allow places to be part of many of them, and ideally make regions bigger than the size of the map, in order to minimize the need for multiple queries.
That was just an outline of the idea, an actual implementation would be a little bit more complicated, but I don't have one at hand.
Another option is using geohashes, which are actually pretty cool, you can read a write up about them, along with code samples, here: Scalable, fast, accurate geo apps using Google App Engine + geohash + faultline correction
You didn't say how frequently the data points are updated, but assuming 1) they're updated infrequently and 2) there are only hundreds of points, then consider just querying them all once, and storing them sorted in memcache. Then your handler function would just fetch from memcache and filter in memory.
This wouldn't scale indefinitely but it would likely be cheaper than querying the Datastore every time, due to the way App Engine pricing works.