I'm using the leaflet package in R to generate a map with a large number of circles on it. The goal is a map I can publish to my website. The problem I'm having is that as I increase the number of circles, the resulting map loads very slowly, I get "unresponsive script" warnings and ultimately it completely freezes up my browser.
I know this sort of thing is possible, because I've found a leaflet map that works the way I want mine to work:
http://cartologic.com/geoapps/map_viewer/5/ny-crimes-2014-dot-density-map
I notice on the above map that the circles don't appear "clickable" like the circles on my map, and that they seem to load in square chunks. I have a hunch that these things are related to my problem. Unfortunately, I'm too much of a novice at leaflet/javascript stuff to figure this out on my own.
Here is a toy example illustrating my problem:
library("leaflet")
library("htmlwidgets")
dots <- data.frame(x=c(runif(10000, -93.701281, -93.533053)),
y=c(runif(10000, 41.515962, 41.644369)))
m <- leaflet(dots) %>%
addTiles('http://{s}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}.png') %>%
setView(-93.617167, 41.580166, zoom = 12) %>%
addCircles(~x, ~y, weight = 1, radius = 5,
color = "#FFA500", stroke = TRUE, fillOpacity = 0.1)
m
saveWidget(widget = m, file="example.html", selfcontained = TRUE)
mapview can help you here. It builds upon the leaflet library for smaller data sets, but uses special javascript functionality for larger data.
your example with 1 Mio. points:
library(mapview)
library(sp)
dots <- data.frame(x=c(runif(1000000, -93.701281, -93.533053)),
y=c(runif(1000000, 41.515962, 41.644369)))
coordinates(dots) <- ~ x + y
proj4string(dots) <- "+init=epsg:4326"
mapview(dots)
It may still take a while to render, but once rendered it should be quite responsive. Note that mapview is designed to work with spatial* objects, that is why we need the calls to set the coordinate slot and the projection.
For more information have a look here:
http://environmentalinformatics-marburg.github.io/web-presentations/20150723_mapView.html
Hope that helps.
If you want to add a large number of vector objects to a map, it is rare that it can be done easily.
Notice that the raster data is broken into tiles so that all the information does not have to be shown at one time. For your vector data (in this case, circles) you have to do the same thing.
Basically what I like to do is to break the large data set into smaller (vector) tiles, with the same boundaries as the raster tiles you are showing. Duplicate the data if you want it to appear at several zoom level. As you are showing circle, imagine that you partition the circles' center points on the tile boundary.
I have an application similar to this where I basically partition my vector data on tile boundaries and store the information in geojson files. When I get an event that the raster tile has been loaded I can then load the equivalent vector file as a geojson layer (same thing when the raster tile is unloaded). In this way, you can limit the amount of vector data that has to be displayed at any one time.
If you have a lot of points, they are not really going to be visible at low zoom levels anyway, so it might be better just to show them at an appropriate zoom level (perhaps with a different representation at low zooms - like a heat map). This will keep the amount of data being shown at any one time lower.
Since this question has a few upvotes, I'll generally describe both of the solutions I found. Maybe if I have time later I'll get all the files together on GitHub.
First, I found TileMill. Simply load a data file of coordinates into TileMill, style the way you want them to appear, and output tiles (png). Host those tiles on the web somewhere and load them with leaflet. This process was a bit too manual for my liking because TileMill kept crashing when I loaded in csv files that were too large for it to render on my machine.
I found the best solution was use Processing, adapting Robert Manduca's code here: https://github.com/rmanduca/jobmaps. I don't use Python so I rewrote those parts in R and modified the Processing code according to my specifications.
Mapdeck (released on CRAN Aug 2018) uses WebGL (through Deck.gl) and is designed to handle millions of points (depending on your system's hardware of course)
library(mapdeck)
set_token("MAPBOX_TOKEN")
n <- 1e6
dots <- data.frame(x=c(runif(n, -93.701281, -93.533053)),
y=c(runif(n, 41.515962, 41.644369)))
dots$letter <- sample(letters, size = n, replace = T)
mapdeck(
style = mapdeck_style('dark')
) %>%
add_scatterplot(
data = dots
, lon = "x"
, lat = "y"
, fill_colour = "letter"
, radius = 5
, fill_opacity = 50
, layer_id = "dots"
)
..and then Google-maps "divide the waters from the waters"
Well, not in the biblical sense but..
I would like to know what options I have in order to verify if a point of [Lat, Lon] is Land or Water.
Google Maps obviously has this data (the bodies of water are blue) - but is there something in the API that I can use for that? And if not - are they not serving it because they never thought of it? Or because it is too complicated?
I have not found any info on the matter - except some similar questions here (like finding type of terrain, or elevation - but it is not exactly what I need).
Is there separated layer for that? An option? Command? Or should I go to do that manually?
The only way that I can think of how to approach this (should I need to do that manually) is to check every served tile for the exact point - and then check RGB value for that Google map hue.
This is only on theory - because in practice - I have no idea how to accomplish that, the first obstacle being that I do not know how I can convert a pixel location on a tile to [LatLon] point for example
A ready made solution would be much easier.
Note that I do not need ALL the water in the world (for example - I do not care about streams, small ponds, most rivers or your neighbor's swimming pool. I need the points where a person can venture without the aid of a floating vehicle)
EDIT I
After reading comments:
The elevation method is not reliable, there are too many places BELOW sea-level (you can see a list of the "deepest" 10 here http://geology.com/below-sea-level/ ) and there are too many land-locked water bodies ABOVE sea level (lakes).
The reverse geolocation method is not reliable because it will return a Geo-political entity, like city, or state - or ZERO many times.
I have already looked into those pseudo-solutions before asking the question - but none of them actually answered the question - those methods are bad "guessing" at best.
These are 2 different ways, you may try:
You can use Google Maps Reverse Geocoding . In result set you can determine whether it is water by checking types. In waters case the type is natural_feature. See more at this link http://code.google.com/apis/maps/documentation/geocoding/#Types.
Also you need to check the names of features, if they contain Sea, Lake, Ocean and some other words related to waters for more accuracy. For example the deserts also are natural_features.
Pros - All detection process will be done on client's machine. No need of creating own server side service.
Cons - Very inaccurate and the chances you will get "none" at waters is very high.
You can detect waters/lands by pixels, by using Google Static Maps. But for this purpose you need to create http service.
These are steps your service must perform:
Receive latitude,longitude and current zoom from client.
Send http://maps.googleapis.com/maps/api/staticmap?center={latitude,longitude}&zoom={current zoom`}&size=1x1&maptype=roadmap&sensor=false request to Google Static Map service.
Detect pixel's color of 1x1 static image.
Respond an information about detection.
You can't detect pixel's color in client side. Yes , you can load static image on client's machine and draw image on canvas element. But you can't use getImageData of canvas's context for getting pixel's color. This is restricted by cross domain policy.
Prons - Highly accurate detection
Cons - Use of own server resources for detection
It doesn't seem possible with any current Google service.
But there are other services, like Koordinates Vector JSON Query service! You simply query the data in the URL, and you get back a JSON/XML response.
Example request: http://api.koordinates.com/api/vectorQuery.json?key=YOUR_GEODATA_KEY&layer=1298&x=-159.9609375&y=13.239945499286312&max_results=3&radius=10000&geometry=true&with_field_names=true
You have to register and supply your key and selected layer number. You can search all their repository of available layers. Most of the layers are only regional, but you can find global also, like the World Coastline:
When you select a layer, you click on the "Services" tab, you get the example request URL. I believe you just need to register and that's it!
And now the best:
You can upload your layer!
It is not available right away, hey have to process it somehow, but it should work! The layer repository actually looks like people uploaded them as they needed.
There is a free web API that solves exactly this problem called onwater.io. It isn't something built into Google maps, but given a latitude and longitude it will accurately return true or false via a get request.
Example on water:
https://api.onwater.io/api/v1/results/23.92323,-66.3
{
lat: 23.92323,
lon: -66.3,
water: true
}
Example on land:
https://api.onwater.io/api/v1/results/42.35,-71.1
{
lat: 42.35,
lon: -71.1,
water: false
}
Full disclosure I work at Dockwa.com, the company behind onwater. We built onwater to solve this problem ourselves and help the community. It is free to use (paid for high volume) and we wanted to share :)
I thought it was more interesting to do this query locally, so I can be more self-reliant: let's say I want to generate 25000 random land coordinates at once, I would rather want to avoid calls to possibly costly external APIs. Here is my shot at this in python, using the python example mentionned by TomSchober. Basically it looks up the coordinates on a pre-made 350MB file containing all land coordinates, and if the coordinates exist in there, it prints them.
import ogr
from IPython import embed
import sys
drv = ogr.GetDriverByName('ESRI Shapefile') #We will load a shape file
ds_in = drv.Open("land_polygons.shp") #Get the contents of the shape file
lyr_in = ds_in.GetLayer(0) #Get the shape file's first layer
#Put the title of the field you are interested in here
idx_reg = lyr_in.GetLayerDefn().GetFieldIndex("P_Loc_Nm")
#If the latitude/longitude we're going to use is not in the projection
#of the shapefile, then we will get erroneous results.
#The following assumes that the latitude longitude is in WGS84
#This is identified by the number "4236", as in "EPSG:4326"
#We will create a transformation between this and the shapefile's
#project, whatever it may be
geo_ref = lyr_in.GetSpatialRef()
point_ref=ogr.osr.SpatialReference()
point_ref.ImportFromEPSG(4326)
ctran=ogr.osr.CoordinateTransformation(point_ref,geo_ref)
def check(lon, lat):
#Transform incoming longitude/latitude to the shapefile's projection
[lon,lat,z]=ctran.TransformPoint(lon,lat)
#Create a point
pt = ogr.Geometry(ogr.wkbPoint)
pt.SetPoint_2D(0, lon, lat)
#Set up a spatial filter such that the only features we see when we
#loop through "lyr_in" are those which overlap the point defined above
lyr_in.SetSpatialFilter(pt)
#Loop through the overlapped features and display the field of interest
for feat_in in lyr_in:
# success!
print lon, lat
check(-95,47)
I tried a dozen coordinates, it works wonderfully. The "land_polygons.shp" file can be downloaded here, compliments of OpenStreetMaps. (I used the first WGS84 download link myself, maybe the second works as well)
This what I use and it is working not too bad... you can improve the test if you have more cpu to waste by adding pixels.
function isItWatter($lat,$lng) {
$GMAPStaticUrl = "https://maps.googleapis.com/maps/api/staticmap?center=".$lat.",".$lng."&size=40x40&maptype=roadmap&sensor=false&zoom=12&key=YOURAPIKEY";
//echo $GMAPStaticUrl;
$chuid = curl_init();
curl_setopt($chuid, CURLOPT_URL, $GMAPStaticUrl);
curl_setopt($chuid, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($chuid, CURLOPT_SSL_VERIFYPEER, FALSE);
$data = trim(curl_exec($chuid));
curl_close($chuid);
$image = imagecreatefromstring($data);
// this is for debug to print the image
ob_start();
imagepng($image);
$contents = ob_get_contents();
ob_end_clean();
echo "<img src='data:image/png;base64,".base64_encode($contents)."' />";
// here is the test : I only test 3 pixels ( enough to avoid rivers ... )
$hexaColor = imagecolorat($image,0,0);
$color_tran = imagecolorsforindex($image, $hexaColor);
$hexaColor2 = imagecolorat($image,0,1);
$color_tran2 = imagecolorsforindex($image, $hexaColor2);
$hexaColor3 = imagecolorat($image,0,2);
$color_tran3 = imagecolorsforindex($image, $hexaColor3);
$red = $color_tran['red'] + $color_tran2['red'] + $color_tran3['red'];
$green = $color_tran['green'] + $color_tran2['green'] + $color_tran3['green'];
$blue = $color_tran['blue'] + $color_tran2['blue'] + $color_tran3['blue'];
imagedestroy($image);
var_dump($red,$green,$blue);
//int(492) int(570) int(660)
if($red == 492 && $green == 570 && $blue == 660)
return 1;
else
return 0;
}
Checkout this article. It accurately detects if something is on the water without needing a server. It's a hack that relies on the custom styling feature in Google Maps.
In addition to the reverse geocoding -- as Dr Molle has pointed out, it may return ZERO_RESULTS -- you could use the Elevation service. If you get zero results by reverse geocoding, get the elevation of the location. Generally, the sea gets a negative number as the seabed is below sea level. There's a fully-worked example of the elevation service.
Bear in mind that as Google don't make this information available any other method is just a guess and guesses are inherently inaccurate. However using the type returned by reverse geocoding, or the elevation if type is not available, will cover most eventualities.
This method is totally unreliable.
In fact, the returned data will totally depend on what part of the world you are working with.
For example, I am working in France.
If I click on the sea on the coast of France, Google will return the nearest LAND location it can "guess" at.
When I requested information from Google for this same question, they answered that they are unable to accurately return that the point requested in on a water mass.
Not a very satisfactory answer, I know.
This is quite frustrating, especially for those of us who provide the user with the ability to click on the map to define a marker position.
If all else fails you could always try checking the elevation at the point and for some distance about - not many things other than water tend to be completely flat.
Unfortunately this answer isn't within the Google Maps API and the referenced resource is not free, but there's a web service provided by DynamicGeometry that exposes an operation GetWaterOrLand which accepts a latitude/longitude pair (you can see a demo here).
My understanding of how this is implemented is by using water body shape files. How exactly these shape files are used with the Google Maps API, but you might be able to get some insight from the linked demo.
Hope that helps in some way.
Here's another example in pure JavaScript: http://jsfiddle.net/eUwMf/
As you can see, the ideia is basically the same as rebe100x, getting the image from Google static map API, and read the first pixel:
$("#xGps, #yGps").change(function() {
var img = document.getElementById('mapImg');
// Bypass the security issue : drawing a canvas from an external URL.
img.crossOrigin='anonymous';
var xGps = $("#xGps").val();
var yGps = $("#yGps").val();
var mapUrl = "http://maps.googleapis.com/maps/api/staticmap?center=" + xGps + "," + yGps +
"&zoom=14&size=20x20&maptype=roadmap&sensor=false";
// mapUrl += "&key=" + key;
$(img).attr("src", mapUrl);
var canvas = $('<canvas/>')[0];
canvas.width = img.width;
canvas.height = img.height;
canvas.getContext('2d').drawImage(img, 0, 0, img.width, img.height);
var pixelData = canvas.getContext('2d').getImageData(1, 1, 1, 1).data;
if (pixelData[0] == 164 &&
pixelData[1] == 190 &&
pixelData[2] == 220) {
$("#result").html("Water");
} else {
$("#result").html("Not water");
}
});
See the answer I gave to a similar question - it uses "HIT_TEST_TERRAIN" from the Earth Api to achieve the function.
There is a working example of the idea I put together here: http://www.msa.mmu.ac.uk/~fraser/ge/coord/
If List<Address> address returns 0 , you can assume this location as ocean or Natural Resources.Just add Below Code in Your response Method of Google Places API Response.
Initialize Below List as mentioned
List<Address> addresses = geocoder.getFromLocation(latLng.latitude, latLng.longitude, 1);
if (addresses.size()==0)
{
Toast.MakeText(getApplicationContext,"Ocean or Natural Resources selected",Toast.LENGTH_SHORT).show();
}else{
}
I would recommend rolling your own here. You can use tools like GDAL to query the contents under a point in a shapefile. You can get shapefiles for US geography from many sources including the US Census Bureau.
This can be done via GDAL binaries, the source C, or via swig in Java, Python, and more.
Census Maps
GDAL Information
Point Query Example in Python
Here is a simple solution
Because Google does not provide reliable results with regards to coordinates that lay on either ocean or inland bodies of water you need to use another backup service, such as Yandex, to help provide that critical information when it is missing. You most likely would not want to use Yandex as your primary geocoder because Google is far superior in the reliability and completeness of the worlds data, however Yandex can be very useful for the purpose of retrieving data when it relates to coordinates over bodies of water, so use both.
Yandex Documentation: https://api.yandex.com.tr/maps/doc/geocoder/desc/concepts/input_params.xml
The steps to retrieve Ocean name:
1.) Use Google first to reverse geocode the coordinate.
2.) If Google returns zero results, it is 99% likely the coordinate lies over an ocean. Now make a secondary reverse geocoding request with the same coordinates to Yandex. Yandex will return a JSON response with for the exact coordinates, within this response will be two "key":"value" pairs of importance
["GeoObject"]["metaDataProperty"]["GeocoderMetaData"]["kind"]
and
["GeoObject"]["name"]
Check the kind key, if it == "hydro" you know you are over a body of water, and because Google returned zero results it is 99.99% likely this body of water is an ocean. The name of the ocean will be the above "name" key.
Here is an example of how I use this strategy written in Ruby
if result.data["GeoObject"]["metaDataProperty"]["GeocoderMetaData"]["kind"] == "hydro"
ocean = result.data["GeoObject"]["name"]
end
The steps to retrieve an Inland Body of Water name:
For this example assume our coordinate lies in a lake somewhere:
1.) Use Google first to reverse geocode the coordinate.
2.) Google will most likely return a result that is a prominent default address on land nearby. In this result it supplies the coordinates of the address it returned, this coordinate will not match the one you provided. Measure the distance between the coordinate you supplied and the one returned with the result, if it is significantly different (for example 100 yards) then perform a secondary backup request with Yandex and check to see the value of the "kind" key, if it is "hydro" then you know the coordinate lies on water. Because Google returned a result as opposed to the example above, it is 99.99% likely this is an inland body of water so now you can get the name. If "kind" does not == "hydro" then use the Google geocoded object.
["GeoObject"]["metaDataProperty"]["GeocoderMetaData"]["kind"]
and
["GeoObject"]["name"]
Here is the same code written in Ruby to get inland_body_of_water
if result.data["GeoObject"]["metaDataProperty"]["GeocoderMetaData"]["kind"] == "hydro"
inland_body_of_water = result.data["GeoObject"]["name"]
end
A note about Licensing: As far as I know Google does not allow you to use their data to display on any other maps other than those Google offers. Yandex however has very flexible licensing, and you can use their data to be displayed on Google maps.
Also Yandex has a a high rate limit of 50,000 request / day free of charge, and with no required API key.
I managed to get quite close by using the Google Elevation API. Here's an image of the results:
You see the hexagons pretty much stay on land even though a rectangular perimeter is defined that goes partly over water. In this case I did a quick check from Google Maps itself and the minimum elevation on land was about 8-9m so that was my threshold. The code is mostly copy/pasted from Google documentation and Stack Overflow, here's the full gist:
https://gist.github.com/dvas0004/fd541a0502528ebfb825
As a complete novice to Python I couldn't get SylvainB's solution to work with the python script that checks if coordinates are on land. I managed to figure it out however, by downloading OSGeo4W (https://trac.osgeo.org/osgeo4w/) and then installed everything I needed pip, Ipython, and checked that all the imports specified were there. I saved the following code as a .py file.
Code to check if coordinates are on land
###make sure you check these are there and working separately before using the .py file
import ogr
from IPython import embed
from osgeo import osr
import osgeo
import random
#####generate a 1000 random coordinates
ran1= [random.uniform(-180,180) for x in range(1,1001)]
ran2= [random.uniform(-180,180) for x in range(1,1001)]
drv = ogr.GetDriverByName('ESRI Shapefile') #We will load a shape file
ds_in = drv.Open("D:\Downloads\land-polygons-complete-4326\land-polygons-complete-4326\land_polygons.shp") #Get the contents of the shape file
lyr_in = ds_in.GetLayer(0) #Get the shape file's first layer
#Put the title of the field you are interested in here
idx_reg = lyr_in.GetLayerDefn().GetFieldIndex("P_Loc_Nm")
#If the latitude/longitude we're going to use is not in the projection
#of the shapefile, then we will get erroneous results.
#The following assumes that the latitude longitude is in WGS84
#This is identified by the number "4236", as in "EPSG:4326"
#We will create a transformation between this and the shapefile's
#project, whatever it may be
geo_ref = lyr_in.GetSpatialRef()
point_ref=osgeo.osr.SpatialReference()
point_ref.ImportFromEPSG(4326)
ctran=osgeo.osr.CoordinateTransformation(point_ref,geo_ref)
###check if the random coordinates are on land
def check(runs):
lon=ran1[runs]
lat=ran2[runs]
#Transform incoming longitude/latitude to the shapefile's projection
[lon,lat,z]=ctran.TransformPoint(lon,lat)
#Create a point
pt = ogr.Geometry(ogr.wkbPoint)
pt.SetPoint_2D(0, lon, lat)
#Set up a spatial filter such that the only features we see when we
#loop through "lyr_in" are those which overlap the point defined above
lyr_in.SetSpatialFilter(pt)
#Loop through the overlapped features and display the field of interest
for feat_in in lyr_in:
return(lon, lat)
###give it a try
result = [check(x) for x in range(1,11)] ###checks first 10 coordinates
I tried to get it to work in R but I had a nightmare trying to get all the packages you need to install so stuck to python.
Here's a typed async function that returns true or false if a lat/lng is water or not. No need to pay for external api's. You must enable static maps on google cloud though.
async function isLatLngWater(lat: number, lng: number) {
return new Promise<boolean>((resolve) => {
const img = new Image();
img.crossOrigin = "Anonymous";
img.onload = () => {
const canvas = document.createElement("canvas");
const ctx = canvas.getContext("2d");
ctx!.drawImage(img, 0, 0);
const { data } = ctx!.getImageData(10, 10, 1, 1);
if (data[0] == 156 && data[1] == 192 && data[2] == 249) {
canvas.remove();
resolve(true);
} else {
canvas.remove();
resolve(false);
}
};
img.src =
"https://maps.googleapis.com/maps/api/staticmap?center=" +
lat +
"," +
lng +
"&size=40x40&maptype=roadmap&sensor=false&zoom=20&key=" +
import.meta.env.VITE_GM_MSTAT;
});
}
There an API service called IsItWater.com which will let you check:
Request
curl 'https://isitwater-com.p.rapidapi.com/?latitude=41.9029192&longitude=-70.2652276&rapidapi-key=YOUR-X-RAPIDAPI-KEY'
Response
{
"water": true,
"latitude": 41.9029192,
"longitude": -70.2652276
}
I have a different solution here.
In current google map implementation, it does not calculate direction/distance from a water location to land location and vice versa. Why dont we use this logic to determine if the point is land or water.
For example lets take this example
if we want to determine, if a point x is land or water, then
let us check the direction between point x and a known point y which is land. If it determines the direction/distance then point x is land or else it is water.