I was wondering if it is possible to extract the parameters of a JavaScript function with Scrapy, from a code similar to this one:
<script type="text/javascript">
var map;
function initialize() {
var fenway = new google.maps.LatLng(43.2640611,2.9388228);
};
}
</script>
I would like to extract the coordinates 43.2640611 and 2.9388228.
This is where re() method would help.
The idea is to locate the script tag via xpath() and use re() to extract the lat and lng from the script tag's contents. Demo from the scrapy shell:
$ scrapy shell index.html
>>> response.xpath('//script').re(r'new google\.maps\.LatLng\(([0-9.]+),([0-9.]+)\);')
[u'43.2640611', u'2.9388228']
where index.html contains:
<script type="text/javascript">
var map;
function initialize() {
var fenway = new google.maps.LatLng(43.2640611,2.9388228);
};
}
</script>
Of course, in your case the xpath would not be just //script.
FYI, new google\.maps\.LatLng\(([0-9.]+),([0-9.]+)\); regular expression uses the saving groups ([0-9.]+) to extract the coordinate values.
Also see Using selectors with regular expressions.
Disclaimer: I haven't tried this approach, but here's how I would think about it if I was constrained to using Scrapy and didn't want to parse JavaScript the way alecxe suggested above. This is a finicky, fragile hack :-)
You can try using scrapyjs to execute the JavaScript code from your scrapy crawler. In order to capture those parameters, you'd need to do the following:
Load the original page and save it to disk.
Modify the page to replace google.maps.LatLng function with your own (see below). make sure to run your script AFTER google js is loaded.
Load the modified page using scrapyjs (or the instance of webkit created by it)
Parse the page, look for the two special divs created by your fake LatLng function that contain the extracted lat and lng variables.
More on step 2: Make your fake LatLng function modify the HTML page to expose lat and lng variables so that you could parse them out with Scrapy. Here is some crude code to illustrate:
var LatLng = function LatLng(lat, lng) {
var latDiv = document.createElement("div");
latDiv.id = "extractedLat";
latDiv.innerHtml = lat;
document.body.appendChild(latDiv);
var lngDiv = document.createElement("div");
lngDiv.id = "extractedLng";
lngDiv.innerHtml = lng;
document.body.appendChild(lngDiv);
}
google = {
map: {
LatLng: LatLng
}
};
Overall, this approach sounds a bit painful, but could be fun to try.
Related
I am using Google Apps Script to create a page, on which I would like to embed maps. The maps themselves would be static, but the map could be different depending on other parameters (it’s a genealogy page, and I’d like to display a map of birth and death locations, and maybe some other map points, based on a selected individual).
Using Google’s Maps service, I know that I can create a map, with a couple points built in.
Function getMapImage() {
var map = Maps.newStaticMap()
.setSize(600,400)
.addMarker('Chicago, Illinois') // markers would be based on a passed parm; this is just test data
.addMarker('Pocatello, Idaho');
// *** This is where I am looking for some guidance
return(); // obviously, I'm not returning a blank for real
}
Within the map class, there are a number of things I can do with it at this point.
I could create a URL, and pass that back. That appears to require an API account, which at this point, I do not have (and ideally, would like to avoid, but maybe I’ll have to do that). It also appears that I will run into CORB issues with that, which I think is beyond my knowledge (so if that’s the solution, I’ll be back for more guidance).
I could create a blob as an image, and pass that back to my page. I have tried this using a few different examples I have found while researching this.
Server Side
function getMapImage() {
var map = Maps.newStaticMap()
.setSize(600,400)
.addMarker('Chicago, Illinois')
.addMarker('Pocatello, Idaho');
var mapImage = map.getAs("image/png");
// OR
// var mapImage = map.getBlob();
return(mapImage);
}
Page side
<div id=”mapDiv”></div>
<script>
$(function() {
google.script.run.withSuccessHandler(displayMap).getMapImage();
}
function displayMap(mapImage) {
var binaryData = [];
binaryData.push(mapImage);
var mapURL = window.URL.createObjectURL(new Blob(binaryData, {type: "image/png"}))
var mapIMG = "<img src=\'" + mapURL + "\'>"
$('#mapDiv').html(mapIMG);
}
</script>
The page calls getMapImage() on the server, and the return data is sent as a parm to displayMap().
var mapIMG ends up resolving to <img src='blob:https://n-a4slffdg23u3pai7jxk7xfeg4t7dfweecjbruoa-0lu-script.googleusercontent.com/51b3d383-0eef-41c1-9a50-3397cbe83e0d'> This version doesn't create any errors in the console, which other options I tried did. But on the page, I'm just getting the standard 16x16 image not found icon.
I’ve tried a few other things based on what I’ve come across in researching this, but don’t want to litter this post with all sorts of different code snippets. I’ve tried a lot of things, but clearly not the right thing yet.
What’s the best / correct (dare I ask, simplest) way to build a map with Google’s Map class, and then serve it to a web page?
EDIT: I added a little more detail on how the server and page interact, in response to Tanaike's question.
Modification points:
I think that in your script, Blob is returned from Google Apps Script to Javascript using google.script.run. Unfortunately, in the current stage, Blob data cannot be directly sent from from Google Apps Script to Javascript. I think that this might be the reason of your issue.
In this case, I would like to propose to directly create the data URL at the Google Apps Script side. When your script is modified, it becomes as follows.
Modified script:
Google Apps Script side:
function getMapImage() {
var map = Maps.newStaticMap()
.setSize(600, 400)
.addMarker('Chicago, Illinois')
.addMarker('Pocatello, Idaho');
var blob = map.getAs("image/png"); // or map.getBlob()
var dataUrl = `data:image/png;base64,${Utilities.base64Encode(blob.getBytes())}`;
return dataUrl;
}
Javascript side:
$(function() {
google.script.run.withSuccessHandler(displayMap).getMapImage();
});
function displayMap(mapURL) {
var mapIMG = "<img src=\'" + mapURL + "\'>"
$('#mapDiv').html(mapIMG);
}
In your Javascript side, $(function() {google.script.run.withSuccessHandler(displayMap).getMapImage();} is not enclosed by ). Please be careful this.
Note:
In my environment, when I saw <div id=”mapDiv”></div>, this double quote ” couldn't be used. So if in your environment, an error occurs by <div id=”mapDiv”></div>, please modify ” to " like <div id="mapDiv"></div>.
Reference:
base64Encode(data)
As you can see here, I need those markers values such as lat lng address.
From that link, i tried to go to those link. But still can not inspect anything.
i tried this in Js but it did not return anything
var url = "https://www.google.com/maps/d/u/0/viewer?ll=41.042268%2C29.001695&spn=0.06531%2C0.072004&hl=en&t=m&vpsrc=6&msa=0&source=embed&ie=UTF8&mid=zsrW6PXLEpZk.kTAwhQ8bgERs" + "&sensor=false";
$.getJSON(url, function (data) {
for(var i=0;i<data.results.length;i++) {
var adress = data.results[i].formatted_address;
alert(adress);
}
});
http://jsfiddle.net/CursedChico/NNuDe/341/
How can i get? I am used to Js, angularjs, java and android.
You can't get markers data from that url you provided, it uses kml format to get values. Anyways, you want to get location, address etc. Basically you want informations of all markers. Here is the solution:
1) Get KML file of the map.
What is KML data?
KML is a file format used to display geographic data in an Earth
browser such as Google Earth
In your link, you can download it by clicking left menu->download kml->export to kml and you have the kml file. Than parse the kml data into json using a parser like this.
2) Check if the company let you use JSONP
If the page you are looking for have JSONP set, you can easily retrieve data by injecting JSONP link into your HTML as a script and get than using the callback function. Here is a working project which gets lat and long datas by calling JSONP.
function setJSONP(code) {
var script = document.createElement('script');
script.src = "http://www.nike.com/store-locator/locations?jsoncallback=callback" +
"&country_code=US" +
"&format=JSON" +
"&type=country" +
"&_=1461335132869";
document.getElementsByTagName('head')[0].appendChild(script);
}
?jsoncallback=yourFunctionName
then use your function, in this example callback() is our function.
function callback(data) {
//use data here which is returned JSON from server
var lat = data.lat;
...
}
I am a bit confused on how to go about developing an application, using Google maps javascript API + the 'Places' library.
What I want to have:
an html page with 2 divs:
a. left div: contains the google map + results/markers based on a given location ( I have that already)
b. right div: contains a list of the results with drop-down menus to filter the results on the map, on the left div.
So my questions are:
a. The response is already loaded in an object for the map on the left div, as in this example:
https://developers.google.com/maps/documentation/javascript/examples/place-search
Do I need to load it again - in order to get this information in XML and parse it in the right div?
My xml request is for example:
https://maps.googleapis.com/maps/api/place/nearbysearch/xml?location=34.680617,33.043263&radius=500&types=store&key=AIzaSyDQ9fpqxWt0F5EsztFt-HSjcSQruJSGeik
Where would I insert this (in the above example code) in order to get the xml response and parse the file/node information in text format on the right div?
b. In the callback function there is a loop that goes through the results:
function callback(results, status) {
if (status == google.maps.places.PlacesServiceStatus.OK) {
for (var i = 0; i < results.length; i++) {
createMarker(results[i]);
}
}
}
Could I use that same loop in order to parse the xml nodes without having to load the xml file again?
If yes, the google developers site does not have an example of how to do that. It only lists a few xPath expressions:
developers.google.com/maps/documentation/webservices/index#ParsingXML
but it does not provide a full complete example from scratch.
i.e. how to use xPath to parse the xml file with javascript
c. if we need to have 2 separate responses (the google maps api + the web service), how will these 2 communicate between them then?
Not sure if I got it all wrong here...
thanks,
k.
Yes you should be able to use the same loop to parse the xml you could try something like:
function callback(results, status)
{
if (status == google.maps.places.PlacesServiceStatus.OK)
{
for (var i = 0; i < results.length; i++)
{
// Untested but use JQuery and try this:
var name = $(results[i]).find('name').text();
var location = $(results[i]).find('location');
var lat = $(location).find('lat').text();
var lng = $(location).find('long').text();
var obj =
{
name: name,
lat : lat,
lng: lng
};
createMarker(obj);
}
}
}
I would keep track of the markers your creating in an array or something to manage them on the map.
I am trying to get the address or location of a google maps marker. However when I view the source code of the website I get a weird javascript call. Is there any way to resolve this to a usable Long/Lat?
This is what I see.
<script type="text/javascript">
$(window).load(function(){
//Get spu from unit id's
var spuMap = new Array();
// initialization for property page is done in seperate function
if ($("#pdp-container").length == 0){
ha.map.property.init({location: [{a:'%32%30%2E
%35%34%35%35%32%35%39%35%34%32%31%34%37%30%37', b:'%2D%31%30%35%2E
%32%37%39%34%30%37%35%30%31%32%32%30%37', cLat:'%32%30%2E
%35%34%35%35%32%35%39%35%34%32%31%34%37%30%37', cLong:'%2D%31%30%35%2E
%32%37%39%34%30%37%35%30%31%32%32%30%37', id:'268534', exact:true, zoom:16, maxZoom:20, type:'u'}],
mType: 'property'}, spuMap);
}
});
</script>
For some strange reason these parameters are URI encoded so simply call decodeURI on cLat and cLong, eg.
decodeURI(cLat);
And when you use it make sure to convert it to a number as google api takes coordinates as numbers.
I load an AT5 file into a google map object using the following code:
map = new GMap2(document.getElementById("map_canvas"));
geoXml = new GGeoXml(at5);
GEvent.addListener(geoXml, "load", function() {
geoXml.gotoDefaultViewport(map);
// I would like to read the AT5 contents here
});
map.addOverlay(geoXml);
Thanks for any kind of help.
This does not refer to Version 3 of the Maps API. GMap2, GGeoXml and GEvent are all part of Version 2.
GGeoXml is opaque: you cannot inspect it. Use a third-party parser if you want the content to be available.
There's a rather old list available at http://econym.org.uk/gmap/extensions.htm -- have a look at EGeoXml and GeoXml.