For my school project, I need the data on this website:
https://services2.hdb.gov.sg/web/fi10/emap.html
Specifically, I need the longitude and latitude, address, flat type and the number of units of the flat type in csv or excel format so that I can do some spatial analysis.
How do I create an automated process to get all the data from the website and not clicking all the individual buildings?
Appreciate any assistance on this. Thank you.
I got a look at the network traffic when this site is querying it's data. There you can find the XHR-Requests where also the streetname and other things are written down. You may use this to collect your data.
Related
I have created a React Native app and want a function where the user can specify in which city they live so that it can be shown to other users. It should be a text input where the user can search for a city and get a dropdown of choices to choose from. The city should then preferably be saved as an ID in our database that can be converted back to a localized version of the city for all users. Let's say someone lives in the UK it should say Gothenburg and if you live in Sweden it should say Göteborg. I have looked at the following options but none is optimal for us:
Google Places Autocomplete API to fetch the city, save the place ID in our database, and reverse geocode it. This becomes very expensive for us because it is a social app and each user would do approximately 20 geocoding requests when starting the app
Create a database with names of cities. This is not very scalable as you need to add new entries for each new country you want to support
Let users manually enter their city. This is prone to errors because users can write whatever they want
I would be happy to receive some suggestions on how to solve this issue.
Here are the things you currently want:
Provide your users an Autocomplete functionality
Get the city based on their selection
Save the city and store it on your database
provide users a localize text of the city they chose
For the third one, storing the city provided by the API may not be allowed by Google. I can see that you are aware of the terms, but here is the Service Specific Terms of what is allowed to be cached.
Depending on your use case tho, this can still be doable. In terms of pricing and efficiency, see this pricing sheet for your reference. Right now, you are doing Places Autocomplete and Reverse Geocoding the selection to get the City. You are actually paying a little more. Assuming that you are even using per session autocomplete, this is what you are paying for every 1000 request:
Autocomplete without Places Details - Per Session = $17.00 +
Geocoding API = $5.00
Total Cost: $22 per thousand calls
So instead of using Reverse Geocoding, you can instead use Place Details which serves the same purpose. The city can be retrieve on a Place Detail request which is under the Basic Data, with the Address Component field. See this link for the list of fields Place Details can retrieve. When retrieving fields with Basic Data, those are free of charge, unlike Contact and Atmosphere Data. So make sure to make use of the "fields" parameter for the Place Detail request that you will use as if you did not specify that, all Data will be retrieve and you will be charged accordingly. So if you use that with Autocomplete, your cost now will be:
Autocomplete (included with Places Details) - Per Session + Places Details + Basic Data = $17.00 per thousand calls
Now that you have the City, you would now want to present it in the customer's local language. Place Details contains the optional parameter "language". And by simply getting the customer's current language in the device, you can achieve this. Here is how you can get it for your reference.
I would like to know what is the best way to get the distance between 2 (zip code) or postal code in Canada. I have seen post about MySQL doing so with US zip code.
Or I guess that I could probably translate a zip code to coordinates and then triangulate distances? If anyone can share tips on this I would apreciate!
Im working with Angular JS and Node JS but I could do with PHP as well.
An Api should be free and be able to handle large ammount of querys.
Thank You
Google Maps API can do it, by setting origin and destination on the distancematrix API.
https://maps.googleapis.com/maps/api/distancematrix/json?units=imperial&origins=M4C4Y7&destinations=H1A0A2
I would like to dynamically create a corresponding JSON file every time a new merchant signs up to my site.
For example:
Burger King signs up to be a merchant on my site. I add Burger king to my merchants.json file. How would I dynamically create a file that gets inserted into that JSON object that can later be used to pull up data specific to that merchant, on that merchants page. For example, a JSON file full of products or deals.
Is this even the right way to go about it?
Can someone point me in the right direction please?
This seems like a very common usage scenario but I don't see any examples online that explain this application structure thoroughly.
NOTE: I am using AngularJS
EDIT: Thanks for the tips guys, after asking around in the #AngularJS channel on IRC, I've been told to go the extra mile and create an API. I'll probably use sails.js to help with that. This obviously isn't how I was planning to do things, but as you guys pointed out, it wasn't the best practice; not by a long shot.
1) You'd need a small server-side PHP script that accepts the new JSON file sent by the client
2) Browser requests merchants.json from the server
3) Load it with JSON.parse()
4) Add the merchant to the Object
4) JSON.stringify(object)
5) Send back to the server.
Now, this is a horrible horrible idea. Use a server-side database for storing any kind of information on your clients -- MySQL, whatever. JSON is for transporting data, not storing it. You can write some PHP scripts to dynamically generate a page for your merchant based on the data in the database -- so much easier and so much more secure. The method above will expose the whole client database to the client, and based on your specifications above, I don't see another way.
I've tried a few different ways trying to get this to work. So essentially I have a list of parks that I pull with one $http get method, each park object has a weather field inside of it. After the initial load I am trying to parse non-restful .json files with the weather status (pulled by a background process server side) and do an association with the weather object based on zipcode.
So there are about 30 parks. The way I pulled the weather info on the background (so as to not hit the rate limit) was to just pull them for each zipcode (relative to park location) and place them in a file named by zipcode. So for example 90201.json, 08291.json, etc
I've tried doing something like this so far: http://plnkr.co/edit/66BjKESPnYGWgQwOzEPN?p=info
But I think I'm probably messing up in a few places, any advice would be greatly appreciated.
Thanks!
I've been assigned a research project to enhance social networking based adaptive e-learning and to do so I need to be able to extract several (hundreds, maybe thousands) of status updates or tweets in order to perform factor analysis on key words. Apparently this can be done with javascript but I have never used javascript before so I'm a bit lost. I know I need a Twitter API but not sure even how to use one. Anybody have any idea how I can do this?
use statuses/followers for getting all followers of a user and statuses/friends_timeline to get tweets by your friend. Response will be in JSON or XML format which can be parsed and used very easily.
http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-statuses%C2%A0followers
http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-statuses-friends_timeline