Is it possible to query a UK Postcode Database that is available offline and gather Geo Coordinates with JavaScript?
I am under the impression that the database is too vast to do this without online/local database management.
Has anyone found a solution for this problem?
Thanks.
EDIT:
The data will need to be used by a private organisation and does not need to be either hugely accurate nor updated regularly. (Basically the user needs to find the nearest location to their specific postcode - < 10 locations). I Think my concern is seeing the 550+mb databases.
It also needs to work purely in the browser offline using JavaScript. JSON format would be preferred.
This is possible and there are few choices but first a few questions.
To what level of accuracy do you need the coordinates
The more exact, the more it will cost
How much time and effort do you want to put into long term maintenance
The data is regularly updated so make sure you have a business process in place
Are you building a solution for an existing company, or trying something out as a startup?
It is a large amount of data, but is manageable. The practicalities depend more upon the amount of time and money you want to invest.
There are two approaches:
1) Build your own
If needing premise level coordinates, you'll need to get a copy of the postcode file from the Royal Mail here and depending upon your level of accuracy you'll then need to also grab a supply of coordinate information from someone such as Ordnance Survey.
Load and link together the supplies into some sort of database or other store you are comfortable working in and make sure this process is easily repeatable once a month when the data comes through.
For just postcode level, I think the Code-point Open data should be good for you. If you are not worried about individual premises and postcode level is good enough. That comes to about 150MB in CSV files but you can get it a lot smaller than that by just storing the data you need.
The next step and develop a look up mechanism. The quality of your look up will depend upon your use case. If you are just after a rough location from the postcode this will be simpler than if you are after it down to the nearest cm and need a full premise address.
2) Go third party.
There are many companies around that can do this for you. My company - Experian Data Quality - provide tools to allow you to do this offline as a library or SOAP based service.
Once integrated, they will allow your users to quickly and easily find their correct address and then in the same call enhance the address with location (and many other types) of information.
This will cope with typos, data updates etc. Fill in the form or give us a call for a demo if sounds useful http://www.qas.co.uk/solutions/data-quality-software/data-capture/address-checker.htm
There is sample integration code available that will allow you to hook it into your website with JavaScript.
You can buy the database from the Royal Mail:
http://www.poweredbypaf.com/end-user/products/data-products/paf-raw-data/
This is the official database from the Royal Mail and can be downloaded and used offline.
Related
Currently, I have to fill out 2 of the same forms with the same language & part numbers on the Web Based business system "Netsuit" made by Oracle (extremely annoying & waste of time). I need to use a software/code system to read one form entry and duplicate it to the other automatically, just still feeling out the best way to do this and get it to transfer/skim properly.
This is between 2 sister companies, each value(Part) has a different part number linked to them, but internally they cannot be linked due to reporting purposes and which company sales what.
One company starts with 100XXX-XX numbers and the other starts with 300XXX-XX numbers for the parts. Again, they are basically the same Parts.
Not sure if Tampermonkey or java will be able to do this properly as I don't even know where to start.
Any recommendations or walkthough on the best way to do this would be awesome, I know it might be a little hard since its 2 different item systems.
Maybe just pull the description of the items since they will be almost the same?
You can create a user event script on the first company and RESTlet on the second one.
The user event script on the first company will create a JSON object of the item that is being created and pre-process the changes in the part number(or any other changes that is required) for the second company and send it to the second company's RESTlet. The RESTlet will now then create the item for the second company.
By using this approach you don't need any 3rd party application to deal with.
Good evening,
my project uses the MEAN Stack and has a few collections and a single database from which the data is retrieved.
Thinking about how the user would interface itself with the webapp I am going to build, I figured that my idea of the application is quite a bit of a waste.
Now, the application is hosted on a private server on the LAN, making it very fast on requests and it's running an express server.
The application is made around employee management, services and places where the services can take place. Just describing, so to have an idea.
The "ring to rule them all" is pretty much the first collection, services, which starts the core of the application. There's a page that let's you add rows, one for each service that you intend to do and within that row you choose an employee to "run the service", based on characteristics that this employee has, meaning that if the service is about teaching Piano, the employee must know how to play Piano. The same logic works for the rest of the "columns" that will build up my row into a full service recognized by the app as such.
Now, what I said above is pretty much information retrieval from a database and logic to make the application model the information retrieved and build something with it.
My question or rather my doubt comes from how I imagined the querying to work for each field that is part of the service row. Right now I'm thinking about querying the database (mongodb) each time I have to pick a value for a field, but if you consider that I might want to add a 100 rows, each of which would have 10 fields, that would make up for a lot of requests to the database. Now, that doesn't seem elegant, nor intelligent to me, but I can't come up with a better solution or idea.
Any suggestions or rule of thumbs for a MEAN newb?
Thanks in advance!
EDIT: Answer to a comment question which was needed.
No, the database is pretty static (unless the user willingly inserts a new value, say a new employee that can do a service). That wouldn't happen very often. Considering the query that would return all the employees for a given service, those employees would (ideally) be inside an associative array, with the possibility to be "pop'd" from it if chosen for a service, making them unavailable for further services (because one person can't do two services at the same time). Hope I was clear, I'm surely not the best person at explaining oneself.
It would query the database on who is available when a user looks at that page and another query if the user assigns an employee to do a service.
In general 1 query on page load and another when data is submitted is standard.
You would only want to use an in memory cache for
frequent queries but most databases will do this automatically.
values that change frequently like:
How many users are connected
Last query sent
Something that happens on almost every query (>95%)
So I have a database with a bunch of google maps markers and I give users the ability to 'follow' markers. This 'following' information is stored in a separate relational database.
When I load my map and all the associated markers, I would like each marker to display an infowindow (I suspect you all know what this looks like) showing how many people are following the marker. i.e. "23 people are following this marker"
The problem with this would seem to be that if I count the relational database 'followers' while loading the markers, this could be extremely slow, given 1000+ markers. Instead I am thinking of making a marker counter and including it in the original marker database, so I only have to query one database.
Having never made a counter as such (and not finding many specifics on the web), I am wondering if there are any pitfalls I should be aware of? I can see problems arising when users delete their accounts, as I would have to update the counter based on that (not impossible, just kind of a pain), but this could get complicated if I allowed the users to add back their account (as with facebook or something).
Would there be any way for that counter to count the relational database entries periodically so I do no have to update the counter?
from a noob, sincere thanks for any help! It is greatly appreciated.
Databases are pretty good at counting, quickly! The recommended approach is to not store calculated values in your operational database. Try it that way first, measure performance, if it's the slowest part of your app, and there are significant costs associated with that, then start storing counts.
If you need to store counts, then consider using caching on your web server, or materialized views and re-generating the counts as required.
I'm looking into some options for a mapping project (web, probably HTML5 Canvas based). Ideally, we are creating a navigational map (i.e. Google Maps) that would have our own very simple imagery (not satellite), but we are looking into ways of getting state / city lines, and maybe other important features. Does anyone know of a good database or project that contains this data correlated to longitude and latitude? Or are we looking at potentially drawing these lines ourselves?
There is a huge amount of choice available, but it really does depend on exactly what your looking for, what level of detail and which countries.
One of the first places you'll want to look is open geo:
http://opengeo.org/
These guys have links to all the open source geographic stuff that's available more or less.
second to that is the OS-Geo consortium:
http://www.osgeo.org/
Again these guys maintain pre-made kits and links to the majority of data & applications available.
Once you go beyond here, then you likely will need to start looking at country organisations like the USGS:
http://www.usgs.gov/pubprod/
and
http://egsc.usgs.gov/isb/pubs/factsheets/worldmaps.html
you'll most likely also want to take a look at some of the country specific data sets available such as the UK's ordnance survey:
http://www.ordnancesurvey.co.uk/oswebsite/products/os-opendata.html
and the USA Tiger data sets:
http://www.census.gov/geo/www/tiger/
finally, you could take a look at the likes of "Open street map" (And other similar products)
http://www.openstreetmap.org/
Open street map originally set out to cover the UK, but seems to be going at least Europe wide now (Maybe even world wide)
This is all I can remember off the top of my head at the moment, but one other tip I'll give, sites belonging to the top GIS software providers such as ARC-GIS & Mapinfo all have lists of mapping data providers on their websites, and some even have free data available too, an afternoons browsing should uncover you a lot of sources.
http://openstreetmap.org should have what you need.
Apologies in advance for a long question: I do want to give all the relevant information.
In our (quite large) web application, we have a generic code for entering addresses (there could be a number of different addresses: business address, users' address, online shop delivery address, etc.) The addresses can be anywhere in the world, although the site itself is in English (and for now we have no plans to change this aspect). The standard address has these fields:
Street address
City
State/County/Province
Postal/ZIP code
Country
Some fields are optional, of course (e.g. there are no postcodes in Republic of Ireland, for example and there are no state/county/province division in many countries). The issue we're having is exactly with the state/county/province field: as it can be anywhere in the world, we are currently using <input type='text'/> for this field. However now users put anything they feel like into it - and we don't even have unified values for where they should be (e.g. for Boston, Massachusetts, some user put MA, some put Mass, some put Massachusetts, some put Middlesex county, Ma, and so on - I'm not even talking about all the misspellings). This makes any statistics by geography almost useless.
To mitigate this issue, we're moving to a different way of entering addresses: the user must select the country first, then based on the country selection we will display a dropdown <select> element with the list of states, counties, provinces, etc. valid for that country. If the country doesn't have this division (as far as our system is aware), then we revert back to the plain text field. So far, so good.
Now, for the actual question. We have a table in a DB that contains this county/state/province/etc division per country. The volume of data is not large: at present, 7 countries with 262 counties/states/provinces across all of them (i.e. total 262 rows in the table). I'm sure this will grow, but not hugely. There are two ways to handle this:
Pre-load all this data, put it into global javascript variables and in the onchange of the dropdown for the country update the corresponding dropdown for the state/county/province.
Use AJAX in the onchange for the country dropdown to load the country-specific list from the database as/when it's needed.
Which option, in your opinion, is the better one (preferably with some reasoning as to why)?
I would pre-load the data: have all the data in a javascript file (as JSON, for instance), minify it, gzip it and send it to the client. This is text data. Should not take much of the bandwidth.
I would use Ajax only for really dynamic items, or when you have a lot of data and you cannot load all of the data on load (because the volume is large and the user is going to use only a subset of them).
If you are really worried about the bandwidth: you have users that may not have a high speed connection, users that use a mobile phone to access your site etc, then you can detect a slow connection and fallback to Ajax.
Assuming your audiences are desktop users, I would tend to go for the first option, as the size of data you're loading doesn't seem to be too large. You can even put more intelligence into it to improve the user experience, such as loading the data in batches, in order of popularity.
If you are targeting mobile users, then you may want to lean towards the second option.
Either case, you will want to try it out yourself with the slowest connection speed you may be expecting, making sure you are delivering acceptable user experience even for the worst case. The chances are you will end up choosing an option somewhere between those two you listed: pre-load some data, and load the others in the background and/or on-demand.
I'd load the datas once the page is loaded. Your page won't suffer from decreased loading time, and the datas will be ready when the onchange event will be fired.