So I have a database with a bunch of google maps markers and I give users the ability to 'follow' markers. This 'following' information is stored in a separate relational database.
When I load my map and all the associated markers, I would like each marker to display an infowindow (I suspect you all know what this looks like) showing how many people are following the marker. i.e. "23 people are following this marker"
The problem with this would seem to be that if I count the relational database 'followers' while loading the markers, this could be extremely slow, given 1000+ markers. Instead I am thinking of making a marker counter and including it in the original marker database, so I only have to query one database.
Having never made a counter as such (and not finding many specifics on the web), I am wondering if there are any pitfalls I should be aware of? I can see problems arising when users delete their accounts, as I would have to update the counter based on that (not impossible, just kind of a pain), but this could get complicated if I allowed the users to add back their account (as with facebook or something).
Would there be any way for that counter to count the relational database entries periodically so I do no have to update the counter?
from a noob, sincere thanks for any help! It is greatly appreciated.
Databases are pretty good at counting, quickly! The recommended approach is to not store calculated values in your operational database. Try it that way first, measure performance, if it's the slowest part of your app, and there are significant costs associated with that, then start storing counts.
If you need to store counts, then consider using caching on your web server, or materialized views and re-generating the counts as required.
Related
On my web-app, the user can request different data lines. Those data lines have an unique "statusID" each, say "18133". When the user requests to load the data, it is either loaded from the server or from indexedDB(that part im trying to figure out). In order to make it as fast as possible, I want the index to be the timestamp of the data, as I will request ranges which are smaller than the actual data in the indexedDB. However, I am trying to figure out how to create the stores and store the data properly. I tried to dynamically create stores everytime data with a new Id is requested, but creating stores is only possible in "onupradeneeded". I also thought about storing everything in the same store, but I fear that the performance will make that bad. I do not really now how to approach this thing.
What I do know: If you index a value, it means that the data is sorted, which is exactly what I want. I dont know if the following is possible but this would solve my issue too: store everthing in the same store, index by "statusID" and index by "timestamp". This way, it would be fast too i guess.
Note that I am talking about many many datapoints, possible in the millions.
You can index by multiple values, allowing you to get all by statusID and restricting to a range for your timestamp. So I'd go with the one datastore solution. Performance should not be an issue.
This earlier post may be helpful: Javascript: Searching indexeddb using multiple indexes
Good evening,
my project uses the MEAN Stack and has a few collections and a single database from which the data is retrieved.
Thinking about how the user would interface itself with the webapp I am going to build, I figured that my idea of the application is quite a bit of a waste.
Now, the application is hosted on a private server on the LAN, making it very fast on requests and it's running an express server.
The application is made around employee management, services and places where the services can take place. Just describing, so to have an idea.
The "ring to rule them all" is pretty much the first collection, services, which starts the core of the application. There's a page that let's you add rows, one for each service that you intend to do and within that row you choose an employee to "run the service", based on characteristics that this employee has, meaning that if the service is about teaching Piano, the employee must know how to play Piano. The same logic works for the rest of the "columns" that will build up my row into a full service recognized by the app as such.
Now, what I said above is pretty much information retrieval from a database and logic to make the application model the information retrieved and build something with it.
My question or rather my doubt comes from how I imagined the querying to work for each field that is part of the service row. Right now I'm thinking about querying the database (mongodb) each time I have to pick a value for a field, but if you consider that I might want to add a 100 rows, each of which would have 10 fields, that would make up for a lot of requests to the database. Now, that doesn't seem elegant, nor intelligent to me, but I can't come up with a better solution or idea.
Any suggestions or rule of thumbs for a MEAN newb?
Thanks in advance!
EDIT: Answer to a comment question which was needed.
No, the database is pretty static (unless the user willingly inserts a new value, say a new employee that can do a service). That wouldn't happen very often. Considering the query that would return all the employees for a given service, those employees would (ideally) be inside an associative array, with the possibility to be "pop'd" from it if chosen for a service, making them unavailable for further services (because one person can't do two services at the same time). Hope I was clear, I'm surely not the best person at explaining oneself.
It would query the database on who is available when a user looks at that page and another query if the user assigns an employee to do a service.
In general 1 query on page load and another when data is submitted is standard.
You would only want to use an in memory cache for
frequent queries but most databases will do this automatically.
values that change frequently like:
How many users are connected
Last query sent
Something that happens on almost every query (>95%)
I am developing a web app based on the Google App Engine.
It has some hundreds of places (name, latitude, longitude) stored in the Data Store.
My aim is to show them on google map.
Since they are many I have registered a javascript function to the idle event of the map and, when executed, it posts the map boundaries (minLat,maxLat,minLng,maxLng) to a request handler which should retrieve from the data store only the places in the specified boundaries.
The problem is that it doesn't allow me to execute more than one inequality in the query (i.e. Place.latminLat, Place.lntminLng).
How should I do that? (trying also to minimize the number of required queries)
You could divide the map into regions, make an algorithm to translate the current position into a region, and then get the places by an equality query. In this case you would need overlapping regions, allow places to be part of many of them, and ideally make regions bigger than the size of the map, in order to minimize the need for multiple queries.
That was just an outline of the idea, an actual implementation would be a little bit more complicated, but I don't have one at hand.
Another option is using geohashes, which are actually pretty cool, you can read a write up about them, along with code samples, here: Scalable, fast, accurate geo apps using Google App Engine + geohash + faultline correction
You didn't say how frequently the data points are updated, but assuming 1) they're updated infrequently and 2) there are only hundreds of points, then consider just querying them all once, and storing them sorted in memcache. Then your handler function would just fetch from memcache and filter in memory.
This wouldn't scale indefinitely but it would likely be cheaper than querying the Datastore every time, due to the way App Engine pricing works.
Is it possible to query a UK Postcode Database that is available offline and gather Geo Coordinates with JavaScript?
I am under the impression that the database is too vast to do this without online/local database management.
Has anyone found a solution for this problem?
Thanks.
EDIT:
The data will need to be used by a private organisation and does not need to be either hugely accurate nor updated regularly. (Basically the user needs to find the nearest location to their specific postcode - < 10 locations). I Think my concern is seeing the 550+mb databases.
It also needs to work purely in the browser offline using JavaScript. JSON format would be preferred.
This is possible and there are few choices but first a few questions.
To what level of accuracy do you need the coordinates
The more exact, the more it will cost
How much time and effort do you want to put into long term maintenance
The data is regularly updated so make sure you have a business process in place
Are you building a solution for an existing company, or trying something out as a startup?
It is a large amount of data, but is manageable. The practicalities depend more upon the amount of time and money you want to invest.
There are two approaches:
1) Build your own
If needing premise level coordinates, you'll need to get a copy of the postcode file from the Royal Mail here and depending upon your level of accuracy you'll then need to also grab a supply of coordinate information from someone such as Ordnance Survey.
Load and link together the supplies into some sort of database or other store you are comfortable working in and make sure this process is easily repeatable once a month when the data comes through.
For just postcode level, I think the Code-point Open data should be good for you. If you are not worried about individual premises and postcode level is good enough. That comes to about 150MB in CSV files but you can get it a lot smaller than that by just storing the data you need.
The next step and develop a look up mechanism. The quality of your look up will depend upon your use case. If you are just after a rough location from the postcode this will be simpler than if you are after it down to the nearest cm and need a full premise address.
2) Go third party.
There are many companies around that can do this for you. My company - Experian Data Quality - provide tools to allow you to do this offline as a library or SOAP based service.
Once integrated, they will allow your users to quickly and easily find their correct address and then in the same call enhance the address with location (and many other types) of information.
This will cope with typos, data updates etc. Fill in the form or give us a call for a demo if sounds useful http://www.qas.co.uk/solutions/data-quality-software/data-capture/address-checker.htm
There is sample integration code available that will allow you to hook it into your website with JavaScript.
You can buy the database from the Royal Mail:
http://www.poweredbypaf.com/end-user/products/data-products/paf-raw-data/
This is the official database from the Royal Mail and can be downloaded and used offline.
Are there any best practices for returning large lists of orders to users?
Let me try to outline the problem we are trying to solve. We have a list of customers that have 1-5,000+ orders associated to each. We pull these orders directly from the database and present them to the user is a paginated grid. The view we have is a very simple "select columns from orders" which worked fine when we were first starting but as we are growing, it's causing performance/contention problems. Seems like there are a million and one ways to skin this cat (return only a page worth of data, only return the last 6 months of data, etc.) but like I said before just wondering if there are any resources out there that provide a little more hand holding on how to solve this problem.
We use SQL Server as our transaction database and select the data out in XML format. We then use a mixture of XSLT and Javascript to create our grid. We aren't married to the presentation solution but are married to the database solution.
My experience.
Always set default values in the UI for the user that are reasonable. You don't want them clicking "Retrieve" and getting everything.
Set a limit to the number of records that can be returned.
Only return from the database the records you are going to display.
If forward/backward consistencency is important, store the entire results set from the query in a temp table and return just the page you need to display. When paging up/down retrieve the next set from the temp table.
Make sure your indexs are covering your queries.
Use different queries for different purposes. Think "Open Orders" vs "Closed Orders". These might perfrom much better as different queries instead of one generic query.
Set parameter defualts in the stored procedures. Protect your query from a UI that is not setting reasonable limits.
I wish we did all these things.
I'd recommend doing some profiling to find the actual bottlenecks. Perhaps you have access to Visual Studio Profiler? http://msdn.microsoft.com/en-us/magazine/cc337887.aspx There are plenty of good profilers out there.
Otherwise, my first stop would be pagination to bring back less records from the db, which is easier on the connection and the memory footprint. Take a look at this (I'm assuming you're on SQL Server >= 2005)
http://www.15seconds.com/issue/070628.htm
I"m not sure from the question exactly what UI problem you are trying to solve.
If it's that the customer can't work with a table that is just one big amorphous blob, then let him sort on the fields: order date, order number, your SKU number, his SKU number maybe, and I guess others,too. He might find it handy to do a multi-column stable sort, too.
If it's that the table headers scroll up and disappears when he scrolls down through his orders, that's more difficult. Read the SO discussion to see if the method there gives a solution you can use.
There is also a JQuery mechanism for keeping the header within the viewport.
HTH
EDIT: plus I'll second #Iain 's answer: do some profiling.
Another EDIT: #Scott Bruns 's answer reminded me that when we started designing the UI, the biggest issue by far was limiting the number of records the user had to look at. So, yes I agree with Scott that you should give the user some way to see only a limited number of records right from the start; that is, before he ever sees a table, he has told you a lot about what he wants to see.
Stupid question, but have you asked the users of your application for input on what records that they would like to see initially?