So I learned that it's not exactly an easy task to create a progress bar for SQL queries. So I've tried to come up with an alternative solution to indicate when a task will be done. I figured I could time how long it would take to execute one query in the database and then use that information to create an estimation depending on how many queries that has to be performed and then create a progress bar from that information. I know it will not be a 100% reliable solution, because if something went wrong it would still show the progress bar, but it would at least give the user some indication when the job will be done, rather than just having a loading spinner.
Is there a better solution to this problem?
edit to answer some questions
It's the update function that takes time. I have maybe 15.000 inputs in the database and I have to update all 15.000 through an API. So first I have to pull out the id of all those and use them with an API to get the updated information, and then I have to execute all the 15.000 queries with the updated information. All this takes time. I doubt it can be done quickly. With a database of 15.000 queries it took me about 2 hours. I don't believe a spinner for 2 hours without an indication of when it will be done is realiable. I need some sort of estimation.
Note that updating the database is something that will be done very rarely. Like maybe once a week the user will update the database.
You can use reverse-ajax and then return a result into an iframe.
On the server-side, use repsonse.flush (ASP.NET) to get a partial result to the client when one iteration step is completed.
Output JavaScript code like:
window.parent.fnReportProgress(25.3)
Also, make sure you don't update on every progress message.
Instead store the messages in an array, and then poll the array every x milliseconds (setInterval) until complete. If you don't put all messages into an array, but instead update the progress bar immediately, you will get a bluescreen in IE, and only in IE. Chrome + FF work perfectly fine.
If you only need to support halfway modern browsers (IE10+) on halfway modern servers (IIS 8+), you can also use WebSockets.
WebSockets will also work on servers < IIS8, but then you'll have to develop the web-sockets server app separately, on a separate port.
Related
So, I have a main indexedDB objectstore with around 30.000 records on which I have to run full text search queries. Doing this with the ydn fts plugin this generates a second objectstore with around 300.000 records. Now, as generating this 'index' datastore takes quite long I figured it had be faster to distribute the content of the index as well. This in turn generated a zip file of around 7MB which after decompressing on the client side gives more than 40MB of data. Currently I loop over this data inserting it one by one (async, so callback time is used to parse next lines) which takes around 20 minutes. As I am doing this in the background of my application through a webworker this is not entirely unacceptable, but still feels hugely inefficient. Once it has been populated the database is actually fast enough to be even used on mid to high end mobile devices, however the population time of 20 minutes to one hour on mobile devices is just crazy. Any suggestions how this could be improved? Or is the only option minimizing the number of records? (which would mean writing my own full text search... not something I would look forward to)
Your data size is quite large for mobile browser. Unless user constantly using your app, it is not worth sending all data to client. You should use server side for full text search, while catching opportunistically as illustrated by this example app. In this way, user don't have to wait for full text search indexing.
Full-text search require to index all tokens (words) except some stemming words. Stemming is activated only when lang is set to en. You should profile your app which parts is taking time. I guess browser is taking most of the time, in that case, you cannot do much optimization other than parallelization. Sending indexed data (as you described above) will not help much (but please confirm by comparing). Also Web worker will not help. I assume your app have no problem with slow respond due to indexing.
Do you have other complaint other than slow indexing time?
I'm designing a MongoDB database that works with a script that periodically polls a resource and gets back a response which is stored in the database. Right now my database has one collection with four fields , id, name, timestamp and data.
I need to be able to find out which names had changes in the data field between script runs, and which did not.
In pseudocode,
if(data[name][timestamp]==data[name][timestamp+1]) //data has not changed
store data in collection 1
else //data has changed between script runs for this name
store data in collection 2
Is there a query that can do this without iterating and running javascript over each item in the collection? There are millions of documents, so this would be pretty slow.
Should I create a new collection named timestamp for every time the script runs? Would that make it faster/more organized? Is there a better schema that could be used?
The script runs once a day so I won't run into a namespace limitation any time soon.
OK, this is a neat question b/c the short is basically: you will have to iterate and run javascript over each item.
The part where this gets "neat" is that this isn't really different from what an SQL solution would have to do. I mean, you're basically joining a table to itself where x.1=x.1 and y.1=y.2. Even if the relational DB can handle such a beast, it's definitely not going to be fast with millions of entries.
So the truth is, you're doing this right way. Here are the extra details I would use to make this cleaner.
Ensure that you have an index on Name/Timestamp.
Run a db.mycollection.find().foreach() across the data set.
Foreach entry you're going to a) Perform comparison. b) Save appropriately. c) Update a flag indicating that this record has been processed.
On future loads you should be able to add a query to your find. db.mycollection.find({flag:{$exists:false}}).foreach()
Use db.eval() to help with speed.
The reason for the "Name/Timestamp" index is that you're going to be looking up each "successor" by "Name/Timestamp", so you want to be quick here.
The reason for the "processed" flag is that you should never have to re-run the same item. If given timestamp 'n' you find 'n+1', then that's the only 'n+1' you're going to have.
Honestly, if you're only running this once / day, it's quite likely that the speed will be just fine, especially if you only running on new records. Just assume that it's going to take several minutes.
Right guys, all autocomplete plugins and functions that I've found, they only update upon keyup/down, etc. This is fine but the search only begins occurring once the user has stopped typing and if they are typing a phrase or word, the script is unable to instantly start suggesting, etc.
I know this'll be a very simple fix or suggestion for some of you guys, so any help would be greatly appreciated as to how I can convert it to be instantly as a key is pressed.
An example of the desired effect is Google Suggest or Facebook search, a search is fired instantly per key press or change, how do I emulate this?
Thanks!
Is this what you mean? Or do you want Ajax to retrieve from a database?
var data = "Core Selectors Attributes Traversing Manipulation CSS Events Effects Ajax Utilities".split(" ");
$("#example").autocomplete(data);
JQuery
Edit: I'm not sure I know what you mean, because this example seems to work identical to Google Suggest or Facebook. If your database was small you could download the cache into the variable data upon page load. If your database was slightly larger you LIMIT the cache to only X number of responses for each alphabetical character or series of characters. (ie. WHERE city LIKE 'aa%' LIMIT 10 AND WHERE...)
It depends on how big the space you're searching is and how good your servers are. Facebook search for (I assume people's names) is quick because you're only really searching through a thousand or so contacts. Google is fast because they invest a lot of money in infrastructure and cache a lot of the responses.
On one of my projects I've used this jQuery plugin and it provides excellent performance on cached results. We used it to provide autocomplete functionality on a list of about 6K contacts (names, etc). Is this what you had in mind?
The Wicket web framework has the concept of a "throttling" behavior. Normally, AJAX requests in Wicket applications are queued against an "ajax channel", which triggers a request instantly if none is running. If a request is already running, the next request is queued, and triggered when the current one returns.
"Throttling" lets the behavior delay itself for a certain amount of time (say, two seconds). If the behavior fires again in the same period, the callback for the most recent behavior replaces the callback for the current queued behavior. (For example, the user starts typing "albuquerque", which triggers the events "A" then "AL", then "ALB". The system might trigger "A", then "ALB", skipping over "AL" because it was replaced by "ALB" while sitting in the queue.) The object of this is to fire a behavior instantly on each keypress, but prevent the server from being flooded with unnecessary requests.
Check out the wicket ajax source code:
https://github.com/apache/wicket/blob/wicket-1.4.8/wicket/src/main/java/org/apache/wicket/ajax/wicket-ajax.js
For more about the web framework, see:
http://wicket.apache.org
Say we have users A and B which visit the same URL containing a button. When A clicks on the button, I want something on B's website to change immediately while B is on it, e.g. a text to be added. I want this to happen with a delay of less than 150ms.
Is this realistic? Could you give me hints as to what I should search for, or toy examples which illustrate this? Thanks.
I think you should take a look at a Push/Comet server. A very popular one right now is NGINX's push module: http://pushmodule.slact.net/
This is how you can create chat room for example. At least that is what it sounds like you explained.
****update****
As for your latency question, I don't think 150ms is realistic, you realize that it is a full round trip at least plus a DB read and write. Polling will not give you a very snappy experience for the user, this is because your JS might decide to send it's response right before the user completes the action and you'd have to wait until your JS sends the request again for the user "B" to see the update. This could be a long time, maybe like 10 seconds? You wouldn't to use polling in my opinion because it's very wasteful, and makes cacheing pretty tough as well.
I'd go with push. Unfortunately Apache doesn't have a reliable push service like Nginx.
There are 2 main approaches to this:
You can make ajax queries asking if the state has changed every, say, 5 seconds.
HTTP Streaming
This article lists 2 more approaches: http://www.infoq.com/news/2007/07/pushvspull
You can do this easily with php and mysql or some kind of database. Is there something preventing you from using a database? If so you can write to files using php which would let you store these values for user A and B.
I was planning to use jquery autocomplete for a site and have implemented a test version. Im now using an ajax call to retrieve a new list of strings for every character input. The problem is that it gets rather slow, 1.5s before the new list is populated. What is the best way to make autocomplete fast? Im using cakephp and just doing a find and with a limit of 10 items.
This article - about how flickr does autocomplete is a very good read. I had a few "wow" experiences reading it.
"This widget downloads a list of all
of your contacts, in JavaScript, in
under 200ms (this is true even for
members with 10,000+ contacts). In
order to get this level of
performance, we had to completely
rethink how we send data from the
server to the client."
Try preloading your list object instead of doing the query on the fly.
Also the autocomplete has a 300 ms delay by default.
Perhaps remove the delay
$( ".selector" ).autocomplete({ delay: 0 });
1.5-second intervals are very wide gaps to serve an autocomplete service.
Firstly optimize your query and db
connections. Try keeping your db connection
alive with memory caching.
Use result caching methods if your
service is highly used to ignore re-fetchs.
Use a client cache (a JS list) to keep the old requests on the client. If user types back and erases, it is going to be useful. Results will come from the frontend cache instead of backend point.
Regex filtering on the client side wont be costly, you may give it a chance.
Before doing some optimizations you should first analyze where the bottle-neck is. Try to find out how long each step (input → request → db query → response → display) takes. Maybe the CakePHP implementation has a delay not to send a request for every character entered.
Server side on PHP/SQL is slow.
Don't use PHP/SQL. My autocomplete written on C++, and uses hashtables to lookup. See performance here.
This is Celeron-300 computer, FreeBSD, Apache/FastCGI.
And, you see, runs quick on huge dictionaries. 10,000,000 records isn't a problem.
Also, supports priorities, dynamic translations, and another features.
The real issue for speed in this case I believe is the time it takes to run the query on the database. If there is no way to improve the speed of your query then maybe extending your search to include more items with a some highly ranked results in it you can perform one search every other character, and filter through 20-30 results on the client side.
This may improve the appearance of performance, but at 1.5 seconds, I would first try to improve the query speed.
Other than that, if you can give us some more information I may be able to give you a more specific answer.
Good luck!
Autocomplete itself is not slow, although your implementation certainly could be. The first thing I would check is the value of your delay option (see jQuery docs). Next, I would check your query: you might only be bringing back 10 records but are you doing a huge table scan to get those 10 records? Are you bringing back a ton of records from the database into a collection and then taking 10 items from the collection instead of doing server-side paging on the database? A simple index might help, but you are going to have to do some testing to be sure.