I'm making an application in which I need to generate Unique IDs.
When generating IDs the best way to avoid clashes, is it simply brute force generate-then-check'ing, or is there a way to garuantee unique generation.
I'm sure the brute force method would do well for a while however I have a feeling companies like Google aren't using this method.
The other things is how to actually generate them in node.js.
I thought of generating int's from '0 - 63' and then parsing to base-64 like this:
var map = {
0:0, 1:1 ... 16:"Q" ... 63:"/"
}; /* I used object so you can see the indexes,
this would be an array - or even a string? */
for (var i = IDLENGTH, id=""; i--;) id+=map[~~(Math.random*64)];
However this seems inneficient, especially having the map in the first place.
I saw this peice of code in another SO post
console.log(new Buffer("Hello World").toString('base64'));
> SGVsbG8gV29ybGQ=
Which seems to make more sense, but this doesn't seem to fit what I need, isn't this conversion of char-sets?
This is quite an open question, but you probably want to generate an UUID. Basically it's a random 128 bit number. If done correctly, the probability for a duplicate is extremely low. I believe for most applications it's not strictly needed to check, but you might still want to do it.
Use a well tested library instead of implementing this yourself, I recommend node-uuid. Generating good random numbers is very similar to crypto operations - it's not trivial to get right.
Related
I'm working on a MongoDB database and so far have stored some information as Numbers instead of Strings because I assumed that would be more efficient. For example, I store countries following ISO 3166-1 numeric and sex following ISO/IEC 5218. But so far I have not found a similar standard for languages, ISO 639 does not appear to have a matching list of numeric codes.
What would be the right way to do this? Should I just use the String codes?
Thanks!
If you're a fan of the numbers, you can use country calling codes, although they "only" represent the ITU members (193 countries according to Wikipedia). But hey, they have Somalia and Palestine, so that's a good hint about how global this is.
However, storing everything in an encoded format (numbers here) implies a decoding step on the fly when any piece of data is requested (with translation tables stored in RAM instead of DB's ROM). Probably on the server whose CPU is precious, but you might have deported the issue on the client, overworking the precious, time-critical server-client link in the process.
So, back in the 90s, when a 40MB HDD was expensive, that might have been interesting. Today, the cost of storing data vs. the cost of processing data is not on the same side of 1... Not counting the time it takes you to think and implement the transformations. All being said "IMHO", I think this level of efficiency actually kills efficiency. ;)
EDIT: Oops, just realized I misthought (does that verb even exist?) the country/language issue. Countries you have sorted out already, my bad. I know no numbered list of languages. However, the second part of the post might still be relevant...
If you are after raw performance and/or want to achieve really small data sizes, I would suggest you use either the three-letter (higher granularity) or the two-letter (lower granularity) codes from IOC ISO-639-1/2.
To my knowledge, there's no helper or anything for this standard built into any programming language that I know, so you'd need to build your own translator (code<->full name) which, however, should be trivial.
And as others already mentioned, you have to assess the cost involved with this (e.g. not being able to simply look at the data and understand it right away anymore) for yourself. I personally do recommend keeping data sizes small since BSON parsing and string operations are horribly expensive compared to dealing with numbers (or shorter strings for that matter). When dealing with small data sets, this won't make a noticeable difference. If, however, you need to churn through millions of documents or more optimizations like this can become mission critical.
I need to create complete random numbers using a seed in JavaScript. I'm not using the built-in Math.random(), but rather something else that can take a seed and generate a random number based on that.
This solution is supposed to serve a situation in which two users log in at the same time to a website (it happens A LOT, and I'm getting a lot of users with identical IDs because of it). Math.random() isn't working for me, and I can't use timestamps because these also don't provide an accurate number (they're not being sampled every MS). I also don't want to use any ajax call in order to get an IP or something similar.
Is there anything anyone can think of that's either unique, or might be rare enough to use to create a good seed?
**EDIT: ** This is NOT a duplicate of the GUID generation question, because that one is still using Math.random(). I can't use that function anywhere in my code. I sometime have thousands of hits at more-or-less the exact same moment, and that's what screws up the random. It's also the reason why I need to find some attribute I can use as a seed.
Hoping someone can spot the error, because I'm having trouble
Alright, I built my own JSON.stringify for just custom large objects. It may not be exactly to specification for some edge case things, but's only meant for stringify on large objects that I'm building myself.
Well, it works, and works well for most objects, but I have an Object I'm trying to stringify and it's failing and printing this before exiting:
node.js:134
throw e; // process.nextTick error, or 'error' event on first tick
^
undefined
Not very helpful. The object is fine because the regular call to JSON.stringify(object) works fine, and when I iterate over the object with for (var x in obj) if (obj.hasOwnProperty(x)) { myStringify(obj); } that works fine, but if I call it on the top level of the object, it goes to hell... It doesn't really make sense to me, and the only thing I can think of is the level if recursion is somehow breaking something...
The Parser : https://gist.github.com/958776 - The stringify function I'm calling
ObjectIterator.js : https://gist.github.com/958777 - Mostly to provide the asynchronous iteration
Edit So, I iterated over the object one level deep and compared the resulting string to the string of JSON.stringify(sameLevelDeep) and they're equal. Since the output is equal, I'm not sure that it's how I'm parsing something, but possible that it's such a large object or the amount of recursion is so high?
Edit 2 So, I "fixed" the problem, I guess. Instead of every 25th iteration being pushed to the next event loop, I push every fifth. I'm not sure why this would make a difference but it does... I guess the question is now "Why does that make a difference"?
Okay well, beyond it being a very specific question helping a very specific person, I would like to take this to a different place, that might also remove your problem and maybe help others.
Since you are not specifying why you are going through this process, I will have to break it down and guess -- and provide a solution for each guessed idea.
1. (Browser) You are trying to use JavaScript to crunch data, and provide the user with a result
Downloading at least several megabytes of raw data ("some of these objects are 5-10million characters") on a webpage to process and display a result is far from optimal, you should probably be doing this operation on the server side and download the pre-calculated result.
Besides, no matter what you are doing, JavaScript does not support threads.
setTimeout(1, function() { JSON.stringify(data); }); shouldn't be much different from what you are doing.
2. (Browser) You are trying to display the downloaded content
You should attempt downloading smaller chunks instead of the whole 10+ million character content using the built-in JSON.stringify method.
3. (Non-browser) You are trying to use JavaScript for an application that requires threading
You should consider using a different programming language for this application.
In summary
I think you are climbing the wrong mountain, you can achieve the same thing walking around it without breaking sweat. If you want to climb a mountain for kicks, there are mountains out there that need it -- but it's not this one.
Translation: Work on the architecture to obsolete the obstacle instead of trying to solve it, if you want to solve a problem there are problems that need a solving -- but it's not this one.
I have a simple piece of data that I'm storing on a server, as a plain string. It is kind of ridiculous, but it looks like this:
name|date|grade|description|name|date|grade|description|repeat for a long time
this string can be up to 1.4mb in size. The idea is that it's a bunch of student records, just strung together with a simple pipe delimeter. It's a very poor serialization method.
Once this massive string is pushed to the client, it is split along the pipes into student records again, using javascript.
I've been timing how long it takes to create, and split, these strings on the client side. The times are actually quite good, the slowest run I've seen on a few different machines is 0.2 seconds for 10,000 'student records', which has a final string size of ~1.4mb.
I realize this is quite bizarre, just wondering if there are any inherent problems with creating and splitting such large strings using javascript? I don't know how different browsers implement their javascript engines. I've tried this on the 'major' browsers, but don't know how this would perform on earlier versions of each.
Yeah looking for any comments on this, this is more for fun than anything else!
Thanks
String splitting for 1.4mb data is not a problem for decent machines, instead you should worry about the internet connection speed of your users. I've tried to do spell check with 800 kb dictionary (which is half of your data), main issue was loading time.
But looks like your students records data could be put in database, and might not need to load everything at loading time, So, how about do a pagination to show user records or use ajax to request to search certain user names?
If it's a really large string it may pay to continuously slice the string with 'string'.slice(from, to) to only process a smaller subset, appending all of the individual items to the end of the output with list.push() or something similar might work.
String split methods are probably the most efficient way of doing this though, even in IE. Processing individual characters using string.charAt(x) is extremely slow and will often show a security error as it stalls the browser. Using string split methods would certainly be much faster than splitting using regular expressions.
It may also be possible to encode the data using a JSON array, some newer browsers such as IE8/Webkit/FF3.5 have fast JSON parsing built in using JSON.parse(data). But using eval(JSON) may overflow the browser if there's enough data, so is probably a bad idea. It may pay to compare for performance though.
A much better approach in a lot of cases is to use AJAX and only load some of the data at once from the server, which would also save download time.
Besides S. Mark's excellent comments about local vs. x-fer speed and the tip to re-encode using AJAX, I suggest a (longterm) move away from JavaScript in the Browser (assuming that's were it runs) to either a non-browser implementation of JS (or possibly another language).
A browser based JS seems a week link in a data-x-fer chain and nothing I would want to run unmonitored, since the browsers are upgraded from time to time and breaking your JS-x-fer might be an unanticipates side effect!
I have a big data should be shown in a Table.
I use javascript to fill the table instead of priting in HTML.
Here is a sample data I use:
var aUsersData = [[1, "John Smith", "...."],[...],.......];
the problem is that Firefox warns me that "There is a heavy script running, should i continue or stop?"
I don't want my visitors see the warning. how can I make performance better? jQuery? pure script? or another library you suggest?
you can use the method here to show a progress bar and not have the browser lock up on you.
http://www.kryogenix.org/days/2009/07/03/not-blocking-the-ui-in-tight-javascript-loops
I am using almost that method on this page:
http://www.bacontea.com/bb/
to get the browser not to hang and show feedback while loading.
jQuery doesn't usually make things faster, just easier. I use jQuery to populate tables, but they're pretty small (at most 2 columns by 40 rows). How much data are you populating into the table? This could be the limiting factor.
If you post some of your table-populating code we can see if it's possible to improve performance in any way.
My suspicion is that it won't make much difference either way, although sometimes adding a layer of abstraction like jQuery can impact performance. Alternately, the jQuery team may have found an obscure, really fast way of doing something that you would have done in a more obvious, but slower, way if you weren't using it. It all depends.
Two suggestions that apply regardless:
First, since you're already relying on your users having JavaScript enabled, I'd use paging and possibly filtering as well. My suspicion is that it's building the table that takes the time. Users don't like to scroll through really long tables anyway, adding some paging and filtering features to the page to only show them the entries from the array they really want to see may help quite a lot.
Second, when building the table, the way you're do it can have a big impact on performance. It almost seems a bit counter-intuitive, but with most browsers, building up a big string and then setting the innerHTML property of a container is usually faster than using the DOM createElement function over and over to create each row and cell. The fastest overall tends to be to push strings onto an array (rather than repeated concatenation) and then join the array:
var markup, rowString;
markup = [];
markup.push("<table><tbody>");
for (index = 0, length = array.length; index < length; ++index) {
rowString = /* ...code here to build a row as a string... */;
markup.push(rowString);
}
markup.push("</tbody></table>");
document.getElementById('containerId').innerHTML = markup.join("");
(That's raw JavaScript/DOM; if you're already using jQuery and prefer, that last line can be rewritten $('#containerId').html(markup.join(""));)
This is faster than using createElement all over the place because it allows the browser to process the HTML by directly manipulating its internal structures, rather than responding to the DOM API methods layered on top of them. And the join thing helps because the strings are not constantly being reallocated, and the JavaScript engine can optimize the join operation at the end.
Naturally, if you can use a pre-built, pre-tested, and pre-optimised grid, you may want to use that instead -- as it can provide the paging, filters, and fast-building required if it's a good one.
You can try a JS templating engine like PURE (I'm the main contributor)
It is very fast on all browsers, and keeps the HTML totally clean and separated from the JS logic.
If you prefer the <%...%> type of syntax, there are plenty of other JS template engines available.