Related
I am making a trivia system, and the variable triviaA is changed to the updated Answer every time there is a new question. I was wondering how I could use Regex to make it so that if triviaA = 'eagle' then if someone submitted the answer as eagle but with a small spelling mistake, the if statement would still return both the triviaA answer and the players answer return true.
I'm new to Regex, please excuse my mistakes.
You can dynamically build a regular expression that matches all possible "errors" (wrong/missing/extra letter) and apply it to the source:
function fuzzyContains(word, str) {
let r = [];
for (let i = 0; i < word.length; i++) {
// wrong letter
r.push(word.slice(0, i) + '[a-z]' + word.slice(i + 1));
// missing letter
r.push(word.slice(0, i) + '' + word.slice(i + 1));
// extra letter
r.push(word.slice(0, i) + '[a-z]' + word.slice(i));
}
return new RegExp(r.join('|')).test(str)
}
console.log(fuzzyContains('eagle', 'fly, ewgle, fly'))
console.log(fuzzyContains('eagle', 'fly, eagl, fly'))
console.log(fuzzyContains('eagle', 'fly, eaggle, fly'))
console.log(fuzzyContains('eagle', 'fly, eagly, fly'))
See https://norvig.com/spell-correct.html for other interesting ideas.
This would actually be quite difficult to do with RegEx. However, there's a concept in computing called "edit distance", which is a measure of how "similar" two strings are, and there are known algorithms to calculate that. Which means there are Node packages to calculate it :)
For instance, there's the aptly named Node package edit-distance: https://www.npmjs.com/package/edit-distance (Note: edit distance is sometimes called Levenshtein edit distance, named after the man who first studied it.)
I'll give you an example using that package; other packages may work somewhat differently to calculate the same thing.
There are three types of changes between two strings: an insertion, where a character is added in one that's not in the other; a deletion, where a character is removed in one that's there in the other; and an update/substitution, where a letter is changed between the strings. Using the edit-distance package, you define a function that assigns a cost to each of these types of changes. Then you call the package's levenshtein method, passing it the two strings and the three functions, and it returns an object with a distance property that is the sum score.
Assuming your cost functions return non-negative values, a score of 0 means the two strings are identical, and higher numbers mean they're more different. So you can use this to compare the entered value with the correct string and, if the result is lower than a certain threshold, accept it as "correct excluding typos".
Say you want to have a set of 1- to 2-digit hexadecimal numbers, so 256 numbers. Just using a small set to get at the problem, but it would work with any sized string.
So you have a potential N or 256 numbers in this case. You are going to "generate" a new ID for every new data record that comes your way. So it starts of and randomly gives you af, then 1d, then 8a, etc.
The straightforward naïve way to do this is to just simply generate all the numbers in order, then shuffle them, and just pop from the set. This works fine when you only have 256 numbers. But if you have millions or billions of numbers it is impractical as you might have a lot of waisted generated IDs not being used for long periods of time. I would like to avoid this.
So my question is what is the or a fastest way to create a unique key string like this, without generating all of them in advance, and without going in order just incrementing by 1 or whatnot. That is, the key should be seemingly random.
One way I can imagine is using a trie to store the already used/generated values. Then when you are to get a new value, you generate a random one, then check the trie to see if it's already used. I have no idea how to tell how efficient this is though, but it seems like it would be very bad performing once you start running out of IDs and are down to the last few ones in the set. You would generate lots of already generated IDs, and traverse the trie for each, so it would be slow.
I am wondering if there is a more efficient way of doing this, without generating them all in advance. Also, the data records won't be used in figuring out the ID, as the records might be extremely large and complex.
Maybe there is a way to sort of randomly traverse (and generate) a trie at once, and in that way generate the ID since you end up at a unique random place in the trie. Something along those lines perhaps, I don't know.
Also, I am not sophisticated with hashing so I don't know if there would be any good methods with that.
I assume that you could generate sequential IDs; that is, that you have a reliable way of knowing exactly how many IDs have been generated to date. Then it is sufficient to encrypt this count with any reasonably fast encryption algorithm.
The encryption would be done on the count as a binary number, and the encrypted result with most algorithms would be the same size, also binary. If desired, you could base-64 or hex encode the result to make it easier to use as a character string.
Since encryption must be a bijection (that is, a one-to-one mapping) in order for decryption to be possible, this is guaranteed to produce a different result each time until the total ID count overflows. If it is a reasonable encryption function, then the result will appear random (otherwise the cipher would be vulnerable).
I am not sure how performant it will be but my idea is use a object or Map and Math.random()
let obj = {}
function generateRandomId(){
let id = Math.abs( 0.5 - Math.random()) * 1000
if(obj[id]){
generateRandomId()
} else {
obj[id] = true
}
return id
}
console.log(generateRandomId())
console.log(generateRandomId())
console.log(generateRandomId())
console.log(generateRandomId())
But if you are ok with using a modules i find this one is most useful
uuid this generates RFC4122 UUIDS.
I think that there should be some tradeoff between speed, flexibility and efficiency.
On one had pseudo random generators will give you that even distribution of keys and will be reasonably fast to generate. However checking for an existing id would be slow. You can use bloom filters (saving memory) or tries but then as you said at some point you should increase the space.
Another option is to use Gray code which will produce every key (but not at random order). You need to keep track of the last issued code.
I think a mixing function is what you want. It will move bits around in your input to produce an output of the same length. It's reversible so each input corresponds to a unique output.
Since you want the input data to not take part in the id generation, you'll need a surrogate id. You can assign an incrementing id to each record and use the mix function to scramble the id.
You will get something like:
Record A => id == 1 => mixed id == 0x7ed55d16
Record B => id == 2 => mixed id == 0xc761c23c
etc.
See here for some inspiration:
https://crypto.stackexchange.com/questions/12145/need-32-bit-mixing-function-that-has-perfect-avalanche-between-octets
https://gist.github.com/badboy/6267743
I am considering something like this:
var trie = buildTrie()
var id1 = genId(trie)
var id2 = genId(trie)
console.log(id1,id2)
function buildTrie() {
var trie = buildNode(0)
return trie
function buildNode(level) {
if (level == 7) { // 8 bits
var node = {
available: true,
leaf: true
}
return node
} else {
var a = buildNode(level + 1)
var b = buildNode(level + 1)
var node = {
availableLeft: true,
availableRight: true,
left: a,
right: b
}
a.parent = node
b.parent = node
return node
}
}
}
function genId(node) {
var bytes = []
step(node, bytes)
var id = parseInt(bytes.join(''), 2).toString(16)
return id
function step(node, bytes) {
if (node.leaf) {
node.available = false
var c = node
var p = c.parent
while (p) {
if (p.left == c) {
p.availableLeft = false
} else if (p.right == c) {
p.availableRight = false
}
if (!p.availableLeft && !p.availableRight) {
c = p
p = p.parent
} else {
p = false
}
}
}
var randomDirection = Math.random() >= 0.5
if (randomDirection) {
if (node.availableLeft) {
bytes.push(0)
step(node.left, bytes)
} else if (node.availableRight) {
bytes.push(1)
step(node.right, bytes)
}
} else {
if (node.availableRight) {
bytes.push(1)
step(node.right, bytes)
} else if (node.availableLeft) {
bytes.push(0)
step(node.left, bytes)
}
}
}
}
Maybe there is a better way.
I am trying to find out the best way to implement nextId() in a Javascript app that issues unique IDs for its objects. If it matters, it's a program that I am doing as a learning experience that's pure JS, HTML, and CSS (no libraries, frameworks, DBMS, etc.). I saw a similar question on here on SO (although I wasn't able to find it again for the link) with answers that included not only storing a list of possible ids paired with a boolean value to determine if the id is used, but also storing deleted ids in a recycling list to use for future objects that need it. I think the latter option sounds better, but I'm sure there are even more ways to do it. Does anyone know if there is a pattern, algorithm, or otherwise best practice for this task?
EDIT:
I would like to allow users to share data at some point soon in the application's life, so IDs that already exist would likely become an issue at some point. I would like the IDs to be permanent as I will be persisting data with LocalStorage. A simple integer will work which I will prefix with a letter or two to identify the type of object. It would also be nice to fill in the holes, so the integer doesn't get too high when users use it long-term (wishful thinking).
Also, all objects are constructed from strings at the beginning of the program (I know it's insane).
If you just need an id that is unique per the lifetime of a page, you can use a simple monotomically increasing counter in the page:
var getUniqueID = (function() {
var cntr = 0;
return function() {
return cntr++;
};
})();
var idA = getUniqueID();
var idB = getUniqueID();
To make sure your ids are unique among all users is a taller task. Without involving a central server that coins unique ids for you, the general concept here is to create an id that is a combination of three things:
A token that is unique to the user (like a userID)
A token that is guaranteed to be unique for the session (like what we have above)
A random value.
Done right, there can never be a collision between two different users (because the userID is in the id) and the counter makes it so no user ever generates the same id twice in the same session and the random value makes the odds of a user generating the same id themselves in the same session extremely small.
var getGUID = (function() {
var cntr = 0;
return function(userID) {
var rand = Math.random().toString().replace(".", "");
return userID + "_" + rand + "_" + cntr++;
};
})();
var idA = getGUID(myUserID);
var idB = getGUID(myUserID);
Note: this is the simpler approach on GUID generation that assumes you already have a userID. There is a whole lot of research and literature on various strategies for generating a GUID which you can certainly read a lot more about if you want something beyond this. Some references on the topic:
http://en.wikipedia.org/wiki/Globally_unique_identifier
http://betterexplained.com/articles/the-quick-guide-to-guids/
http://www.uddi.org/pubs/draft-leach-uuids-guids-01.txt
Depending on the use case, I like to create a complex unique id:
function newGUID(){
var result = '';
var hexcodes = "0123456789abcdef".split("");
for (var index = 0; index < 32; index++) {
var value = Math.floor(Math.random() * 16);
switch (index) {
case 8:
result += '-';
break;
case 12:
value = 4;
break;
}
result += hexcodes[value];
}
return result;
}
You could use UUIDs as IDs. There's one answer here in SO where you can generate UUIDs in JS. UUIDs are usually enough to be used as IDs. Just to be sure that the id isn't a dupe, you can have an object whose keys are the used IDs. As IDs are generated, you can keep track of them by adding them in the object. You can then look them up by doing obj.hasOwnProperty(id).
You can do it like
var idStorage = {};
var id;
// generate ID that's unique and hasn't been used
do{
id = guid();
} while (idStorage.hasOwnProperty(id));
idStorage[id] = true;
// ID is usable
Also, IDs are supposed to be unique. They should never be reused at all.
AngularJS has a very simply approach to generating unique IDs: just increment a global counter. From src/Angular.js:
var uid = 0;
// ...
/**
* A consistent way of creating unique IDs in angular.
*
* Using simple numbers allows us to generate 28.6 million unique ids per second for 10 years before
* we hit number precision issues in JavaScript.
*
* Math.pow(2,53) / 60 / 60 / 24 / 365 / 10 = 28.6M
*
* #returns {number} an unique alpha-numeric string
*/
function nextUid() {
return ++uid;
}
Of course, you should choose a solution depending on what time frame the generated IDs should be unique. The above solution will generate IDs which are unique for one session of one web page. For a simple single-page application (which is Angular's use case), this will do just fine. If you need them to be unique across multiple page loads, or unique for the same user, you'll need to persist the uid for a bit longer (in a cookie or in a database). If they need to be unique for a longer time, you might also need to look into longer IDs with more than 2^53 possible values.
I would use a UUID v4 since these IDs would always be unique in any circumstance (no additional logic to check if these ids are in use or to recicle old ones), just issue a new id whenever you need one.
Very simple implementation in JS as follows:
function generateUUID(){
return 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, function(c) {
var r = Math.random()*16|0, v = c == 'x' ? r : (r&0x3|0x8);
return v.toString(16);
});
}
$(function() {
$('button').bind('click', function() {
$('input').val(generateUUID());
});
});
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<input type="text" style="width: 350px;"/><button>GENERATE NEW ID</button>
Well, it's a bit unclear what you expect from these ids, however if you only want a unique id per type of entity in your system and your data must only live in memory, then you could use the following approach:
Note: I saw from the comments you wanted a prefix which represents the entity.
function identitySequence(base) {
var id = base || 0;
return function () {
return ++id;
};
}
//Lets say you had some sort of repository for every entity
function InMemoryUserRepository(nextId) {
this._nextId = nextId;
}
InMemoryUserRepository.prototype = {
constructor: InMemoryUserRepository,
get nextId() { return 'user-' + this._nextId(); }
//other operations, like save...
};
var userRepository = new InMemoryUserRepository(identitySequence());
userRepository.nextId; //user-1
Now, lets say you wanted your id sequence to be persistent in the localStorage, here's what you could do (extending the above code):
var userRepository = new InMemoryUserRepository(localStorageIdentitySequence('userIdSeq'));
userRepository.nextId; //user-1
//Reload page
userRepository.nextId; //user-2
function localStorageIdentitySequence(storageKey) {
var next = identitySequence(+localStorage.getItem(storageKey));
return function () {
return +(localStorage[storageKey] = next());
};
}
This works fine for a single machine, however if you want a unique id generator that will generate unique IDs across machines, then this approach will not work. You would have to generate the ID from a server that can be accessed by all clients, or you can generate a GUID on the client instead. There would be no way to know if the GUID was already generated by another client, but that would be very unlikely.
Sorry for the title, couldn't come up with a better one.
Let's say we have that JavaScript code string:
var n = Math.floor(Math.random() * 10),
className;
if (n === 1) {
className = "a";
} else if (n === 2) {
className = "b";
} else {
className = "c";
}
document.querySelector("." + className);
The idea is that I want to get all the possible strings sent to that particular function (document.querySelector). So I want to get ['.a', '.b', '.c']. There could also be multiple variables involved, modified several times in the code, so that the list would be much longer.
Now how do I do that in Python? I've looked at PyV8 but there is no documentation, so that's not an option; same for python-spidermonkey which is way outdated.
This is not an easy problem. You're looking for static code analysis to generate all possible paths through your function. Consider the following code and ask yourself how to determine whether an alert will run:
var n = Math.floor(Math.random() * 10),
if (Math.sqrt(n) > n) {
alert('a');
}
The computer doesn't "know" that Math.sqrt(n) will always be smaller than n. Without running the code, how do I determine that the alert won't show up?
In simple cases a library might be able to do it but when your function has numerous possible paths and utilizes many functions you'll need some hefty analysis to get the correct answer.
Well, you could take the Monte Carlo approach: log all arguments passed to document.querySelector and run the code against a variety of inputs.
Please forgive me if this is answered on SO somewhere already. I've searched, and it seems as though this is a fairly specific case.
Here's an example of the JSON (NOTE: this is very stripped down - this is dynamically loaded, and currently there are 126 records):
var layout = {
"2":[{"id":"40","attribute_id":"2","option_id":null,"design_attribute_id":"4","design_option_id":"131","width":"10","height":"10",
"repeat":"0","top":"0","left":"0","bottom":"0","right":"0","use_right":"0","use_bottom":"0","apply_to_options":"0"},
{"id":"41","attribute_id":"2","option_id":"115","design_attribute_id":"4","design_option_id":"131","width":"2","height":"1",
"repeat":"0","top":"0","left":"0","bottom":"4","right":"2","use_right":"0","use_bottom":"0","apply_to_options":"0"},
{"id":"44","attribute_id":"2","option_id":"118","design_attribute_id":"4","design_option_id":"131","width":"10","height":"10",
"repeat":"0","top":"0","left":"0","bottom":"0","right":"0","use_right":"0","use_bottom":"0","apply_to_options":"0"}],
"5":[{"id":"326","attribute_id":"5","option_id":null,"design_attribute_id":"4","design_option_id":"154","width":"5","height":"5",
"repeat":"0","top":"0","left":"0","bottom":"0","right":"0","use_right":"0","use_bottom":"0","apply_to_options":"0"}]
};
I need to match the right combination of values. Here's the function I currently use:
function drawOption(attid, optid) {
var attlayout = layout[attid];
$.each(attlayout, function(k, v) {
// d_opt_id and d_opt_id are global scoped variable set elsewhere
if (v.design_attribute_id == d_att_id
&& v.design_option_id == d_opt_id
&& v.attribute_id == attid
&& ((v.apply_to_options == 1 || (v.option_id === optid)))) {
// Do stuff here
}
});
}
The issue is that I might iterate through 10-15 layouts (unique attid's), and any given layout (attid) might have as many as 50 possibilities, which means that this loop is being run A LOT.
Given the multiple criteria that have to be matched, would an AJAX call work better? (This JSON is dynamically created via PHP, so I could craft a PHP function that could possibly do this more efficently),
or am I completely missing something about how to find items in a JSON object?
As always, any suggestions for improving the code are welcome!
EDIT:
I apologize for not making this clear, but the purpose of this question is to find a way to improve the performance. The page has a lot of javascript, and this is a location where I know that performance is lower than it could be.
First and foremost you should measure and only act if there is a real performance concern. You need exact numbers like 200ms or 80% time is spent there. "It runs a lot" doesn't mean anything. The browser can loop very fast.
You can improve constant factors as others mentioned, like using a native for loop instead of jQuery.each. Chaching global variables won't help you too much in this case.
If you really want to improve efficency you should find a better algorithm than O(n). Assuming that you only use this data for finding elements matching a certain criteria you can use JS objects as hashes to achive a O(1) performance.
Just an example for you specific case:
var layout = {
"2": { "2,,4,131,10,0": ["40", "93"], "2,115,4,131,0": ["41"] },
"4": { ... },
...
};
It's fairly easy to generate this output in php, and then you just use a lookup to find ids matching your particular criteria.
IMHO, a simple hashmap index will probably work best. This does require you to loop over the data ahead of time, but the index can be easily appended to and cached.
Once the index is generated, this should be O(1) for lookups, and will handle multiple entries per key.
var layout = {
"2":[[...], ...],
"5":[[...], ...]
};
var index = {};
function makeKey(data) {
return data.join('_');
}
for(var l in layout) {
var cur = layout[l];
for(var i in cur) {
var item = cur[i];
var key = makeKey([item.p1, item.p2, ...]);
index[key] = index[key] || [];
index[key].push(item);
}
}
function find(attid, optid) {
var key = makeKey([attid, optid, 1, d_att_id, ...]);
return index[key]; //this is an array of results
}
My first suggestion would be to stop using $.each if you want to squeeze out every bit of performance you can. jQuery.each does a bit more than a traditional loop. Take a look at this jsFiddle with your browser's debugger running (i.e. Safari's/Chrome's web developer tools) and step through the fiddle until execution fully returns from jQuery.
For reference, the fiddle's code is:
var myArr = [{"foo":"bar"},{"answer":42}];
debugger;
$.each(myArr, function(k, v) {
console.log(k + ': ');
console.dir(v);
});
Now run through the second version:
var myArr = [{"foo":"bar"},{"answer":42}],
i, j;
debugger;
for (i = 0, j = myArr.length; i < j; i += 1) {
console.log('myArr[' + i + ']: ');
console.dir(myArr[i]);
}
Notice that there are far fewer operations being executed in the second version. So that's a tiny bit of performance gain.
Second, eliminate as many lookups outside of the local scope as you can. Even if you cache a reference to your global variables (boo!) then you can save a lot of time given that this loop will be executed possibly hundreds of times. So instead of:
function foo(a, b) {
if (a === globalA && b === globalB) {
// Do stuff
}
}
You'd do:
function foo(a, b) {
var gA = globalA,
gB = globalB;
if (a === gA && b === gB) {
// Do stuff
}
}
As for pairing down the conditional based on the object members, I'm not seeing much else that could be improved. The object properties you are checking are top level, and you're looking at local instances of each object (so the scope chain lookups are short).
Without knowing more about how this is actually supposed to work, those are the best recommendations I can make. However, I can make the guess that your idea of starting with simpler JSON data would be a big improvement. If you know what the layout, and its constraints, is, then requesting the specific details from the server would mean you don't have to check so many conditions. You could simply ask the server for the details that you actually need to implement and loop through those (to update the DOM or whatever).
I see that the you are searching by
5 fields: v.design_attribute_id,v.design_option_id,v.attribute_id,v.apply_to_options,v.option_id.
What you could do is add an extra field to the objects called "key" that is a composite of the values in those fields.
Here's an example
{
"key": "4_131_2_0_0" //i picked 0 to represent null, but you can pick any number
"id": "40",
"attribute_id": "2",
"option_id": null,
"design_attribute_id": "4",
"design_option_id": "131",
"width": "10",
"height": "10",
"repeat": "0",
"top": "0",
"left": "0",
"bottom": "0",
"right": "0",
"use_right": "0",
"use_bottom": "0",
"apply_to_options": "0"
}
Note though that you must normalize the length of each value.
Meaning that if one objects optionId is 1 and another object optionID is 566 you must represent
the first optionId as 001 in the key string.
With this field you can then sort the array on the server side before returning it to the client.
Then you can use a binary search to find the values on the client.
Using the binary search implementation located here
http://www.nczonline.net/blog/2009/09/01/computer-science-in-javascript-binary-search/
Your search function would look something like
function drawOption(attid, optid) {
var attlayout = layout[attid];
var needle = d_att_id + "_" + d_opt_id + "_" + attid + "_" + optid; //remember to normalize length if you have to
var idx = binarySearch(attlayout,needle);
var item;
if(idx !== -1){
item = attlayout[idx];
//do something
}
}
Another method you can try using this composite key idea is to have the server return
the layout objects in one big object mapped by
attid,v.design_attribute_id,v.design_option_id,v.attribute_id,v.apply_to_options,v.option_id
Then you can look up in O(1) time.
It would look something like
function drawOption(attid, optid) {
var needle = attid + "_" + d_att_id + "_" + d_opt_id + "_" + attid + "_" + optid; //remember to normalize length if you have to
var item = layout[needle];
if(typeof item !== "undefined"){
//do something
}
}
When trying to improve your code, it is always better to check which functions are taking time using firebug profiling. You can either profile by clicking on profile button in firebug's console panel and then run your code or using firebug's profiling commands in your code
From the code that you have given, only a few improvement points can be given.
$.each is slow compared to native looping solutions. For the best
looping solutions, check out this JsPref test
It would be better to change the JSON to use arrays instead of object literals. It is said to be more faster to retrieve values.
I have experience to such issue before, my js array of objects consist of 8 thousands record and more.
My experience is not about the performance, but the readability, maintainability, scalable of the codes.
hence I developed an JS Object Query Library 2 years ago: JSOQL
http://code.google.com/p/jsoql/
It works like SQL to allow you query your js array of objects with syntax similar to SQL.
The example usage is something like this, I cant really remember, but you can download the example usage in the download tab.
new JSQOL().Select([field1, field2, field3 ...]).From({ ... }) .Where(fn) .Offset(int) .Limit(int) .Get();
Note:
{...} is your array of objects, or an object it self.
Hope it helps, you can send me message if you need more information.
It's not going to work everywhere but your problem sounds like something that can be done with webworkers
Another thing I would look at if you dont have webworkers is trying not to block the ui to long. If you can chunk it into bits of about 40ms and then setTimeout the next chunck for just a few ms later the user will have a more pleasant experience. This needs a little fiddling but users will start to notice stuff when something takes longer than somewhere between 50 and 100ms
Have you considered using the jQuery grep function?
jQuery grep
And jquery grep on json object array for an example.
Here is one technique that will probably yield better performance at the expense of using a bit more memory.
I'll leave my code examples simple just to illustrate the concept.
First, you'll want to pre-process your JSON data into some additional arrays that act as indexes. Here is an example of what the final arrays might look like after pre-processing:
var layouts_by_attribute = {
// attribute_id => array(layouts)
2: [40, 41, 44],
5: [326]
};
var layouts_by_design_attribute_id = {
// design_attribute_id => array(layouts)
4: [40, 41, 44, 326]
};
Finding a layout by attribute is now very quick:
function findByAttribute(attribute_id) {
return layouts = layouts_by_attribute[attribute_id];
}
function findByDesignAttribute(design_attribute_id) {
return layouts = layouts_by_design_attribute[design_attribute_id];
}