Related
I am wondering how to generate a GUID given an input string, such that the same input string results in the same GUID (sort of like an MD5 hash). The problem with MD5 hashes is they just guarantee low collision rate, rather than uniqueness. Instead I would like something like this:
guid('v1.0.0') == 1231231231123123123112312312311231231231
guid('v1.0.1') == 6154716581615471658161547165816154716581
guid('v1.0.2') == 1883939319188393931918839393191883939319
How would you go about implementing this sort of thing (ideally in JavaScript)? Is it even possible to do? I am not sure where to start. Things like the uuid module don't take a seed string, and they don't let you use a custom format/alphabet.
I am not looking for the canonical UUID format, but rather a GUID, ideally one made up of just integers.
What you would need is define a one-to-one mapping of text strings (such as "v1.0.0") onto 40 digit long strings (such as "123123..."). This is also known as a bijection, although in your case an injection (a simple one-to-one mapping from inputs to outputs, not necessarily onto) may be enough. As you note, hash functions don't necessarily ensure this mapping, but there are other possibilities, such as full-period linear congruential generators (if they take a seed that you can map one-to-one onto input string values), or other reversible functions.
However, if the set of possible input strings is larger than the set of possible output strings, then you can't map all input strings one-to-one with all output strings (without creating duplicates), due to the pigeonhole principle.
For example, you can't generally map all 120-character strings one-to-one with all 40-digit strings unless you restrict the format of the 120-character strings in some way. However, your problem of creating 40-digit output strings can be solved if you can accept limiting input strings to no more than 1040 values (about 132 bits), or if you can otherwise exploit redundancy in the input strings so that they are guaranteed to compress losslessly to 40 decimal digits (about 132 bits) or less, which may or may not be possible. See also this question.
The algorithm involves two steps:
First, transform the string to a BigInt by building up the string's charCodeAt() values similarly to the stringToInt method given in another answer. Throw an error if any charCodeAt() is 0x80 or greater, or if the resulting BigInt is equal to or greater than BigInt(alphabet_length)**BigInt(output_length).
Then, transform the integer to another string by taking the mod of the BigInt and the output alphabet's size and replacing each remainder with the corresponding character in the output alphabet, until the BigInt reaches 0.
One approach would be to use the method from that answer:
/*
* uuid-timestamp (emitter)
* UUID v4 based on timestamp
*
* Created by tarkh
* tarkh.com (C) 2020
* https://stackoverflow.com/a/63344366/1261825
*/
const uuidEmit = () => {
// Get now time
const n = Date.now();
// Generate random
const r = Math.random(); // <- swap this
// Stringify now time and generate additional random number
const s = String(n) + String(~~(r*9e4)+1e4);
// Form UUID and return it
return `${s.slice(0,8)}-${s.slice(8,12)}-4${s.slice(12,15)}-${[8,9,'a','b'][~~(r*3)]}${s.slice(15,18)}-${s.slice(s.length-12)}`;
};
// Generate 5 UUIDs
console.log(`${uuidEmit()}
${uuidEmit()}
${uuidEmit()}
${uuidEmit()}
${uuidEmit()}`);
And simply swap out the Math.random() call to a different random function which can take your seed value. (There are numerous algorithms out there for creating a seedable random method, so I won't try prescribing a particular one).
Most random seeds expect numeric, so you could convert a seed string to an integer by just adding up the character values (multiplying each by 10^position so you'll always get a unique number):
const stringToInt = str =>
Array.prototype.slice.call(str).reduce((result, char, index) => result += char.charCodeAt(0) * (10**(str.length - index)), 0);
console.log(stringToInt("v1.0.0"));
console.log(stringToInt("v1.0.1"));
console.log(stringToInt("v1.0.2"));
If you want to generate the same extract string every time, you can take a similar approach to tarkh's uuidEmit() method but get rid of the bits that change:
const strToInt = str =>
Array.prototype.slice.call(str).reduce((result, char, index) => result += char.charCodeAt(0) * (10**(str.length - index)), 0);
const strToId = (str, len = 40) => {
// Generate random
const r = strToInt(str);
// Multiply the number by some things to get it to the right number of digits
const rLen = `${r}`.length; // length of r as a string
// If you want to avoid any chance of collision, you can't provide too long of a string
// If a small chance of collision is okay, you can instead just truncate the string to
// your desired length
if (rLen > len) throw new Error('String too long');
// our string length is n * (r+m) + e = len, so we'll do some math to get n and m
const mMax = 9; // maximum for the exponent, too much longer and it might be represented as an exponent. If you discover "e" showing up in your string, lower this value
let m = Math.floor(Math.min(mMax, len / rLen)); // exponent
let n = Math.floor(len / (m + rLen)); // number of times we repeat r and m
let e = len - (n * (rLen + m)); // extra to pad us to the right length
return (new Array(n)).fill(0).map((_, i) => String(r * (i * 10**m))).join('')
+ String(10**e);
};
console.log(strToId("v1.0.0"));
console.log(strToId("v1.0.1"));
console.log(strToId("v1.0.2"));
console.log(strToId("v1.0.0") === strToId("v1.0.0")); // check they are the same
console.log(strToId("v1.0.0") === strToId("v1.0.1")); // check they are different
Note, this will only work with smaller strings, (probably about 10 characters top) but it should be able to avoid all collisions. You could tweak it to handle larger strings (remove the multiplying bit from stringToInt) but then you risk collisions.
I suggest using MD5...
Following the classic birthday problem, all things being equal, the odds of 2 people sharing a birthday out of a group of 23 people is ( see https://en.wikipedia.org/wiki/Birthday_problem )...
For estimating MD5 collisions, I'm going to simplify the birthday problem formula, erring in the favor of predicting a higher chance of a collision...
Note though that whereas in the birthday problem, a collision is a positive result, in the MD5 problem, a collision is a negative result, and therefore providing higher than expected collision odds provides a conservative estimate of the chance of a MD5 collision. Plus this higher predicted chance can in some way be considered a fudge factor for any uneven distribution in the MD5 output, although I do not believe there is anyway to quantify this without a God computer...
An MD5 hash is 16 bytes long, resulting in a range of 256^16 possible values. Assuming that the MD5 algorithm is generally uniform in its results, lets suppose we create one quadrillion (ie, a million billion or 10^15) unique strings to run through the hash algorithm. Then using the modified formula (to ease the collision calculations and to add a conservative fudge factor), the odds of a collision are...
So, after 10^15 or one quadrillion unique input strings, the estimated odds of a hash collision are on par with the odds of winning the Powerball or the Mega Millions Jackpot (which are on order of 1 in ~300,000,000 per https://www.engineeringbigdata.com/odds-winning-powerball-grand-prize-r/ ).
Note too that 256^16 is 340282366920938463463374607431768211456, which is 39 digits, falling within the desired range of 40 digits.
So, suggest using the MD5 hash ( converting to BigInt ), and if you do run into a collision, I will be more than glad to spot you a lottery ticket, just to have a chance to tap into your luck and split the proceeds...
( Note: I used https://keisan.casio.com/calculator for the calculations. )
While UUID v4 is just used for random ID generation, UUID v5 is more like a hash for a given input string and namespace. It's perfect for what you describe.
As you already mentioned, You can use this npm package:
npm install uuid
And it's pretty easy to use.
import {v5 as uuidv5} from 'uuid';
// use a UUIDV4 as a unique namespace for your application.
// you can generate one here: https://www.uuidgenerator.net/version4
const UUIDV5_NAMESPACE = '...';
// Finally, provide the input and namespace to get your unique id.
const uniqueId = uuidv5(input, namespace);
I am streaming data from a CSV file and I am pushing certain values to an array. Some of these values are very large, and may be exported as exponential values (e.g. 5.02041E+12) and some may be normal values.
I'd like to have an if statement that checks to see if these values are exponential or not, and if they are I will pass them to a function that converts them into 'normal' numbers (e.g. 5020410000000). Is there a quick way to do this?
(These values are passed to an API call which is why they need to be converted to 'normal' values)
Example of what this may look like:
valueOne = 5.02041E+12;
valueTwo = 1234;
if (valueOne.isExponential) {
**pass to converting function**
}
//Output = 5020410000000
if (valueTwo.isExponential) {
**pass to converting function**
}
//Output = 1234 (unchanged)
I'd expect all values in the array to therefore be 'normal' values (i.e. NOT is exponential form)
Numbers are numbers, the values 5.02041E+12 and 5020410000000 do not differ internally:
// values in code:
var withe=5.02041E+12;
var withoute=5020410000000;
console.log(withe,"?=",withoute,withe===withoute);
// values parsed from strings:
var stringwithe="5.02041E+12";
var stringwithoute="5020410000000";
var a=parseFloat(stringwithe);
var b=parseFloat(stringwithoute);
console.log(a,"?=",b,a===b);
And you can also see that when you simply display a number, it will not use the scientific notation by default, actually you would have to ask for it via using toExponential()
One thing you can worry about is the internal precision of Number. It has a method isSafeInteger() and various fields, like MAX_SAFE_INTEGER. Surpassing that value can lead to unexpected results:
var a=Number.MAX_SAFE_INTEGER;
console.log("This is safe:",a,Number.isSafeInteger(a));
a++;
for(var i=0;i<5;i++)
console.log("These are not:",a++,Number.isSafeInteger(a));
So the loop can not increment a any further, because there is no such number as 9007199254740993 here. The next number which exists after 9007199254740992 is 9007199254740994. But these numbers are more than 1000x greater than the 5020410000000 in the question.
You can just use toPrecision on every number and ensure it converts
const valueOne = 5.02041E+12;
const valueTwo = 1234;
const precisionValueOne = valueOne.toPrecision(); // "5020410000000"
const precisionValue2 = valueTwo.toPrecision(); // "1234"
You can then, optionally convert it back to numbers:
sanitizedValueOne = Number(precisionValueOne); // 5020410000000
sanitizedValueTwo = Number(precisionValueTwo); // 1234
Do a RegExp match for E+ and probably for E-
The 'number' you are starting with must be text, but you should do a bit of sanity checking too.
Might be a good idea to check whether it is larger than MaxInt before you try any Math-based conversions.
This is an extension of this SO question
I made a function to see if i can correctly format any number. The answers below work on tools like https://regex101.com and https://regexr.com/, but not within my function(tried in node and browser):
const
const format = (num, regex) => String(num).replace(regex, '$1')
Basically given any whole number, it should not exceed 15 significant digits. Given any decimal, it should not exceed 2 decimal points.
so...
Now
format(0.12345678901234567890, /^\d{1,13}(\.\d{1,2}|\d{0,2})$/)
returns 0.123456789012345678 instead of 0.123456789012345
but
format(0.123456789012345,/^-?(\d*\.?\d{0,2}).*/)
returns number formatted to 2 deimal points as expected.
Let me try to explain what's going on.
For the given input 0.12345678901234567890 and the regex /^\d{1,13}(\.\d{1,2}|\d{0,2})$/, let's go step by step and see what's happening.
^\d{1,13} Does indeed match the start of the string 0
(\. Now you've opened a new group, and it does match .
\d{1,2} It does find the digits 1 and 2
|\d{0,2} So this part is skipped
) So this is the end of your capture group.
$ This indicates the end of the string, but it won't match, because you've still got 345678901234567890 remaining.
Javascript returns the whole string because the match failed in the end.
Let's try removing $ at the end, to become /^\d{1,13}(\.\d{1,2}|\d{0,2})/
You'd get back ".12345678901234567890". This generates a couple of questions.
Why did the preceding 0 get removed?
Because it was not part of your matching group, enclosed with ().
Why did we not get only two decimal places, i.e. .12?
Remember that you're doing a replace. Which means that by default, the original string will be kept in place, only the parts that match will get replaced. Since 345678901234567890 was not part of the match, it was left intact. The only part that matched was 0.12.
Answer to title question: your function doesn't replace, because there's nothing to replace - the regex doesn't match anything in the string. csb's answer explains that in all details.
But that's perhaps not the answer you really need.
Now, it seems like you have an XY problem. You ask why your call to .replace() doesn't work, but .replace() is definitely not a function you should use. Role of .replace() is replacing parts of string, while you actually want to create a different string. Moreover, in the comments you suggest that your formatting is not only for presenting data to user, but you also intend to use it in some further computation. You also mention cryptocurriencies.
Let's cope with these problems one-by-one.
What to do instead of replace?
Well, just produce the string you need instead of replacing something in the string you don't like. There are some edge cases. Instead of writing all-in-one regex, just handle them one-by-one.
The following code is definitely not best possible, but it's main aim is to be simple and show exactly what is going on.
function format(n) {
const max_significant_digits = 15;
const max_precision = 2;
let digits_before_decimal_point;
if (n < 0) {
// Don't count minus sign.
digits_before_decimal_point = n.toFixed(0).length - 1;
} else {
digits_before_decimal_point = n.toFixed(0).length;
}
if (digits_before_decimal_point > max_significant_digits) {
throw new Error('No good representation for this number');
}
const available_significant_digits_for_precision =
Math.max(0, max_significant_digits - digits_before_decimal_point);
const effective_max_precision =
Math.min(max_precision, available_significant_digits_for_precision);
const with_trailing_zeroes = n.toFixed(effective_max_precision);
// I want to keep the string and change just matching part,
// so here .replace() is a proper method to use.
const withouth_trailing_zeroes = with_trailing_zeroes.replace(/\.?0*$/, '');
return withouth_trailing_zeroes;
}
So, you got the number formatted the way you want. What now?
What can you use this string for?
Well, you can display it to the user. And that's mostly it. The value was rounded to (1) represent it in a different base and (2) fit in limited precision, so it's pretty much useless for any computation. And, BTW, why would you convert it to String in the first place, if what you want is a number?
Was the value you are trying to print ever useful in the first place?
Well, that's the most serious question here. Because, you know, floating point numbers are tricky. And they are absolutely abysmal for representing money. So, most likely the number you are trying to format is already a wrong number.
What to use instead?
Fixed-point arithmetic is the most obvious answer. Works most of the time. However, it's pretty tricky in JS, where number may slip into floating-point representation almost any time. So, it's better to use decimal arithmetic library. Optionally, switch to a language that has built-in bignums and decimals, like Python.
I'm a javascript code monkey, so this is virgin territory for me.
I have two "strings" that are just zeros and ones:
var first = "00110101011101010010101110100101010101010101010";
var second = "11001010100010101101010001011010101010101010101";
I want to perform a bitwise & (which I've never before worked with) to determine if there's any index where 1 appears in both strings.
These could potentially be VERY long strings (in the thousands of characters). I thought about adding them together as numbers, then converting to strings and checking for a 2, but javascript can't hold precision in large intervals and I get back numbers as strings like "1.1111111118215729e+95", which doesn't really do me much good.
Can I take two strings of unspecified length (they may not be the same length either) and somehow use a bitwise & to compare them?
I've already built the loop-through-each-character solution, but 1001^0110 would strike me as a major performance upgrade. Please do not give the javascript looping solution as an answer, this question is about using bitwise operators.
As you already noticed yourself, javascript has limited capabilities if it's about integer values. You'll have to chop your strings into "edible" portions and work your way through them. Since the parseInt() function accepts a base, you could convert 64 characters to an 8 byte int (or 32 to a 4 byte int) and use an and-operator to test for set bits (if (a & b != 0))
var first = "00110101011101010010101110100101010101010101010010001001010001010100011111",
second = "10110101011101010010101110100101010101010101010010001001010001010100011100",
firstInt = parseInt(first, 2),
secondInt = parseInt(second, 2),
xorResult = firstInt ^ secondInt, //524288
xorString = xorResult.toString(2); //"10000000000000000000"
I am using this line to generate a sha1 id for node.js:
crypto.createHash('sha1').digest('hex');
The problem is that it's returning the same id every time.
Is it possible to have it generate a random id each time so I can use it as a database document id?
243,583,606,221,817,150,598,111,409x more entropy
I'd recommend using crypto.randomBytes. It's not sha1, but for id purposes, it's quicker, and just as "random".
var id = crypto.randomBytes(20).toString('hex');
//=> f26d60305dae929ef8640a75e70dd78ab809cfe9
The resulting string will be twice as long as the random bytes you generate; each byte encoded to hex is 2 characters. 20 bytes will be 40 characters of hex.
Using 20 bytes, we have 256^20 or 1,461,501,637,330,902,918,203,684,832,716,283,019,655,932,542,976 unique output values. This is identical to SHA1's 160-bit (20-byte) possible outputs.
Knowing this, it's not really meaningful for us to shasum our random bytes. It's like rolling a die twice but only accepting the second roll; no matter what, you have 6 possible outcomes each roll, so the first roll is sufficient.
Why is this better?
To understand why this is better, we first have to understand how hashing functions work. Hashing functions (including SHA1) will always generate the same output if the same input is given.
Say we want to generate IDs but our random input is generated by a coin toss. We have "heads" or "tails"
% echo -n "heads" | shasum
c25dda249cdece9d908cc33adcd16aa05e20290f -
% echo -n "tails" | shasum
71ac9eed6a76a285ae035fe84a251d56ae9485a4 -
If "heads" comes up again, the SHA1 output will be the same as it was the first time
% echo -n "heads" | shasum
c25dda249cdece9d908cc33adcd16aa05e20290f -
Ok, so a coin toss is not a great random ID generator because we only have 2 possible outputs.
If we use a standard 6-sided die, we have 6 possible inputs. Guess how many possible SHA1 outputs? 6!
input => (sha1) => output
1 => 356a192b7913b04c54574d18c28d46e6395428ab
2 => da4b9237bacccdf19c0760cab7aec4a8359010b0
3 => 77de68daecd823babbb58edb1c8e14d7106e83bb
4 => 1b6453892473a467d07372d45eb05abc2031647a
5 => ac3478d69a3c81fa62e60f5c3696165a4e5e6ac4
6 => c1dfd96eea8cc2b62785275bca38ac261256e278
It's easy to delude ourselves by thinking just because the output of our function looks very random, that it is very random.
We both agree that a coin toss or a 6-sided die would make a bad random id generator, because our possible SHA1 results (the value we use for the ID) are very few. But what if we use something that has a lot more outputs? Like a timestamp with milliseconds? Or JavaScript's Math.random? Or even a combination of those two?!
Let's compute just how many unique ids we would get ...
The uniqueness of a timestamp with milliseconds
When using (new Date()).valueOf().toString(), you're getting a 13-character number (e.g., 1375369309741). However, since this a sequentially updating number (once per millisecond), the outputs are almost always the same. Let's take a look
for (var i=0; i<10; i++) {
console.log((new Date()).valueOf().toString());
}
console.log("OMG so not random");
// 1375369431838
// 1375369431839
// 1375369431839
// 1375369431839
// 1375369431839
// 1375369431839
// 1375369431839
// 1375369431839
// 1375369431840
// 1375369431840
// OMG so not random
To be fair, for comparison purposes, in a given minute (a generous operation execution time), you will have 60*1000 or 60000 uniques.
The uniqueness of Math.random
Now, when using Math.random, because of the way JavaScript represents 64-bit floating point numbers, you'll get a number with length anywhere between 13 and 24 characters long. A longer result means more digits which means more entropy. First, we need to find out which is the most probable length.
The script below will determine which length is most probable. We do this by generating 1 million random numbers and incrementing a counter based on the .length of each number.
// get distribution
var counts = [], rand, len;
for (var i=0; i<1000000; i++) {
rand = Math.random();
len = String(rand).length;
if (counts[len] === undefined) counts[len] = 0;
counts[len] += 1;
}
// calculate % frequency
var freq = counts.map(function(n) { return n/1000000 *100 });
By dividing each counter by 1 million, we get the probability of the length of number returned from Math.random.
len frequency(%)
------------------
13 0.0004
14 0.0066
15 0.0654
16 0.6768
17 6.6703
18 61.133 <- highest probability
19 28.089 <- second highest probability
20 3.0287
21 0.2989
22 0.0262
23 0.0040
24 0.0004
So, even though it's not entirely true, let's be generous and say you get a 19-character-long random output; 0.1234567890123456789. The first characters will always be 0 and ., so really we're only getting 17 random characters. This leaves us with 10^17 +1 (for possible 0; see notes below) or 100,000,000,000,000,001 uniques.
So how many random inputs can we generate?
Ok, we calculated the number of results for a millisecond timestamp and Math.random
100,000,000,000,000,001 (Math.random)
* 60,000 (timestamp)
-----------------------------
6,000,000,000,000,000,060,000
That's a single 6,000,000,000,000,000,060,000-sided die. Or, to make this number more humanly digestible, this is roughly the same number as
input outputs
------------------------------------------------------------------------------
( 1×) 6,000,000,000,000,000,060,000-sided die 6,000,000,000,000,000,060,000
(28×) 6-sided die 6,140,942,214,464,815,497,21
(72×) 2-sided coins 4,722,366,482,869,645,213,696
Sounds pretty good, right ? Well, let's find out ...
SHA1 produces a 20-byte value, with a possible 256^20 outcomes. So we're really not using SHA1 to it's full potential. Well how much are we using?
node> 6000000000000000060000 / Math.pow(256,20) * 100
A millisecond timestamp and Math.random uses only 4.11e-27 percent of SHA1's 160-bit potential!
generator sha1 potential used
-----------------------------------------------------------------------------
crypto.randomBytes(20) 100%
Date() + Math.random() 0.00000000000000000000000000411%
6-sided die 0.000000000000000000000000000000000000000000000411%
A coin 0.000000000000000000000000000000000000000000000137%
Holy cats, man! Look at all those zeroes. So how much better is crypto.randomBytes(20)? 243,583,606,221,817,150,598,111,409 times better.
Notes about the +1 and frequency of zeroes
If you're wondering about the +1, it's possible for Math.random to return a 0 which means there's 1 more possible unique result we have to account for.
Based on the discussion that happened below, I was curious about the frequency a 0 would come up. Here's a little script, random_zero.js, I made to get some data
#!/usr/bin/env node
var count = 0;
while (Math.random() !== 0) count++;
console.log(count);
Then, I ran it in 4 threads (I have a 4-core processor), appending the output to a file
$ yes | xargs -n 1 -P 4 node random_zero.js >> zeroes.txt
So it turns out that a 0 is not that hard to get. After 100 values were recorded, the average was
1 in 3,164,854,823 randoms is a 0
Cool! More research would be required to know if that number is on-par with a uniform distribution of v8's Math.random implementation
Have a look here: How do I use node.js Crypto to create a HMAC-SHA1 hash?
I'd create a hash of the current timestamp + a random number to ensure hash uniqueness:
var current_date = (new Date()).valueOf().toString();
var random = Math.random().toString();
crypto.createHash('sha1').update(current_date + random).digest('hex');
Do it in the browser, too !
EDIT: this didn't really fit into the flow of my previous answer. I'm leaving it here as a second answer for people that might be looking to do this in the browser.
You can do this client side in modern browsers, if you'd like
// str byteToHex(uint8 byte)
// converts a single byte to a hex string
function byteToHex(byte) {
return ('0' + byte.toString(16)).slice(-2);
}
// str generateId(int len);
// len - must be an even number (default: 40)
function generateId(len = 40) {
var arr = new Uint8Array(len / 2);
window.crypto.getRandomValues(arr);
return Array.from(arr, byteToHex).join("");
}
console.log(generateId())
// "1e6ef8d5c851a3b5c5ad78f96dd086e4a77da800"
console.log(generateId(20))
// "d2180620d8f781178840"
Browser requirements
Browser Minimum Version
--------------------------
Chrome 11.0
Firefox 21.0
IE 11.0
Opera 15.0
Safari 5.1
If Want To Get Unique Identifiers, You should use UUID (Universally Unique Identifier) / GUID (Globally Unique Identifier).
A Hash is Supposed to be Deterministic & Unique & of Fixed Length For Input of any size. So no matter how many times you run the hash function, the output will be the same if you use the same input.
UUIDs Are Unique & Randomly Generated!
There Is A Package called 'uuid' you can install it via npm by
npm install uuid
& In your code import the module by
const { v4:uuidv4} = require('uuid');
// Call The Method uuidv4 or whatever you name it while importing & log it or store it or assign it. The method return a UUID in the form of a string.
console.log(uuidv4());
// Example Output : '59594fc8-6a35-4f50-a966-4d735d8402ea'
Here is the npm link (if you need it) :
https://www.npmjs.com/package/uuid
Using crypto is a good approach cause it's native and stable module,
but there are cases where you can use bcrypt if you want to create a really strong and secure hash. I use it for passwords it has a lot of techniques for hashing, creating salt and comparing passwords.
Technique 1 (generate a salt and hash on separate function calls)
const salt = bcrypt.genSaltSync(saltRounds);
const hash = bcrypt.hashSync(myPlaintextPassword, salt);
Technique 2 (auto-gen a salt and hash):
const hash = bcrypt.hashSync(myPlaintextPassword, saltRounds);
For more examples you can check here: https://www.npmjs.com/package/bcrypt