Develop Reference

Hash algorithm issue with converting code from javascript to python

To get same hash function in javascript and python I tried to convert my javaScript function to python and completely stack with gooooogle int as result in python variant.
javascript
function str_hash(var s) {
var hash = 0, i, chr;
if (s.length === 0) return hash;
for (i = 0; i < s.length; i++)
{
chr = s.charCodeAt(i);
hash = ((hash << 5) - hash) + chr;
hash |= 0;
}
return hash;
};
output is like: -34998534 whatever
my python try:
def get_hash(s):
h = 0
if not s:
return h
for i in range(0, len(s)):
h = ((h << 5) - h) + i
h |= 0
return h
print(get_hash('PUT LONG STRING HERE'))
output is like: 349832894283058945028049523548405975892375489743847490218348927483984793048218934148973940218340298489273942374902938490238482942930729487210948239407329403082738940214 whatever
Any ideas how to fix it?

here is the python equivalent code
def str_hash(s):
hash = 0
for i in range(len(s)):
chr = ord(s[i])
hash = ((hash << 5) - hash) + chr
hash = hash & 0xFFFFFFFF
return hash
print(str_hash("PUT LONG STRING HERE"))
the << operator in javascript is called the bitwise left shift operator. and the native code for this is explained in the below example
let num = 4
let shiftBy = 2
print(num * 2 ** shiftBy) // num * Math.pow(2,shiftBy)
print(num << shiftBy)

Javascript equivalent of Java's UUID class

In java, you can do something like
UUID id = UUID.fromString("eb66c416-4739-465b-9af3-9dc33ed8eef9");
long msb = id.getMostSignificantBits();
long lsb = id.getLeastSignificantBits();
System.out.println(msb + ", " + lsb);
// -1484283427208739237, -7281302710629372167
System.out.println(new UUID(msb, lsb));
// eb66c416-4739-465b-9af3-9dc33ed8eef9
This same example is referenced in another question which is pretty similar, so this would be a follow up. While in the related question problem of lsb, msb -> string was solved, I cannot find solution for reverse problem, string -> msb, lsb
The original solution was
function toUuidString(lsb, msb) {
return `${digits(msb >> 32n, 8n)}-${digits(msb >> 16n, 4n)}-${digits(
msb,
4n
)}-${digits(lsb >> 48n, 4n)}-${digits(lsb, 12n)}`
}
function digits(value, ds) {
const hi = 1n << (ds * 4n)
return (hi | (value & (hi - 1n))).toString(16).slice(1)
}
Now I'd like to have a function that takes in string and returns msb and lsb.
Following original questions' paths, I've discovered java source code and tried to do the equivalent, which would be:
function fromString(name) {
let components = name.split('-')
if (components.length !== 5) {
throw new Error(`Invalid UUID string: ${name}`)
}
for (let index = 0; index < 5; index++) {
components[index] = `0x${components[index]}`
}
let mostSigBits = Number.parseInt(components[0], 16)
mostSigBits <<= 16
mostSigBits |= Number.parseInt(components[1], 16)
mostSigBits <<= 16
mostSigBits |= Number.parseInt(components[2], 16)
let leastSigBits = Number.parseInt(components[3], 16)
leastSigBits <<= 48
leastSigBits |= Number.parseInt(components[4], 16)
return {
leastSigBits,
mostSigBits,
}
}
However, when I try to test this with something like:
const originalUuid = 'eb66c416-4739-465b-9af3-9dc33ed8eef9'
const parts = fromString(originalUuid)
const newUUid = toUuidString(
BigInt(parts.leastSigBits),
BigInt(parts.mostSigBits)
)
console.log('Original', originalUuid)
console.log('New', newUUid)
I do not get equivalent uuids. They have equivalent parts but some parts are missing
Original eb66c416-4739-465b-9af3-9dc33ed8eef9
New 00000000-4739-465b-ffff-ffffbefbeef9
Any ideas what went wrong?

Finally I found the problem - two codes were not strictly equivalent, java source code declared mostSigBits and leastSigBits as long, which cannot be represented in javascript, so we need to use BigInt.
To sum up my question and previous question, javascript equivalent for java's UUID operations would be:
string -> msb, lsb
function fromString(name) {
let components = name.split('-')
if (components.length !== 5) {
throw new Error(`Invalid UUID string: ${name}`)
}
for (let index = 0; index < 5; index++) {
components[index] = `0x${components[index]}`
}
let mostSigBits = BigInt(Number.parseInt(components[0], 16))
mostSigBits <<= 16n
mostSigBits |= BigInt(Number.parseInt(components[1], 16))
mostSigBits <<= 16n
mostSigBits |= BigInt(Number.parseInt(components[2], 16))
let leastSigBits = BigInt(Number.parseInt(components[3], 16))
leastSigBits <<= 48n
leastSigBits |= BigInt(Number.parseInt(components[4], 16))
return {
leastSigBits,
mostSigBits,
}
}
msb, lsb -> string (from the referenced question)
function toUuidString(lsb, msb) {
return `${digits(msb >> 32n, 8n)}-${digits(msb >> 16n, 4n)}-${digits(
msb,
4n
)}-${digits(lsb >> 48n, 4n)}-${digits(lsb, 12n)}`
}
function digits(value, ds) {
const hi = 1n << (ds * 4n)
return (hi | (value & (hi - 1n))).toString(16).slice(1)
}

How to build CRC32 table for Ogg?

From this answer I adapted the code below:
function _makeCRCTable() {
const CRCTable = new Uint32Array(256);
for (let i = 256; i--;) {
let char = i;
for (let j = 8; j--;) {
char = char & 1 ? 3988292384 ^ char >>> 1 : char >>> 1;
}
CRCTable[i] = char;
}
return CRCTable;
}
This code generates table as here, but for Ogg I need another table - as here.
From Ogg documentation:
32 bit CRC value (direct algorithm, initial val and final XOR = 0,
generator polynomial=0x04c11db7)
parseInt('04c11db7', 16)
return 79764919 - I tried this polynomial but resulting table is not correct.
I am new to the CRC field, as I found there are a few variations of CRC32 algorithm.

I'm not sure of javascript precedence, but the xor needs to occur after the shift:
char = char & 1 ? 3988292384 ^ (char >>> 1) : char >>> 1;
However the first table you show seems correct, as table[128] = table[0x80] = 3988292384 = 0xEDB88320 which is 0x104c11db7 bit reversed, then shifted right one bit.
The second table you have is for a left shifting CRC, where table[1] = x04c11db7. In this case the inner loop would include something like this:
let char = i << 24;
for (let j = 8; j--;) {
char = char & 0x80000000 ? 0x04c11db7 ^ char << 1 : char << 1;
}
Example C code for comparison, generates crc for the patterns {0x01}, {0x01,0x00}, {0x01,0x00,0x00}, {0x01,0x00,0x00,0x00}.
#include <stdio.h>
typedef unsigned char uint8_t;
typedef unsigned int uint32_t;
uint32_t crctbl[256];
void gentbl(void)
{
uint32_t crc;
uint32_t b;
uint32_t c;
uint32_t i;
for(c = 0; c < 0x100; c++){
crc = c<<24;
for(i = 0; i < 8; i++){
b = crc>>31;
crc <<= 1;
crc ^= (0 - b) & 0x04c11db7;
}
crctbl[c] = crc;
}
}
uint32_t crc32(uint8_t * bfr, size_t size)
{
uint32_t crc = 0;
while(size--)
crc = (crc << 8) ^ crctbl[(crc >> 24)^*bfr++];
return(crc);
}
int main(int argc, char** argv)
{
uint32_t crc;
uint8_t bfr[4] = {0x01,0x00,0x00,0x00};
gentbl();
crc = crc32(bfr, 1); /* 0x04c11db7 */
printf("%08x\n", crc);
crc = crc32(bfr, 2); /* 0xd219c1dc */
printf("%08x\n", crc);
crc = crc32(bfr, 3); /* 0x01d8ac87 */
printf("%08x\n", crc);
crc = crc32(bfr, 4); /* 0xdc6d9ab7 */
printf("%08x\n", crc);
return(0);
}
For JS:
function _makeCRC32Table() {
const polynomial = 79764919;
const mask = 2147483648;
const CRCTable = new Uint32Array(256);
for (let i = 256; i--;) {
let char = i << 24;
for (let j = 8; j--;) {
char = char & mask ? polynomial ^ char << 1 : char << 1;
}
CRCTable[i] = char;
}
return CRCTable;
}
How to use this table:
[1, 0].reduce((crc, byte) => crc << 8 >>> 0 ^ CRCTable[crc >>> 24 ^ byte], 0) >>> 0
Here we added >>> 0 that takes the module of the number - because there is no unsigned int in JS - JavaScript doesn't have integers. It only has double precision floating-point numbers.
Note that for Ogg you must set generated CRC in the reverse order.

16-bit binary arithmetic in Javascript

Javascript has only one number type: a 64-bit floating point.
Using Javascript, I need to implement a hashing algorithm which is designed to be written in C with 16 bit unsigned integers.
The main operation is something like this (pseudocode):
uint16 n = 0;
string s = "abcd1234";
for (int i = 0; i < s.length; i += 1) {
n ^= (n << 2) + (n >> 3) + s[i];
}
return n;
Of course, this produces one result when using uint16 values and a different result if n is a 64-bit floating point.
My best solution to the problem so far is to convert the results of each bitwise operation to <= 16 bits, using a function like this (javascript):
function uint16 (n) {
return parseInt(n.toString(2).slice(-16), 2);
}
and performing the operation something like this (javascript):
for (var i = 0; i < s.length; i +=1 ) {
n ^= uint16(uint16(n << 2) + uint16(n >>> 3) + s.charCodeAt(i));
}
but I'm not 100% confident this will always produce the correct results.
Is there any standard way to emulate 16-bit unsigned bitwise operations on a number value in Javascript?

You can use bitwise AND.
It operates on 32 bit integers, but you can just "and" with 0xffff.
function uint16 (n) {
return n & 0xFFFF;
}
Additionally, bit shift operations (<<, >>>) also operate on 32 bit integers, so you only really need to call the uint16 function before assignment:
for (var i = 0; i < s.length; i +=1 ) {
n ^= uint16((n << 2) + (n >>> 3) + s.charCodeAt(i));
}

I suspect #Amit's version will execute faster but this is a workaround I have had success with:
//create a typed container, we'll work only in the first index.
const idx = 0;
const n = new Uint16Array(1);
for (int i = 0; i < s.length; i += 1) {
n[idx] ^= (n[idx] << 2) + (n[idx] >> 3) + s[i];
}

Javascript CRC32C implementation that is compatible with Intel's SSE4.2 hardware implementation

before I start, a disclaimer: while I can go around C/C++ code, I am no wizard, nor did I ever did enough programming to call myself a capable programmer with it.
I'm trying to use CRC32C to validate data that is coming into our servers from browser. Currently both implementations use the same code (nodeJS on server), but we would like to switch to hardware implementation(blog post, github repo) (when available) and for that I need a correctly functioning version in the browser.
I tried to go with this implementation (and another, internally develop, but also not-working) but using correct polynom (0x82F63B78 instead of 0xEDB88320, and also 0x1EDC6F41 & 0x8F6E37A0) but no polynom that I used produces the correct output.
Continuing in my research I find a post from Mark Adler which includes a software implementation and decide to try to convert it to Javascript (to the best of my understanding of C).
The result:
function crc32c_table_intel() {
var POLY = 0x82f63b78;
var n, crc, k;
var crc32c_table = gen2darr(8, 256, 0);
for (n = 0; n < 256; n++) {
crc = n;
crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
crc32c_table[0][n] = crc;
}
for (n = 0; n < 256; n++) {
crc = crc32c_table[0][n];
for (k = 1; k < 8; k++) {
crc = crc32c_table[0][crc & 0xff] ^ (crc >> 8);
crc32c_table[k][n] = crc;
_crc_tmptable.push(crc32c_table[k][n]);
}
}
return crc32c_table;
}
function crc32c_sw(crci, str) {
var len = str.length;
var crc;
var crc32c_table = crc32c_table_intel();
crc = crci ^ 0xffffffff;
for(var next = 0; next < 7; next++) { // was: while (len && ((uintptr_t)next & 7) != 0) {
crc = crc32c_table[0][(crc ^ str.charCodeAt(next++)) & 0xff] ^ (crc >> 8);
len--;
}
while (len >= 8) {
// was: crc ^= *(uint64_t *)next;
crc ^= str.charCodeAt(next);
crc = crc32c_table[7][crc & 0xff] ^
crc32c_table[6][(crc >> 8) & 0xff] ^
crc32c_table[5][(crc >> 16) & 0xff] ^
crc32c_table[4][(crc >> 24) & 0xff] ^
crc32c_table[3][(crc >> 32) & 0xff] ^
crc32c_table[2][(crc >> 40) & 0xff] ^
crc32c_table[1][(crc >> 48) & 0xff] ^
crc32c_table[0][crc >> 56];
next += 1;
len -= 1;
}
while (len) {
// was: crc = crc32c_table[0][(crc ^ *next++) & 0xff] ^ (crc >> 8);
crc = crc32c_table[0][(crc ^ str.charCodeAt(next++)) & 0xff] ^ (crc >> 8);
len--;
}
return crc ^ 0xffffffff;
}
// a helper function
function gen2darr( rows, cols, defaultValue){
var arr = [];
for(var i=0; i < rows; i++){
arr.push([]);
arr[i].push( new Array(cols));
for(var j=0; j < cols; j++){
arr[i][j] = defaultValue;
}
}
return arr;
}
Still, no luck. No matter what function, what table or what polynom do I use, the results do not match:
SSE4.2: 606105071
JS (example): 1249991249
Then I go around thinking that it must be something with conversion from Javascript strings to C/C++ data and I see that nodeJS implementation uses UTF8 (https://github.com/Voxer/sse4_crc32/blob/master/src/sse4_crc32.cpp#L56) while Javascript uses UCS-2 encoding.
Now, the questions that I have are these:
Is any of these functions valid? The first seems so, for the one I posted I am not sure if I translated all the bitwise operations correctly
How to go around encoding issues? Is that even an encoding issue as I suspect? Does anyone has any other ideas how to ensure that nodeJS HW implementation and client side implemntation return the same output?
Thanks for any ideas!

See this answer for a compatible software implementation and a fast implementation using the hardware instruction.
You have a few problems there. One is that in Javascript you need to use a logical right shift >>> instead of an arithmetic right shift >>. Second is that you are using charCodeAt, which returns the Unicode value of a character, which may be more than one byte. The CRC algorithm operates on a sequence of bytes, not a sequence of Unicode characters. Third is that you're computing the same table every time -- the table should only be computed once. Last is you're jumping straight to a complicated implementation.
As a simple example, this will compute a CRC-32C in Javascript on an array of values expected to be integers in the range 0..255, i.e. bytes:
function crc32c(crc, bytes) {
var POLY = 0x82f63b78;
var n;
crc ^= 0xffffffff;
for (n = 0; n < bytes.length; n++) {
crc ^= bytes[n];
crc = crc & 1 ? (crc >>> 1) ^ POLY : crc >>> 1;
crc = crc & 1 ? (crc >>> 1) ^ POLY : crc >>> 1;
crc = crc & 1 ? (crc >>> 1) ^ POLY : crc >>> 1;
crc = crc & 1 ? (crc >>> 1) ^ POLY : crc >>> 1;
crc = crc & 1 ? (crc >>> 1) ^ POLY : crc >>> 1;
crc = crc & 1 ? (crc >>> 1) ^ POLY : crc >>> 1;
crc = crc & 1 ? (crc >>> 1) ^ POLY : crc >>> 1;
crc = crc & 1 ? (crc >>> 1) ^ POLY : crc >>> 1;
}
return crc ^ 0xffffffff;
}