Floating-point value from bytes reading completely wrong - Node.js - javascript

I am attempting to read a 32-bit IEEE-754 float from a buffer, but it is not reading correctly at all. For example, [00 00 00 3f] becomes 8.828180325246348e-44 instead of 0.5. I have also noticed that negative floats never convert properly. For example, [00 00 20 c1] becomes 1.174988762336359e-41, not -10.0. What am I doing wrong? Is this some floating-point precision issue? This is my code:
function readFloat() {
const value = this.data.readFloatBE(this.offset);
this.offset += 4;
return value;
}
this.data being a Buffer, this.offset being the offset currently read in bytes.
One thing to note is that even with something like this in vanilla JavaScript, I get the same results:
function floatFromBytes(bytes) {
buf = new ArrayBuffer(4);
v = new DataView(buf);
bytes.forEach((b, i) => {
v.setUint8(i, b);
})
return v.getFloat32(0);
}
floatFromBytes([0x00, 0x00, 0x20, 0xc1]); // should be -0.5, but is 1.174988762336359e-41
EDIT: Resolved, turns out the bytes were reversed for some reason.
New code:
function readFloat() {
// This is a bit of a weird IEEE 754 float implementation, but it works
let buf = new ArrayBuffer(4);
let view = new DataView(buf);
let bytes = this.readBytes(4);
// reverse the bytes
for (let i = 0; i < 4; i++) {
view.setUint8(i, bytes[3 - i]);
}
return view.getFloat32(0);
}

As you noted this is just an endian issue. Different systems expect bytes to be ordered in different ways, the most common ordering at the moment is little endian (used by Intel x86 compatible processors, and ARM systems are commonly set to use this mode).
Because JavaScript tries to be CPU agnostic it asks you to choose which order you want to interpret things. The BE in Buffer.readFloatBE stands for big-endian, and there's also a LE version which is what you probably want to use.
For example:
Buffer.from('0000003f','hex').readFloatLE() // => 0.5
Buffer.from('000020c1','hex').readFloatLE() // => -10.0

Related

How do I get the actual size in bytes for a number and a string in JavaScript in a browser environment?

I am trying to get the actual size (in bytes) of a number and a string in browsers e.g. chrome.
I learned that in JavaScript numbers are represented in double precision takes up 64 bits and strings are UTF-16 code unit so it takes either 2 bytes or 4 bytes.
I first tried to use new Blob but it encodes string component characters as UTF-8 not UTF-16. And I know there is a Buffer.from API in Node but it is not available in a browser environment.
My question is how I can get the actual size of a number and a string in bytes from a browser, e.g. chrome?
You can do that natively with the help of TextEncoder
let str1 = 'Beta'; // 'Beta' text in English
let str2 = '贝塔'; // 'Beta' text in Chinese
const encoder = new TextEncoder();
const len1 = encoder.encode(str1).length;
const len2 = encoder.encode(str2).length;
console.log(len1); // 4
console.log(len2); // 6
First of all it is important to realize that the spec doesn't mandate any representation. Just behavior.
Strings are stored in UTF-16 but fortunately for your purpose each index represents 16 bits.
For example
console.log('😠'.length); // Should log 2 because emoji takes 2 16 bit parts
For numbers it depends. V8 represents small integer numbers as actual 32 bit ints.
With https://github.com/substack/node-browserify you can work with buffers in the Browser by using: https://github.com/toots/buffer-browserify.
//in your browsify js file:
require('buffer')
Buffer.byteLength(String.fromCharCode(55555), 'utf16')
Buffer.byteLength(String.fromCharCode(55555, 57000), 'utf16')
iconv-lite: Pure JS character encoding conversion
var iconv = require('iconv-lite');
var buf =iconv.encode("Hello World", 'utf16');
console.log(buf);
console.log(buf.length); // returns 24
Here is my answer to your problem :
function getBytesFromVar(theVar) {
if(theVar !== null && theVar !== undefined) {
switch (typeof theVar) {
case 'string' : {
var encoder = new TextEncoder();
encoder['encoding'] = "utf-16";
return encoder['encode'](theVar).length * 2;
}
case 'number' : {
return 8;
}
case 'boolean' : {
return 4;
}
case 'object' : {
if ( theVar instanceof String) {
var encoder = new TextEncoder();
encoder['encoding'] = "utf-16";
return encoder['encode'](theVar.toString()).length * 2;
} else {
return 0;
}
}
}
}
else {
return 0;
}
}
The getBytesFromVar function take a var and return the number of byte used.
Function use TextEncoder to get the string length and then calculate the bytes.
In case of a string created with:
let str = new String('Alain♥');
function will work with String objects.
ATTENTION: this can't be used to calculate memory footprint in browser as other mechanism of memory management can increase/decrease this values.
Vars can be allocated in different memory segment. For example, String object are created on the heap and string vars are created on string constant pool.
Also vars are manipulated through pointer.
For example, 2 strings vars that contain the same string are created on the string constant pool. First will be allocated, but the second one will be a pointer to the first one. So the size in memory byte will not be simply twice the size of the string.
Good post about that: https://levelup.gitconnected.com/bytefish-vs-new-string-bytefish-what-is-the-difference-a795f6a7a08b
Use case:
var myString='Alain♥';
var myNumber = 120;
var objString = new String('Alain♥');
var myFloat = 105.456;
console.log('%o is %o bytes', myString, getBytesFromVar(myString));
console.log('%o is %o bytes', myNumber, getBytesFromVar(myNumber));
console.log('%o is %o bytes', objString, getBytesFromVar(objString));
console.log('%o is %o bytes', myFloat, getBytesFromVar(myFloat));
I have used the npm module object-sizeof for this. You can use it to get the size of integer or string variables in bytes. This is a sample usage,
var sizeof = require('object-sizeof');
console.log(sizeof(123)); //prints 8
You can do that natively with the help of TextEncoder
let str1 = 'Beta'; // 'Beta' text in English
let str2 = '贝塔'; // 'Beta' text in Chinese
const encoder = new TextEncoder();
const len1 = encoder.encode(str1).length;
const len2 = encoder.encode(str2).length;
console.log(len1); // 4
console.log(len2); // 6

Getting wrong result for binary to decimal even after using BigInt() in javascript

I am trying to add two given binary strings after converting them to decimals(Numbers) and then converting back the resulting decimal(Number) to string.
I am getting the wrong binary to decimal even after using BigInt().
let a = "10100000100100110110010000010101111011011001101110111111111101000000101111001110001111100001101";
let b="110101001011101110001111100110001010100001101011101010000011011011001011101111001100000011011110011";
var twoSum = function(a, b) {
let a1=BigInt(parseInt(a, 2));
let b1=BigInt(parseInt(b,2));
let aStr=a1.toString(10);
let bStr=b1.toString(10);
console.log(aStr)
console.log(bStr)
};
console.log(twoSum(a, b));
Output:
24847893154024981755840167936
526700554598729745018195542016
Correct result is : 24847893154024981730169397005 & 526700554598729746900966573811
I don't why I am getting the wrong result of binary to decimal.
parseInt returns a Number. Due to Number's limited precision being less than the length of your input strings, you've lost precision at that point. Converting that Number to a BigInt afterwards doesn't bring the lost precision back.
The solution is to convert the strings to BigInt directly. There was supposed to be a BigInt.parseInt function, but TC39 (the committee that standardizes JavaScript) never got around to finishing that. In the meantime, for non-decimal inputs, the BigInt constructor does understand strings starting with 0x... (hex), 0o... (octal), and 0b... (binary). The latter is what you want in this case. So you could fix your code as follows:
function parseBinaryToBigInt(a) {
return BigInt('0b' + a);
}
function twoSum(a, b) {
let a1 = parseBinaryToBigInt(a);
let b1 = parseBinaryToBigInt(b);
let aStr = a1.toString(10);
let bStr = b1.toString(10);
console.log(aStr)
console.log(bStr)
};
let a = "10100000100100110110010000010101111011011001101110111111111101000000101111001110001111100001101";
// Binary to decimal
const d = BigInt('0b' + a);
// Turn the binary
const b = BigInt(a);

Is there any way to see a number in it's 64 bit float IEEE754 representation

Javascript stores all numbers as double-precision 64-bit format IEEE 754 values according to the spec:
The Number type has exactly 18437736874454810627 (that is, 264−253+3)
values, representing the double-precision 64-bit format IEEE 754
values as specified in the IEEE Standard for Binary Floating-Point
Arithmetic
Is there any way to see the number in this form in Javascript?
You can use typed arrays to examine the raw bytes of a number. Create a Float64Array with one element, and then create a Uint8Array with the same buffer. You can then set the first element of the float array to your number, and examine the bytes via the Uint8Array. You'll have to do some shifting and combining for the various pieces of the number of course, but it's not hard.
There are no built-in facilities to do things like extract the exponent.
Based on #Pointy's suggestion I've implemented the following function to get a number in it's 64 bit float IEEE754 representation:
function to64bitFloat(number) {
var f = new Float64Array(1);
f[0] = number;
var view = new Uint8Array(f.buffer);
var i, result = "";
for (i = view.length - 1; i >= 0; i--) {
var bits = view[i].toString(2);
if (bits.length < 8) {
bits = new Array(8 - bits.length).fill('0').join("") + bits;
}
result += bits;
}
return result;
}
console.log(to64bitFloat(12)); // 0100000000101000000000000000000000000000000000000000000000000000
console.log(to64bitFloat(-12)); // 1100000000101000000000000000000000000000000000000000000000000000
You can use Basenumber.js to transform a number into its IEEE754 representation:
let x = Base(326.9);
let y = Base(-326.9);
// specify double precision (64)
let a = x.toIEEE754(64);
let b = y.toIEEE754(64);
console.log(a);
console.log(b);
// You can join them in an unique string this way
console.log(Object.values(a).join(""));
console.log(Object.values(b).join(""));
<script src='https://cdn.jsdelivr.net/gh/AlexSp3/Basenumber.js#main/BaseNumber.min.js'></script>

Reading variable length bits from a binary string

Im new to javascript and node.js, I have a base64 encoded string of data that I need to parse several values from which are of various bit lengths.
I figured I would start by using the Buffer object to read the b64 string but from there I am completely lost.
The data are a series of unsigned integers, The format is something akin to this:
Header:
8 bits - uint
3 bits - uint
2 bits - uint
3 bits - unused padding
6 bits - uint
After that there are recurring sections of either 23 bit or 13 bit length of data each with a couple of fields I need to extract.
An example of a 23 bit section:
3 bit - uint
10 bit - uint
10 bit - uint
My question is this, What is the best way to take an arbitrary number of bits and put the resulting value in a separate uint? Note that some of the values are multi-byte (> 8 bits) so I cant step byte for byte.
I apologize if my explanation is kind of vague but hopefully it will suffice.
One simple way to read any amount of bits is e.g.
function bufferBitReader(buffer) {
var bitPos = 0;
function readOneBit() {
var offset = Math.floor(bitPos / 8),
shift = 7 - bitPos % 8;
bitPos += 1;
return (buffer[offset] >> shift) & 1;
}
function readBits(n) {
var i, value = 0;
for (i = 0; i < n; i += 1) {
value = value << 1 | readOneBit();
}
return value;
}
function isEnd() {
return Math.floor(bitPos / 8) >= buffer.length;
}
return {
readOneBit: readOneBit,
readBits: readBits,
isEnd: isEnd
};
}
You just take your but buffer and initialize the reader by
var bitReader = bufferBitReader(buffer);
Then you can read any number of bits by calling
bitReader.readBits(8);
bitReader.readBits(3);
bitReader.readBits(2);
...
You can test whether you already read all bits by
bitReader.isEnd()
One thing to make sure is the actual order of bit that is expected... some 'bit streams' are expected to get bits from the least significant to the most significant.. this code expects the opposite that the first bit you read is the most significant of the first byte...

Node.JS Big-Endian UCS-2

I'm working with Node.JS. Node's buffers support little-endian UCS-2, but not big-endian, which I need. How would I do so?
According to wikipedia, UCS-2 should always be big-endian so it's odd that node only supports little endian. You might consider filing a bug. That said, switching endian-ness is fairly straight-forward since it's just a matter of byte order. So just swap bytes around to go back and forth between little and big endian, like so:
function swapBytes(buffer) {
var l = buffer.length;
if (l & 0x01) {
throw new Error('Buffer length must be even');
}
for (var i = 0; i < l; i += 2) {
var a = buffer[i];
buffer[i] = buffer[i+1];
buffer[i+1] = a;
}
return buffer;
}

Categories

Resources