I am new to node.js,
I was reading on Node.js Buffers.
I stumbled on buffer.copy method.
Wrote a code which is as follows,
var bufferOne = new Buffer("This is China");
var bufferTwo = new Buffer(20),
startTarget = 0,
sourceStart = 8,
sourceEnd = 0;
var bufferLength = bufferOne.length;
sourceEnd = bufferLength;
console.log("Buffer length "+bufferLength);
bufferOne.copy(bufferTwo,startTarget,sourceStart,sourceEnd);
console.log("Buffer1 "+bufferOne.toString());
console.log("Buffer2 "+bufferTwo.toString());
The output is as follows,
Buffer length 13
Buffer1 This is China
Buffer2 China
But sometimes it also prints garbage output after "China", as follows,
Buffer length 13
Buffer1 This is China
Buffer2 China���*�
Sometimes,
Buffer length 13
Buffer1 This is China
Buffer2 China���
Can somebody please tell me whats going on ?
The length of bufferTwo is 20, but you're only copying 13 bytes. The remaining 7 bytes contain semi-random data.
Documented here:
Passing a number as the first argument to Buffer() (e.g. new Buffer(10)), allocates a new Buffer object of the specified size. The memory allocated for such Buffer instances is not initialized and can contain sensitive data.
Related
I am trying to get the actual size (in bytes) of a number and a string in browsers e.g. chrome.
I learned that in JavaScript numbers are represented in double precision takes up 64 bits and strings are UTF-16 code unit so it takes either 2 bytes or 4 bytes.
I first tried to use new Blob but it encodes string component characters as UTF-8 not UTF-16. And I know there is a Buffer.from API in Node but it is not available in a browser environment.
My question is how I can get the actual size of a number and a string in bytes from a browser, e.g. chrome?
You can do that natively with the help of TextEncoder
let str1 = 'Beta'; // 'Beta' text in English
let str2 = '贝塔'; // 'Beta' text in Chinese
const encoder = new TextEncoder();
const len1 = encoder.encode(str1).length;
const len2 = encoder.encode(str2).length;
console.log(len1); // 4
console.log(len2); // 6
First of all it is important to realize that the spec doesn't mandate any representation. Just behavior.
Strings are stored in UTF-16 but fortunately for your purpose each index represents 16 bits.
For example
console.log('😠'.length); // Should log 2 because emoji takes 2 16 bit parts
For numbers it depends. V8 represents small integer numbers as actual 32 bit ints.
With https://github.com/substack/node-browserify you can work with buffers in the Browser by using: https://github.com/toots/buffer-browserify.
//in your browsify js file:
require('buffer')
Buffer.byteLength(String.fromCharCode(55555), 'utf16')
Buffer.byteLength(String.fromCharCode(55555, 57000), 'utf16')
iconv-lite: Pure JS character encoding conversion
var iconv = require('iconv-lite');
var buf =iconv.encode("Hello World", 'utf16');
console.log(buf);
console.log(buf.length); // returns 24
Here is my answer to your problem :
function getBytesFromVar(theVar) {
if(theVar !== null && theVar !== undefined) {
switch (typeof theVar) {
case 'string' : {
var encoder = new TextEncoder();
encoder['encoding'] = "utf-16";
return encoder['encode'](theVar).length * 2;
}
case 'number' : {
return 8;
}
case 'boolean' : {
return 4;
}
case 'object' : {
if ( theVar instanceof String) {
var encoder = new TextEncoder();
encoder['encoding'] = "utf-16";
return encoder['encode'](theVar.toString()).length * 2;
} else {
return 0;
}
}
}
}
else {
return 0;
}
}
The getBytesFromVar function take a var and return the number of byte used.
Function use TextEncoder to get the string length and then calculate the bytes.
In case of a string created with:
let str = new String('Alain♥');
function will work with String objects.
ATTENTION: this can't be used to calculate memory footprint in browser as other mechanism of memory management can increase/decrease this values.
Vars can be allocated in different memory segment. For example, String object are created on the heap and string vars are created on string constant pool.
Also vars are manipulated through pointer.
For example, 2 strings vars that contain the same string are created on the string constant pool. First will be allocated, but the second one will be a pointer to the first one. So the size in memory byte will not be simply twice the size of the string.
Good post about that: https://levelup.gitconnected.com/bytefish-vs-new-string-bytefish-what-is-the-difference-a795f6a7a08b
Use case:
var myString='Alain♥';
var myNumber = 120;
var objString = new String('Alain♥');
var myFloat = 105.456;
console.log('%o is %o bytes', myString, getBytesFromVar(myString));
console.log('%o is %o bytes', myNumber, getBytesFromVar(myNumber));
console.log('%o is %o bytes', objString, getBytesFromVar(objString));
console.log('%o is %o bytes', myFloat, getBytesFromVar(myFloat));
I have used the npm module object-sizeof for this. You can use it to get the size of integer or string variables in bytes. This is a sample usage,
var sizeof = require('object-sizeof');
console.log(sizeof(123)); //prints 8
You can do that natively with the help of TextEncoder
let str1 = 'Beta'; // 'Beta' text in English
let str2 = '贝塔'; // 'Beta' text in Chinese
const encoder = new TextEncoder();
const len1 = encoder.encode(str1).length;
const len2 = encoder.encode(str2).length;
console.log(len1); // 4
console.log(len2); // 6
I am attempting to read a 32-bit IEEE-754 float from a buffer, but it is not reading correctly at all. For example, [00 00 00 3f] becomes 8.828180325246348e-44 instead of 0.5. I have also noticed that negative floats never convert properly. For example, [00 00 20 c1] becomes 1.174988762336359e-41, not -10.0. What am I doing wrong? Is this some floating-point precision issue? This is my code:
function readFloat() {
const value = this.data.readFloatBE(this.offset);
this.offset += 4;
return value;
}
this.data being a Buffer, this.offset being the offset currently read in bytes.
One thing to note is that even with something like this in vanilla JavaScript, I get the same results:
function floatFromBytes(bytes) {
buf = new ArrayBuffer(4);
v = new DataView(buf);
bytes.forEach((b, i) => {
v.setUint8(i, b);
})
return v.getFloat32(0);
}
floatFromBytes([0x00, 0x00, 0x20, 0xc1]); // should be -0.5, but is 1.174988762336359e-41
EDIT: Resolved, turns out the bytes were reversed for some reason.
New code:
function readFloat() {
// This is a bit of a weird IEEE 754 float implementation, but it works
let buf = new ArrayBuffer(4);
let view = new DataView(buf);
let bytes = this.readBytes(4);
// reverse the bytes
for (let i = 0; i < 4; i++) {
view.setUint8(i, bytes[3 - i]);
}
return view.getFloat32(0);
}
As you noted this is just an endian issue. Different systems expect bytes to be ordered in different ways, the most common ordering at the moment is little endian (used by Intel x86 compatible processors, and ARM systems are commonly set to use this mode).
Because JavaScript tries to be CPU agnostic it asks you to choose which order you want to interpret things. The BE in Buffer.readFloatBE stands for big-endian, and there's also a LE version which is what you probably want to use.
For example:
Buffer.from('0000003f','hex').readFloatLE() // => 0.5
Buffer.from('000020c1','hex').readFloatLE() // => -10.0
I'm making a rhythm game and I need a quick way to get the length of an ogg file. The only way I could think would be to stream the file really fast without playing it but if I have hundreds of songs this would obviously not be practical. Another way would be to store the length of the file in some sort of properties file but I would like to avoid this. I know there must be some way to do this as most music players can tell you the length of a song.
The quickest way to do it is to seek to the end of the file, then back up to the last Ogg page header you find and read its granulePosition (which is the total number of samples per channel in the file). That's not foolproof (you might be looking at a chained file, in which case you're only getting the last stream's length), but should work for the vast majority of Ogg files out there.
If you need help with reading the Ogg page header, you can read the Jorbis source code... The short version is to look for "OggS", read a byte (should be 0), read a byte (only bit 3 should be set), then read a 64-bit little endian value.
I implemented the solution described by ioctlLR and it seems to work:
double calculateDuration(final File oggFile) throws IOException {
int rate = -1;
int length = -1;
int size = (int) oggFile.length();
byte[] t = new byte[size];
FileInputStream stream = new FileInputStream(oggFile);
stream.read(t);
for (int i = size-1-8-2-4; i>=0 && length<0; i--) { //4 bytes for "OggS", 2 unused bytes, 8 bytes for length
// Looking for length (value after last "OggS")
if (
t[i]==(byte)'O'
&& t[i+1]==(byte)'g'
&& t[i+2]==(byte)'g'
&& t[i+3]==(byte)'S'
) {
byte[] byteArray = new byte[]{t[i+6],t[i+7],t[i+8],t[i+9],t[i+10],t[i+11],t[i+12],t[i+13]};
ByteBuffer bb = ByteBuffer.wrap(byteArray);
bb.order(ByteOrder.LITTLE_ENDIAN);
length = bb.getInt(0);
}
}
for (int i = 0; i<size-8-2-4 && rate<0; i++) {
// Looking for rate (first value after "vorbis")
if (
t[i]==(byte)'v'
&& t[i+1]==(byte)'o'
&& t[i+2]==(byte)'r'
&& t[i+3]==(byte)'b'
&& t[i+4]==(byte)'i'
&& t[i+5]==(byte)'s'
) {
byte[] byteArray = new byte[]{t[i+11],t[i+12],t[i+13],t[i+14]};
ByteBuffer bb = ByteBuffer.wrap(byteArray);
bb.order(ByteOrder.LITTLE_ENDIAN);
rate = bb.getInt(0);
}
}
stream.close();
double duration = (double) (length*1000) / (double) rate;
return duration;
}
Beware, finding the rate this way will work only for vorbis OGG!
Feel free to edit my answer, it may not be perfect.
I have the following code:
const uint8 = new Uint8Array(buffer);
let uint8String = "";
for (const n of uint8) uint8String += n.toString(2).padStart(8, "0");
console.log(uint8String);
const uint32 = new Uint32Array(uint8.buffer);
let uint32String = "";
for (const n of uint32) uint32String += n.toString(2).padStart(32, "0");
console.log(uint32String);
Which logs:
10111001111000011100010000100001001111110101100001010101101010001011100001011101011000101000001000111110010110000101010110101000
00100001110001001110000110111001101010000101010101011000001111111000001001100010010111011011100010101000010101010101100000111110
Which are different binary strings. What’s the reason for the difference?
Because Javascript on an Intel machine is Little Endian.
Consider the first four bytes of your two arrays:
10111001 11100001 11000100 00100001 <-- Four unsigned bytes
00100001 11000100 11100001 10111001 <-- One unsigned int
Notice that the bytes are swapped end-to-end? That's because the byte order in the first array is in the order in which you placed the characters, but the second array stores each unsigned int as Little Endian, which means the Least Significant Byte (LSB) is stored at the smallest memory location.
Further Reading
Endianness on Wikipedia
I am trying to read a .stl file
the stl file format has the first 80 bytes are a string
then a float32 that is the number of triangle stored in the file.
then for each triangle 12 float32 values,
then a Uint16
then the 12 Float32 and 1 Uint16 pattern repeats.
I have been able to get the first triangle values, but I can't read the next array of float32 values out because the offset is not divisible by 4 anymore after getting the Uint16 value.
Is there any way to do a Left Shift operation on the arraybuffer or some way to continue to read out the values?
You could pull them out by making a copy
const src = new Uint8Array([1,2,3,4,5,0,0,246,66,0,0,144,64,14,15,16,17,18,19]);
const offset = 5;
const numFloats = 2;
const floats = new Float32Array(src.slice(offset, offset + numFloats * 4).buffer);
console.log(floats);
You could also use a DataView
const src = new Uint8Array([1,2,3,4,5,0,0,246,66,0,0,144,64,14,15,16,17,18,19]);
const offset = 5;
const dataview = new DataView(src.buffer);
const littleEndian = true;
console.log(dataview.getFloat32(offset, littleEndian));
console.log(dataview.getFloat32(offset + 4, littleEndian));