Develop Reference

Having Trouble Parsing a Base64 Encoded Array into Geometry Data [duplicate]

I need to convert a base64 encode string into an ArrayBuffer.
The base64 strings are user input, they will be copy and pasted from an email, so they're not there when the page is loaded.
I would like to do this in javascript without making an ajax call to the server if possible.
I found those links interesting, but they didt'n help me:
ArrayBuffer to base64 encoded string
this is about the opposite conversion, from ArrayBuffer to base64, not the other way round
http://jsperf.com/json-vs-base64/2
this looks good but i can't figure out how to use the code.
Is there an easy (maybe native) way to do the conversion? thanks

Try this:
function _base64ToArrayBuffer(base64) {
var binary_string = window.atob(base64);
var len = binary_string.length;
var bytes = new Uint8Array(len);
for (var i = 0; i < len; i++) {
bytes[i] = binary_string.charCodeAt(i);
}
return bytes.buffer;
}

Using TypedArray.from:
Uint8Array.from(atob(base64_string), c => c.charCodeAt(0))
Performance to be compared with the for loop version of Goran.it answer.

For Node.js users:
const myBuffer = Buffer.from(someBase64String, 'base64');
myBuffer will be of type Buffer which is a subclass of Uint8Array. Unfortunately, Uint8Array is NOT an ArrayBuffer as the OP was asking for. But when manipulating an ArrayBuffer I almost always wrap it with Uint8Array or something similar, so it should be close to what's being asked for.

Goran.it's answer does not work because of unicode problem in javascript - https://developer.mozilla.org/en-US/docs/Web/API/WindowBase64/Base64_encoding_and_decoding.
I ended up using the function given on Daniel Guerrero's blog: http://blog.danguer.com/2011/10/24/base64-binary-decoding-in-javascript/
Function is listed on github link: https://github.com/danguer/blog-examples/blob/master/js/base64-binary.js
Use these lines
var uintArray = Base64Binary.decode(base64_string);
var byteArray = Base64Binary.decodeArrayBuffer(base64_string);

Async solution, it's better when the data is big:
// base64 to buffer
function base64ToBufferAsync(base64) {
var dataUrl = "data:application/octet-binary;base64," + base64;
fetch(dataUrl)
.then(res => res.arrayBuffer())
.then(buffer => {
console.log("base64 to buffer: " + new Uint8Array(buffer));
})
}
// buffer to base64
function bufferToBase64Async( buffer ) {
var blob = new Blob([buffer], {type:'application/octet-binary'});
console.log("buffer to blob:" + blob)
var fileReader = new FileReader();
fileReader.onload = function() {
var dataUrl = fileReader.result;
console.log("blob to dataUrl: " + dataUrl);
var base64 = dataUrl.substr(dataUrl.indexOf(',')+1)
console.log("dataUrl to base64: " + base64);
};
fileReader.readAsDataURL(blob);
}

Javascript is a fine development environment so it seems odd than it doesn't provide a solution to this small problem. The solutions offered elsewhere on this page are potentially slow. Here is my solution. It employs the inbuilt functionality that decodes base64 image and sound data urls.
var req = new XMLHttpRequest;
req.open('GET', "data:application/octet;base64," + base64Data);
req.responseType = 'arraybuffer';
req.onload = function fileLoaded(e)
{
var byteArray = new Uint8Array(e.target.response);
// var shortArray = new Int16Array(e.target.response);
// var unsignedShortArray = new Int16Array(e.target.response);
// etc.
}
req.send();
The send request fails if the base 64 string is badly formed.
The mime type (application/octet) is probably unnecessary.
Tested in chrome. Should work in other browsers.

Pure JS - no string middlestep (no atob)
I write following function which convert base64 in direct way (without conversion to string at the middlestep). IDEA
get 4 base64 characters chunk
find index of each character in base64 alphabet
convert index to 6-bit number (binary string)
join four 6 bit numbers which gives 24-bit numer (stored as binary string)
split 24-bit string to three 8-bit and covert each to number and store them in output array
corner case: if input base64 string ends with one/two = char, remove one/two numbers from output array
Below solution allows to process large input base64 strings. Similar function for convert bytes to base64 without btoa is HERE
function base64ToBytesArr(str) {
const abc = [..."ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"]; // base64 alphabet
let result = [];
for(let i=0; i<str.length/4; i++) {
let chunk = [...str.slice(4*i,4*i+4)]
let bin = chunk.map(x=> abc.indexOf(x).toString(2).padStart(6,0)).join('');
let bytes = bin.match(/.{1,8}/g).map(x=> +('0b'+x));
result.push(...bytes.slice(0,3 - (str[4*i+2]=="=") - (str[4*i+3]=="=")));
}
return result;
}
// --------
// TEST
// --------
let test = "Alice's Adventure in Wonderland.";
console.log('test string:', test.length, test);
let b64_btoa = btoa(test);
console.log('encoded string:', b64_btoa);
let decodedBytes = base64ToBytesArr(b64_btoa); // decode base64 to array of bytes
console.log('decoded bytes:', JSON.stringify(decodedBytes));
let decodedTest = decodedBytes.map(b => String.fromCharCode(b) ).join``;
console.log('Uint8Array', JSON.stringify(new Uint8Array(decodedBytes)));
console.log('decoded string:', decodedTest.length, decodedTest);
Caution!
If you want to decode base64 to STRING (not bytes array) and you know that result contains utf8 characters then atob will fail in general e.g. for character 💩 the atob("8J+SqQ==") will give wrong result . In this case you can use above solution and convert result bytes array to string in proper way e.g. :
function base64ToBytesArr(str) {
const abc = [..."ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"]; // base64 alphabet
let result = [];
for(let i=0; i<str.length/4; i++) {
let chunk = [...str.slice(4*i,4*i+4)]
let bin = chunk.map(x=> abc.indexOf(x).toString(2).padStart(6,0)).join('');
let bytes = bin.match(/.{1,8}/g).map(x=> +('0b'+x));
result.push(...bytes.slice(0,3 - (str[4*i+2]=="=") - (str[4*i+3]=="=")));
}
return result;
}
// --------
// TEST
// --------
let testB64 = "8J+SqQ=="; // for string: "💩";
console.log('input base64 :', testB64);
let decodedBytes = base64ToBytesArr(testB64); // decode base64 to array of bytes
console.log('decoded bytes :', JSON.stringify(decodedBytes));
let result = new TextDecoder("utf-8").decode(new Uint8Array(decodedBytes));
console.log('properly decoded string :', result);
let result_atob = atob(testB64);
console.log('decoded by atob :', result_atob);
Snippets tested 2022-08-04 on: chrome 103.0.5060.134 (arm64), safari 15.2, firefox 103.0.1 (64 bit), edge 103.0.1264.77 (arm64), and node-js v12.16.1

I would strongly suggest using an npm package implementing correctly the base64 specification.
The best one I know is rfc4648
The problem is that btoa and atob use binary strings instead of Uint8Array and trying to convert to and from it is cumbersome. Also there is a lot of bad packages in npm for that. I lose a lot of time before finding that one.
The creators of that specific package did a simple thing: they took the specification of Base64 (which is here by the way) and implemented it correctly from the beginning to the end. (Including other formats in the specification that are also useful like Base64-url, Base32, etc ...) That doesn't seem a lot but apparently that was too much to ask to the bunch of other libraries.
So yeah, I know I'm doing a bit of proselytism but if you want to avoid losing your time too just use rfc4648.

I used the accepted answer to this question to create base64Url string <-> arrayBuffer conversions in the realm of base64Url data transmitted via ASCII-cookie [atob, btoa are base64[with +/]<->js binary string], so I decided to post the code.
Many of us may want both conversions and client-server communication may use the base64Url version (though a cookie may contain +/ as well as -_ characters if I understand well, only ",;\ characters and some wicked characters from the 128 ASCII are disallowed). But a url cannot contain / character, hence the wider use of b64 url version which of course not what atob-btoa supports...
Seeing other comments, I would like to stress that my use case here is base64Url data transmission via url/cookie and trying to use this crypto data with the js crypto api (2017) hence the need for ArrayBuffer representation and b64u <-> arrBuff conversions... if array buffers represent other than base64 (part of ascii) this conversion wont work since atob, btoa is limited to ascii(128). Check out an appropriate converter like below:
The buff -> b64u version is from a tweet from Mathias Bynens, thanks for that one (too)! He also wrote a base64 encoder/decoder:
https://github.com/mathiasbynens/base64
Coming from java, it may help when trying to understand the code that java byte[] is practically js Int8Array (signed int) but we use here the unsigned version Uint8Array since js conversions work with them. They are both 256bit, so we call it byte[] in js now...
The code is from a module class, that is why static.
//utility
/**
* Array buffer to base64Url string
* - arrBuff->byte[]->biStr->b64->b64u
* #param arrayBuffer
* #returns {string}
* #private
*/
static _arrayBufferToBase64Url(arrayBuffer) {
console.log('base64Url from array buffer:', arrayBuffer);
let base64Url = window.btoa(String.fromCodePoint(...new Uint8Array(arrayBuffer)));
base64Url = base64Url.replaceAll('+', '-');
base64Url = base64Url.replaceAll('/', '_');
console.log('base64Url:', base64Url);
return base64Url;
}
/**
* Base64Url string to array buffer
* - b64u->b64->biStr->byte[]->arrBuff
* #param base64Url
* #returns {ArrayBufferLike}
* #private
*/
static _base64UrlToArrayBuffer(base64Url) {
console.log('array buffer from base64Url:', base64Url);
let base64 = base64Url.replaceAll('-', '+');
base64 = base64.replaceAll('_', '/');
const binaryString = window.atob(base64);
const length = binaryString.length;
const bytes = new Uint8Array(length);
for (let i = 0; i < length; i++) {
bytes[i] = binaryString.charCodeAt(i);
}
console.log('array buffer:', bytes.buffer);
return bytes.buffer;
}

made a ArrayBuffer from a base64:
function base64ToArrayBuffer(base64) {
var binary_string = window.atob(base64);
var len = binary_string.length;
var bytes = new Uint8Array(len);
for (var i = 0; i < len; i++) {
bytes[i] = binary_string.charCodeAt(i);
}
return bytes.buffer;
}
I was trying to use above code and It's working fine.

The result of atob is a string that is separated with some comma
,
A simpler way is to convert this string to a json array string and after that parse it to a byteArray
below code can simply be used to convert base64 to an array of number
let byteArray = JSON.parse('['+atob(base64)+']');
let buffer = new Uint8Array(byteArray);

Solution without atob
I've seen many people complaining about using atob and btoa in the replies. There are some issues to take into account when using them.
There's a solution without using them in the MDN page about Base64. Below you can find the code to convert a base64 string into a Uint8Array copied from the docs.
Note that the function below returns a Uint8Array. To get the ArrayBuffer version you just need to do uintArray.buffer.
function b64ToUint6(nChr) {
return nChr > 64 && nChr < 91
? nChr - 65
: nChr > 96 && nChr < 123
? nChr - 71
: nChr > 47 && nChr < 58
? nChr + 4
: nChr === 43
? 62
: nChr === 47
? 63
: 0;
}
function base64DecToArr(sBase64, nBlocksSize) {
const sB64Enc = sBase64.replace(/[^A-Za-z0-9+/]/g, "");
const nInLen = sB64Enc.length;
const nOutLen = nBlocksSize
? Math.ceil(((nInLen * 3 + 1) >> 2) / nBlocksSize) * nBlocksSize
: (nInLen * 3 + 1) >> 2;
const taBytes = new Uint8Array(nOutLen);
let nMod3;
let nMod4;
let nUint24 = 0;
let nOutIdx = 0;
for (let nInIdx = 0; nInIdx < nInLen; nInIdx++) {
nMod4 = nInIdx & 3;
nUint24 |= b64ToUint6(sB64Enc.charCodeAt(nInIdx)) << (6 * (3 - nMod4));
if (nMod4 === 3 || nInLen - nInIdx === 1) {
nMod3 = 0;
while (nMod3 < 3 && nOutIdx < nOutLen) {
taBytes[nOutIdx] = (nUint24 >>> ((16 >>> nMod3) & 24)) & 255;
nMod3++;
nOutIdx++;
}
nUint24 = 0;
}
}
return taBytes;
}
If you're interested in the reverse operation, ArrayBuffer to base64, you can find how to do it in the same link.

How do I get the actual size in bytes for a number and a string in JavaScript in a browser environment?

I am trying to get the actual size (in bytes) of a number and a string in browsers e.g. chrome.
I learned that in JavaScript numbers are represented in double precision takes up 64 bits and strings are UTF-16 code unit so it takes either 2 bytes or 4 bytes.
I first tried to use new Blob but it encodes string component characters as UTF-8 not UTF-16. And I know there is a Buffer.from API in Node but it is not available in a browser environment.
My question is how I can get the actual size of a number and a string in bytes from a browser, e.g. chrome?

You can do that natively with the help of TextEncoder
let str1 = 'Beta'; // 'Beta' text in English
let str2 = '贝塔'; // 'Beta' text in Chinese
const encoder = new TextEncoder();
const len1 = encoder.encode(str1).length;
const len2 = encoder.encode(str2).length;
console.log(len1); // 4
console.log(len2); // 6

First of all it is important to realize that the spec doesn't mandate any representation. Just behavior.
Strings are stored in UTF-16 but fortunately for your purpose each index represents 16 bits.
For example
console.log('😠'.length); // Should log 2 because emoji takes 2 16 bit parts
For numbers it depends. V8 represents small integer numbers as actual 32 bit ints.

With https://github.com/substack/node-browserify you can work with buffers in the Browser by using: https://github.com/toots/buffer-browserify.
//in your browsify js file:
require('buffer')
Buffer.byteLength(String.fromCharCode(55555), 'utf16')
Buffer.byteLength(String.fromCharCode(55555, 57000), 'utf16')

iconv-lite: Pure JS character encoding conversion
var iconv = require('iconv-lite');
var buf =iconv.encode("Hello World", 'utf16');
console.log(buf);
console.log(buf.length); // returns 24

Here is my answer to your problem :
function getBytesFromVar(theVar) {
if(theVar !== null && theVar !== undefined) {
switch (typeof theVar) {
case 'string' : {
var encoder = new TextEncoder();
encoder['encoding'] = "utf-16";
return encoder['encode'](theVar).length * 2;
}
case 'number' : {
return 8;
}
case 'boolean' : {
return 4;
}
case 'object' : {
if ( theVar instanceof String) {
var encoder = new TextEncoder();
encoder['encoding'] = "utf-16";
return encoder['encode'](theVar.toString()).length * 2;
} else {
return 0;
}
}
}
}
else {
return 0;
}
}
The getBytesFromVar function take a var and return the number of byte used.
Function use TextEncoder to get the string length and then calculate the bytes.
In case of a string created with:
let str = new String('Alain♥');
function will work with String objects.
ATTENTION: this can't be used to calculate memory footprint in browser as other mechanism of memory management can increase/decrease this values.
Vars can be allocated in different memory segment. For example, String object are created on the heap and string vars are created on string constant pool.
Also vars are manipulated through pointer.
For example, 2 strings vars that contain the same string are created on the string constant pool. First will be allocated, but the second one will be a pointer to the first one. So the size in memory byte will not be simply twice the size of the string.
Good post about that: https://levelup.gitconnected.com/bytefish-vs-new-string-bytefish-what-is-the-difference-a795f6a7a08b
Use case:
var myString='Alain♥';
var myNumber = 120;
var objString = new String('Alain♥');
var myFloat = 105.456;
console.log('%o is %o bytes', myString, getBytesFromVar(myString));
console.log('%o is %o bytes', myNumber, getBytesFromVar(myNumber));
console.log('%o is %o bytes', objString, getBytesFromVar(objString));
console.log('%o is %o bytes', myFloat, getBytesFromVar(myFloat));

I have used the npm module object-sizeof for this. You can use it to get the size of integer or string variables in bytes. This is a sample usage,
var sizeof = require('object-sizeof');
console.log(sizeof(123)); //prints 8

You can do that natively with the help of TextEncoder
let str1 = 'Beta'; // 'Beta' text in English
let str2 = '贝塔'; // 'Beta' text in Chinese
const encoder = new TextEncoder();
const len1 = encoder.encode(str1).length;
const len2 = encoder.encode(str2).length;
console.log(len1); // 4
console.log(len2); // 6

64bit Hex to Decimal in Javascript

Need to convert 64bit hex to decimal in node, preferably without 3rd party lib.
Input:
Hex: 0x3fe2da2f8bdec5f4
Hex: 0x402A000000000000
Output
Dec: .589134
Dec: 13

You can do this very easily in node.js without any libraries by using Buffer:
const hex = '3fe2da2f8bdec5f4';
const result = Buffer.from( hex, 'hex' ).readDoubleBE( 0 );
console.log( result );
WARNING: The offset of 0 is not optional. Several versions of the node.js API docs show examples of not supplying an offset for most Buffer functions and it being treated as an offset of 0, but due to a bug in node.js versions 9.4.0, 9.5.0, 9.6.0, 9.6.1, and 9.7 you will get slightly incorrect results (EG. 13.000001912238076 instead of exactly 13) if you do not specify an offset to readDoubleBE in those versions.

For those trying to do this in client side javscript
// Split the array by bytes
a = "3fe2da2f8bdec5f4"
b = a.match(/.{2}/g);
// Create byte array
let buffer = new ArrayBuffer(8)
let bytes = new Uint8Array(buffer)
// Populate array
for(let i = 0; i < 8; ++i) {
bytes[i] = parseInt(b[i], 16);
}
// Convert and print
let view = new DataView(buffer)
console.log(view.getFloat64(0, false));

Why Buffer copy prints me some garbage value?

I am new to node.js,
I was reading on Node.js Buffers.
I stumbled on buffer.copy method.
Wrote a code which is as follows,
var bufferOne = new Buffer("This is China");
var bufferTwo = new Buffer(20),
startTarget = 0,
sourceStart = 8,
sourceEnd = 0;
var bufferLength = bufferOne.length;
sourceEnd = bufferLength;
console.log("Buffer length "+bufferLength);
bufferOne.copy(bufferTwo,startTarget,sourceStart,sourceEnd);
console.log("Buffer1 "+bufferOne.toString());
console.log("Buffer2 "+bufferTwo.toString());
The output is as follows,
Buffer length 13
Buffer1 This is China
Buffer2 China
But sometimes it also prints garbage output after "China", as follows,
Buffer length 13
Buffer1 This is China
Buffer2 China���*�
Sometimes,
Buffer length 13
Buffer1 This is China
Buffer2 China���
Can somebody please tell me whats going on ?

The length of bufferTwo is 20, but you're only copying 13 bytes. The remaining 7 bytes contain semi-random data.
Documented here:
Passing a number as the first argument to Buffer() (e.g. new Buffer(10)), allocates a new Buffer object of the specified size. The memory allocated for such Buffer instances is not initialized and can contain sensitive data.

Join up PNG images to an APNG animated image

Is it possible somehow to join up PNG images to an APNG animated image using nodejs?
I've found PHP library only: link

UPNG.js can parse and build APNG files - https://github.com/photopea/UPNG.js
From the readme -
UPNG.js supports APNG and the interface expects "frames".
UPNG.encode(imgs, w, h, cnum, [dels])
imgs: array of frames. A frame is an ArrayBuffer containing the pixel
data (RGBA, 8 bits per channel)
w, h : width and height of the image
cnum: number of colors in the result; 0: all colors (lossless PNG)
dels: array of delays for each frame (only when 2 or more frames)
returns an ArrayBuffer with binary data of a PNG file
UPNG.js can do a lossy minification of PNG files, similar to TinyPNG
and other tools. It performs color quantization using the k-means
algorithm.
Lossy compression is allowed by the last parameter cnum. Set it to
zero for a lossless compression, or write the number of allowed colors
in the image. Smaller values produce smaller files. Or just use 0 for
lossless / 256 for lossy.

There is no library for that, but it is quite simple to implement. Algorithm for merging multiple PNG files into single APNG is described in Wikipedia:
Take all chunks of the first PNG file as a building basis.
Insert an animation control chunk (acTL) after the image header chunk (IHDR).
If the first PNG is to be part of the animation, insert a frame control chunk (fcTL) before the image data chunk (IDAT).
For each of the remaining frames, add a frame control chunk (fcTL) and a frame data chunk (fdAT). Then add the image end chunk (IEND). The content for the frame data chunks (fdAT) is taken from the image data chunks (IDAT) of their respective source images.
Here is an example implementation:
const fs = require('fs')
const crc32 = require('crc').crc32
function findChunk(buffer, type) {
let offset = 8
while (offset < buffer.length) {
let chunkLength = buffer.readUInt32BE(offset)
let chunkType = buffer.slice(offset + 4, offset + 8).toString('ascii')
if (chunkType === type) {
return buffer.slice(offset, offset + chunkLength + 12)
}
offset += 4 + 4 + chunkLength + 4
}
throw new Error(`Chunk "${type}" not found`)
}
const images = process.argv.slice(2).map(path => fs.readFileSync(path))
const actl = Buffer.alloc(20)
actl.writeUInt32BE(8, 0) // length of chunk
actl.write('acTL', 4) // type of chunk
actl.writeUInt32BE(images.length, 8) // number of frames
actl.writeUInt32BE(0, 12) // number of times to loop (0 - infinite)
actl.writeUInt32BE(crc32(actl.slice(4, 16)), 16) // crc
const frames = images.map((data, idx) => {
const ihdr = findChunk(data, 'IHDR')
const fctl = Buffer.alloc(38)
fctl.writeUInt32BE(26, 0) // length of chunk
fctl.write('fcTL', 4) // type of chunk
fctl.writeUInt32BE(idx ? idx * 2 - 1 : 0, 8) // sequence number
fctl.writeUInt32BE(ihdr.readUInt32BE(8), 12) // width
fctl.writeUInt32BE(ihdr.readUInt32BE(12), 16) // height
fctl.writeUInt32BE(0, 20) // x offset
fctl.writeUInt32BE(0, 24) // y offset
fctl.writeUInt16BE(1, 28) // frame delay - fraction numerator
fctl.writeUInt16BE(1, 30) // frame delay - fraction denominator
fctl.writeUInt8(0, 32) // dispose mode
fctl.writeUInt8(0, 33) // blend mode
fctl.writeUInt32BE(crc32(fctl.slice(4, 34)), 34) // crc
const idat = findChunk(data, 'IDAT')
// All IDAT chunks except first one are converted to fdAT chunks
let fdat;
if (idx === 0) {
fdat = idat
} else {
const length = idat.length + 4
fdat = Buffer.alloc(length)
fdat.writeUInt32BE(length - 12, 0) // length of chunk
fdat.write('fdAT', 4) // type of chunk
fdat.writeUInt32BE(idx * 2, 8) // sequence number
idat.copy(fdat, 12, 8) // image data
fdat.writeUInt32BE(crc32(4, length - 4), length - 4) // crc
}
return Buffer.concat([ fctl, fdat ])
})
const signature = Buffer.from('\211PNG\r\n\032\n', 'ascii')
const ihdr = findChunk(images[0], 'IHDR')
const iend = Buffer.from('0000000049454e44ae426082', 'hex')
const output = Buffer.concat([ signature, ihdr, actl, ...frames, iend ])
fs.writeFileSync('output.png', output)

Currently, no it doesn't look like it. Wikipedia lists the available software, and as you can see there's no support for ImageMagick which has a Node wrapper. However, you may find you can download the command line tool apngasm and shell out to it, if you find it's worth your while there are Node command line wrappers to hook this into an existing application using child_process (http://nodejs.org/api/child_process.html).

I'm not sure about nodejs, but you could try APNG-canvas. APNG uses HTML5 (-webkit-canvas), JavaScript (jQuery).
"APNG-canvas is a library for displaing Animated PNG files in the browsers with canvas support (Google Chrome, Internet Explorer 9, Apple Safari)."
Working demo is here.