I'm using a barcode scanner to read a barcode on my website (the website is made in OpenUI5).
The scanner works like a keyboard that types the characters it reads. At the end and the beginning of the typing it uses a special character. These characters are different for every type of scanner.
Some possible characters are:
█
▄
–
—
In my code I use if (oModelScanner.oData.scanning && oEvent.key == "\u2584") to check if the input from the scanner is ▄.
Is there any way to get the code from that character in the \uHHHH style? (with the HHHH being the hexadecimal code for the character)
I tried the charCodeAt but this returns the decimal code.
With the codePointAt examples they make the code I need into a decimal code so I need a reverse of this.
Javascript strings have a method codePointAt which gives you the integer representing the Unicode point value. You need to use a base 16 (hexadecimal) representation of that number if you wish to format the integer into a four hexadecimal digits sequence (as in the response of Nikolay Spasov).
var hex = "▄".codePointAt(0).toString(16);
var result = "\\u" + "0000".substring(0, 4 - hex.length) + hex;
However it would probably be easier for you to check directly if you key code point integer match the expected code point
oEvent.key.codePointAt(0) === '▄'.codePointAt(0);
Note that "symbol equality" can actually be trickier: some symbols are defined by surrogate pairs (you can see it as the combination of two halves defined as four hexadecimal digits sequence).
For this reason I would recommend to use a specialized library.
you'll find more details in the very relevant article by Mathias Bynens
var hex = "▄".charCodeAt(0).toString(16);
var result = "\\u" + "0000".substring(0, 4 - hex.length) + hex;
If you want to print the multiple code points of a character, e.g., an emoji, you can do this:
const facepalm = "🤦🏼♂️";
const codePoints = Array.from(facepalm)
.map((v) => v.codePointAt(0).toString(16))
.map((hex) => "\\u{" + hex + "}");
console.log(codePoints);
["\u{1f926}", "\u{1f3fc}", "\u{200d}", "\u{2642}", "\u{fe0f}"]
If you are wondering about the components and the length of 🤦🏼♂️, check out this article.
Background
I am writing an esoteric language called Jolf. It is used on the lovely site codegolf SE. If you don't already know, a lot of challenges are scored in bytes. People have made lots of languages that utilize either their own encoding or a pre-existing encoding.
On the interpreter for my language, I have a byte counter. As you might expect, it counts the number of bytes in the code. Until now, I've been using a UTF-8 en/decoder (utf8.js). I am now using the ISO 8859-7 encoding, which has Greek characters. Nor does the text upload actually work. I need to count the actually bytes contained within an uploaded file. Also, is there a way to read the contents of said encoded file?
Question
Given a file encoded in ISO 8859-7 obtained from an <input> element on the page, is there any way to obtain the number of bytes contained in that file? And, given "plaintext" (i.e. text put directly into a <textarea>), how might I count the bytes in that as if it was encoded in ISO 8859-7?
What I've tried
The input element is called isogreek. The file resides in the <input> element. The content is ΦX族, a Greek character, a latin character (each of which should be a byte) and a Chinese character, which should be more than one byte (?).
isogreek.files[0].size; // is 3; should be more.
var reader = new FileReader();
reader.readAsBinaryString(isogreek.files[0]); // corrupts the string to `ÖX?`
reader.readAsText(isogreek.files[0]); // �X?
reader.readAsText(isogreek.files[0],"ISO 8859-7"); // �X?
Extended from this comment.
As #pvg mentioned in the comments, the string resulting from readAsBinaryString would be correct, but is corrupted for two reasons:
A. The result is encoded in ISO-8859-1. You can use a function to fix this:
function convertFrom1to7(text) {
// charset is the set of chars in the ISO-8859-7 encoding from 0xA0 and up, encoded with this format:
// - If the character is in the same position as in ISO-8859-1/Unicode, use a "!".
// - If the character is a Greek char with 720 subtracted from its char code, use a ".".
// - Otherwise, use \uXXXX format.
var charset = "!\u2018\u2019!\u20AC\u20AF!!!!.!!!!\u2015!!!!...!...!.!....................!............................................!";
var newtext = "", newchar = "";
for (var i = 0; i < text.length; i++) {
var char = text[i];
newchar = char;
if (char.charCodeAt(0) >= 160) {
newchar = charset[char.charCodeAt(0) - 160];
if (newchar === "!") newchar = char;
if (newchar === ".") newchar = String.fromCharCode(char.charCodeAt(0) + 720);
}
newtext += newchar;
}
return newtext;
}
B. The Chinese character isn't a part of the ISO-8859-7 charset (because the charset supports up to 256 unique chars, as the table shows). If you want to include arbitrary Unicode characters in a program, you will probably need to do one of these two things:
Count the bytes of that program in i.e. UTF-8 or UTF-16. This can be done pretty easily with the library you linked. However, if you want this to be done automatically, you'll need a function that checks if the content of the textarea is a valid ISO-8859-7 file, like this:
function isValidISO_8859_7(text) {
var charset = /[\u0000-\u00A0\u2018\u2019\u00A3\u20AC\u20AF\u00A6-\u00A9\u037A\u00AB-\u00AD\u2015\u00B0-\u00B3\u0384-\u0386\u00B7\u0388-\u038A\u00BB\u038C\u00BD\u038E-\u03CE]/;
var valid = true;
for (var i = 0; i < text.length; i++) {
valid = valid && charset.test(text[i]);
}
return valid;
}
Create your own, custom variant of ISO-8859-7 that uses a specific byte (or more than one) to signify that the next 2 or 3 bytes belong to a single Unicode char. This can be pretty much as simple or complex as you like, from one char signifying a 2-byte char and one signifying a 3-byter to everything between 80 and 9F setting up for the next few. Here's a basic example that uses 80 as the 2-byter and 81 as the 3-byter (assumes the text is encoded in ISO-8859-1):
function reUnicode(text) {
var newtext = "";
for (var i = 0; i < text.length; i++) {
if (text.charCodeAt(i) === 0x80) {
newtext += String.fromCharCode((text.charCodeAt(++i) << 8) + text.charCodeAt(++i));
} else if (text.charCodeAt(i) === 0x81) {
var charcode = (text.charCodeAt(++i) << 16) + (text.charCodeAt(++i) << 8) + text.charCodeAt(++i) - 65536;
newtext += String.fromCharCode(0xD800 + (charcode >> 10), 0xDC00 + (charcode & 1023)); // Convert into a UTF-16 surrogate pair
} else {
newtext += convertFrom1to7(text[i]);
}
}
return newtext;
}
I can go into either method in more detail if you desire.
The three characters you gave as an example are decoded in 6 bytes a6 ce e6 58 8f 97 (0x58 = X). Also: JavaScript works with utf16 which results in some funny things like ("abc".length === "ΦX族".length) being true.
You most probably need to go to the full length and check every single character for its length by its code-value. You may also need to check two characters in some cases (utf-32 to utf-16). A BOM needs to be placed and checked, too, if necessary (always necessary if you work with files of unknown sources).
EDIT: added on request:
The encodings of the characters in JavaScript is always in utf-16, a two byte representation of the character. That was all well and nice until they suddenly (ha!) found out that two bytes are not really sufficient for all of the alphabets of the world, so the expanded the Unicode range to four bytes: utf-32.
Well, the Unicode consortium did so but the ECMA committee did not.
It cannot be said that hell broke loose but it is quite close in some circumstances, and one of those is your case because you want to mix one-byte encodings with multiple-byte encodings, different ones even.
One byte fits well in two bytes but three or more bytes do not fit well in two bytes, so the so called surrogates were invented. These surrogates are also the reason why it is not so simple to reverse a string in JavaScript.
As I said: a large can of worms.
I'm new to Javascript, and I'm writing a Chrome extension which manipulates the Chrome Omnibox.
I have the following code implemented:
chrome.omnibox.onInputEntered.addListener(
function(text) {
console.log('inputEntered: ' + text);
if (text < 07000000){
chrome.tabs.create({url:"PRIVATEURL1"+ text});
}
if (text > 07000000){
chrome.tabs.create({url:"PRIVATEURL2"+ text});
}
});
Currently, it behaves like this:
Enter 07000001 & be brought to PRIVATEURL2.
Enter 00600000 & be brought to PRIVATEURL1.
Enter 1 & be brought to PRIVATEURL1.
All as expected.
However, enter:
04542226 & be brought to PRIVATEURL2.
06000001 & be brought to PRIVATEURL2.
I don't understand, is 04542226 not < 07000000 and therefore I should be brought to PRIVATEURL1?
The answer is rather easy.
Your text is, as evident, a string. Say, "07000001". When put in an arithmetical comparison, it's cast to a number. Which number? Let's see:
> text
"07000001"
> Number(text)
7000001
Now, that's correct. What is not is your number literal, 07000000
It is more or less known that you can write a hexadecimal number as a literal using the 0x notation:
> 0xFF
255
What is less known is that 0 prefix is a literal notation for octal numbers. 7000000 in base 8 is the number 1835008:
> 07000000
1835008
So, you should use 7000000 literal instead:
> "04542226" < 07000000
false
> "04542226" < 7000000
true
Curiously, parseInt/Number will process hexadecimal notation, but as we've seen it ignores leading zeroes. This is set by ECMAScript 5 standard.
Note that you can (and should) pass a second argument, radix, to the function if you expect a certain base.
What I want is to calculate how much time the caret will move from the beginning till the end of the string.
Explanations:
Look this string "" in this fiddle: http://jsfiddle.net/RFuQ3/
If you put the caret before the first quote then push the right arrow ► you will push 3 times to arrive after the second quote (instead of 2 times for an empty string).
The first way, and the easiest to calculate the length of a string is <string>.length.
But here, it returns 2.
The second way, from JavaScript Get real length of a string (without entities) gives 2 too.
How can I get 1?
1-I thought to a way to put the string in a text input, and then do a while loop with a try{setCaret}catch(){}
2-It's just for fun
The character in your question "" is the
Unicode Character 'LANGUAGE TAG' (U+E0001).
From the following Stack Overflow questions,
" Expressing UTF-16 unicode characters in JavaScript"
" How can I tell if a string contains multibyte characters in Javascript?"
we learn that
JavaScript strings are UCS-2 encoded but can represent Unicode code points outside the Basic Multilingual Pane (U+0000-U+D7FF and U+E000-U+FFFF) using two 16 bit numbers (a UTF-16 surrogate pair), the first of which must be in the range U+D800-U+DFFF.
The UTF-16 surrogate pair representing "" is U+DB40 and U+DC01. In decimal U+DB40 is 56128, and U+DC01 is 56321.
console.log("".length); // 2
console.log("".charCodeAt(0)); // 56128
console.log("".charCodeAt(1)); // 56321
console.log("\uDB40\uDC01" === ""); // true
console.log(String.fromCharCode(0xDB40, 0xDC01) === ""); // true
Adapting the code from https://stackoverflow.com/a/4885062/788324, we just need to count the number of code points to arrive at the correct answer:
var getNumCodePoints = function(str) {
var numCodePoints = 0;
for (var i = 0; i < str.length; i++) {
var charCode = str.charCodeAt(i);
if ((charCode & 0xF800) == 0xD800) {
i++;
}
numCodePoints++;
}
return numCodePoints;
};
console.log(getNumCodePoints("")); // 1
jsFiddle Demo
function realLength(str) {
var i = 1;
while (str.substring(i,i+1) != "") i++;
return (i-1);
}
Didn't try the code, but it should work I think.
Javascript doesn't really support unicode.
You can try
yourstring.replace(/[\uD800-\uDFFF]{2}/g, "0").length
for what it's worth
I have a javascript string which is about 500K when being sent from the server in UTF-8. How can I tell its size in JavaScript?
I know that JavaScript uses UCS-2, so does that mean 2 bytes per character. However, does it depend on the JavaScript implementation? Or on the page encoding or maybe content-type?
You can use the Blob to get the string size in bytes.
Examples:
console.info(
new Blob(['😂']).size, // 4
new Blob(['👍']).size, // 4
new Blob(['😂👍']).size, // 8
new Blob(['👍😂']).size, // 8
new Blob(['I\'m a string']).size, // 12
// from Premasagar correction of Lauri's answer for
// strings containing lone characters in the surrogate pair range:
// https://stackoverflow.com/a/39488643/6225838
new Blob([String.fromCharCode(55555)]).size, // 3
new Blob([String.fromCharCode(55555, 57000)]).size // 4 (not 6)
);
This function will return the byte size of any UTF-8 string you pass to it.
function byteCount(s) {
return encodeURI(s).split(/%..|./).length - 1;
}
Source
JavaScript engines are free to use UCS-2 or UTF-16 internally. Most engines that I know of use UTF-16, but whatever choice they made, it’s just an implementation detail that won’t affect the language’s characteristics.
The ECMAScript/JavaScript language itself, however, exposes characters according to UCS-2, not UTF-16.
Source
If you're using node.js, there is a simpler solution using buffers :
function getBinarySize(string) {
return Buffer.byteLength(string, 'utf8');
}
There is a npm lib for that : https://www.npmjs.org/package/utf8-binary-cutter (from yours faithfully)
String values are not implementation dependent, according the ECMA-262 3rd Edition Specification, each character represents a single 16-bit unit of UTF-16 text:
4.3.16 String Value
A string value is a member of the type String and is a
finite ordered sequence of zero or
more 16-bit unsigned integer values.
NOTE Although each value usually
represents a single 16-bit unit of
UTF-16 text, the language does not
place any restrictions or requirements
on the values except that they be
16-bit unsigned integers.
These are 3 ways I use:
TextEncoder
new TextEncoder().encode("myString").length
Blob
new Blob(["myString"]).size
Buffer
Buffer.byteLength("myString", 'utf8')
Try this combination with using unescape js function:
const byteAmount = unescape(encodeURIComponent(yourString)).length
Full encode proccess example:
const s = "1 a ф № # ®"; // length is 11
const s2 = encodeURIComponent(s); // length is 41
const s3 = unescape(s2); // length is 15 [1-1,a-1,ф-2,№-3,#-1,®-2]
const s4 = escape(s3); // length is 39
const s5 = decodeURIComponent(s4); // length is 11
Note that if you're targeting node.js you can use Buffer.from(string).length:
var str = "\u2620"; // => "☠"
str.length; // => 1 (character)
Buffer.from(str).length // => 3 (bytes)
The size of a JavaScript string is
Pre-ES6: 2 bytes per character
ES6 and later: 2 bytes per character,
or 5 or more bytes per character
Pre-ES6
Always 2 bytes per character. UTF-16 is not allowed because the spec says "values must be 16-bit unsigned integers". Since UTF-16 strings can use 3 or 4 byte characters, it would violate 2 byte requirement. Crucially, while UTF-16 cannot be fully supported, the standard does require that the two byte characters used are valid UTF-16 characters. In other words, Pre-ES6 JavaScript strings support a subset of UTF-16 characters.
ES6 and later
2 bytes per character, or 5 or more bytes per character. The additional sizes come into play because ES6 (ECMAScript 6) adds support for Unicode code point escapes. Using a unicode escape looks like this: \u{1D306}
Practical notes
This doesn't relate to the internal implemention of a particular engine. For
example, some engines use data structures and libraries with full
UTF-16 support, but what they provide externally doesn't have to be
full UTF-16 support. Also an engine may provide external UTF-16
support as well but is not mandated to do so.
For ES6, practically speaking characters will never be more than 5
bytes long (2 bytes for the escape point + 3 bytes for the Unicode
code point) because the latest version of Unicode only has 136,755
possible characters, which fits easily into 3 bytes. However this is
technically not limited by the standard so in principal a single
character could use say, 4 bytes for the code point and 6 bytes
total.
Most of the code examples here for calculating byte size don't seem to take into account ES6 Unicode code point escapes, so the results could be incorrect in some cases.
UTF-8 encodes characters using 1 to 4 bytes per code point. As CMS pointed out in the accepted answer, JavaScript will store each character internally using 16 bits (2 bytes).
If you parse each character in the string via a loop and count the number of bytes used per code point, and then multiply the total count by 2, you should have JavaScript's memory usage in bytes for that UTF-8 encoded string. Perhaps something like this:
getStringMemorySize = function( _string ) {
"use strict";
var codePoint
, accum = 0
;
for( var stringIndex = 0, endOfString = _string.length; stringIndex < endOfString; stringIndex++ ) {
codePoint = _string.charCodeAt( stringIndex );
if( codePoint < 0x100 ) {
accum += 1;
continue;
}
if( codePoint < 0x10000 ) {
accum += 2;
continue;
}
if( codePoint < 0x1000000 ) {
accum += 3;
} else {
accum += 4;
}
}
return accum * 2;
}
Examples:
getStringMemorySize( 'I' ); // 2
getStringMemorySize( '❤' ); // 4
getStringMemorySize( '𠀰' ); // 8
getStringMemorySize( 'I❤𠀰' ); // 14
The answer from Lauri Oherd works well for most strings seen in the wild, but will fail if the string contains lone characters in the surrogate pair range, 0xD800 to 0xDFFF. E.g.
byteCount(String.fromCharCode(55555))
// URIError: URI malformed
This longer function should handle all strings:
function bytes (str) {
var bytes=0, len=str.length, codePoint, next, i;
for (i=0; i < len; i++) {
codePoint = str.charCodeAt(i);
// Lone surrogates cannot be passed to encodeURI
if (codePoint >= 0xD800 && codePoint < 0xE000) {
if (codePoint < 0xDC00 && i + 1 < len) {
next = str.charCodeAt(i + 1);
if (next >= 0xDC00 && next < 0xE000) {
bytes += 4;
i++;
continue;
}
}
}
bytes += (codePoint < 0x80 ? 1 : (codePoint < 0x800 ? 2 : 3));
}
return bytes;
}
E.g.
bytes(String.fromCharCode(55555))
// 3
It will correctly calculate the size for strings containing surrogate pairs:
bytes(String.fromCharCode(55555, 57000))
// 4 (not 6)
The results can be compared with Node's built-in function Buffer.byteLength:
Buffer.byteLength(String.fromCharCode(55555), 'utf8')
// 3
Buffer.byteLength(String.fromCharCode(55555, 57000), 'utf8')
// 4 (not 6)
A single element in a JavaScript String is considered to be a single UTF-16 code unit. That is to say, Strings characters are stored in 16-bit (1 code unit), and 16-bit is equal to 2 bytes (8-bit = 1 byte).
The charCodeAt() method can be used to return an integer between 0 and 65535 representing the UTF-16 code unit at the given index.
The codePointAt() can be used to return the entire code point value for Unicode characters, e.g. UTF-32.
When a UTF-16 character can't be represented in a single 16-bit code unit, it will have a surrogate pair and therefore use two code units( 2 x 16-bit = 4 bytes)
See Unicode encodings for different encodings and their code ranges.
The Blob interface's size property returns the size of the Blob or File in bytes.
const getStringSize = (s) => new Blob([s]).size;
I'm working with an embedded version of the V8 Engine.
I've tested a single string. Pushing each step 1000 characters. UTF-8.
First test with single byte (8bit, ANSI) Character "A" (hex: 41).
Second test with two byte character (16bit) "Ω" (hex: CE A9) and the
third test with three byte character (24bit) "☺" (hex: E2 98 BA).
In all three cases the device prints out of memory at
888 000 characters and using ca. 26 348 kb in RAM.
Result: The characters are not dynamically stored. And not with only 16bit. - Ok, perhaps only for my case (Embedded 128 MB RAM Device, V8 Engine C++/QT) - The character encoding has nothing to do with the size in ram of the javascript engine. E.g. encodingURI, etc. is only useful for highlevel data transmission and storage.
Embedded or not, fact is that the characters are not only stored in 16bit.
Unfortunally I've no 100% answer, what Javascript do at low level area.
Btw. I've tested the same (first test above) with an array of character "A".
Pushed 1000 items every step. (Exactly the same test. Just replaced string to array) And the system bringt out of memory (wanted) after 10 416 KB using and array length of 1 337 000.
So, the javascript engine is not simple restricted. It's a kind more complex.
You can try this:
var b = str.match(/[^\x00-\xff]/g);
return (str.length + (!b ? 0: b.length));
It worked for me.