I'm using Java and I have a Base64 encoded string that I wish to decode and then do some operations to transform.
The correct decoded value is obtained in JavaScript through function atob(), but in java, using Base64.decodeBase64() I cannot get an equal value.
Example:
For:
String str = "AAAAAAAAAAAAAAAAAAAAAMaR+ySCU0Yzq+AV9pNCCOI="
With JavaScript atob(str) I get ->
"Æ‘û$‚SF3«àö“Bâ"
With Java new String(Base64.decodeBase64(str)) I get ->
"Æ?û$?SF3«à§ö?â"
Another way I could fixed the issue is to run JavaScript in Java with a Nashorn engine, but I'm getting an error near the "$" symbol.
Current Code:
ScriptEngine engine = new ScriptEngineManager().getEngineByName("JavaScript");
String script2 = "function decoMemo(memoStr){ print(atob(memoStr).split('')" +
".map((aChar) => `0${aChar.charCodeAt(0).toString(16)}`" +
".slice(-2)).join('').toUpperCase());}";
try {
engine.eval(script2);
Invocable inv = (Invocable) engine;
String returnValue = (String)inv.invokeFunction("decoMemo", memoTest );
System.out.print("\n result: " + returnValue);
} catch (ScriptException | NoSuchMethodException e1) {
e1.printStackTrace();
Any help would be appreciated. I search a lot of places but can't find the correct answer.
btoa is broken and shouldn't be used.
The problem is, bytes aren't characters. Base64 encoding does only one thing. It converts bytes to a stream of characters that survive just about any text-based transport mechanism. And Base64 decoding does that one thing in reverse, it converts such characters into bytes.
And the confusion is, you're printing those bytes as if they are characters. They are not.
You end up with the exact same bytes, but javascript and java disagree on how you're supposed to turn that into an ersatz string because you're trying to print it to a console. That's a mistake - bytes aren't characters. Thus, some sort of charset encoding is being used, and you don't want any of this, because these characters clearly aren't intended to be printed like that.
Javascript sort of half-equates characters and bytes and will freely convert one to the other, picking some random encoding. Oof. Javascript sucks in this regard, it is what it is. The MDN docs on btoa explains why you shouldn't use it. You're running into that problem.
Not entirely sure how you fix it in javascript - but perhaps you don't need it. Java is decoding the bytes perfectly well, as is javascript, but javascript then turns those bytes into characters into some silly fashion and that's causing the problem.
What you have there is not a text string at all. The giveaway is the AA's at the beginning. Those map to a number of zero bytes. That doesn't translate to meaningful text in any standard character set.
So what you have there is most likely binary data. Converting it to a string is not going to give you meaningful text.
Now to explain the difference you are seeing between Java and Javascript. It looks to me as if both Java and Javascript are making a "best effort" attempt to convert the binary data as if is was encoded in ISO-8859-1 (aka ISO LATIN-1).
The problem is some of the bytes codes are mapping to unassigned codes.
In the Java case those unassigned codes are being mapped to ?, either when the string is created or when it is being output.
In the Javascript case, either the unassigned codes are not included in the string, or them are being removed when you attempt to display them.
For the record, this is how an online base64 decoder the above for me:
����������������Æû$SF3«àöBâ
The unassigned codes are 0x91 0x82 and 0x93. 0x15 and 0x0B are non-printing control codes.
But the bottom line is that you should not be converting this data into a string in either Java or in Javascript. It should be treated as binary; i.e. an array of byte values.
byte[] data = Base64.getDecoder().decode(str);
Related
I'm definitely missing something about the TextEncoder and TextDecoder behavior. It seems to me like the following code should round-trip, but it doesn't seem to:
new TextDecoder().decode(new TextEncoder().encode(String.fromCharCode(55296))).charCodeAt(0);
Since I'm just encoding and decoding the string, the char code seems like it should be the same, but this returns 65533 instead of 55296. What am I missing?
Based on some spelunking, the TextEncoder.encode() method appears to take an argument of type USVString, where USV stands for Unicode Scalar Value. According to this page, a USV cannot be a high-surrogate or low-surrogate code point.
Also, according to MDN:
A USVString is a sequence of Unicode scalar values. This definition
differs from that of DOMString or the JavaScript String type in that
it always represents a valid sequence suitable for text processing,
while the latter can contain surrogate code points.
So, my guess is your String argument to encode() is getting converted to a USVString (either implicitly or within encode()). Based on this page, it looks like to convert from String to USVString, it first converts it to a DOMString, and then follows this procedure, which includes replacing all surrogates with U+FFFD, which is the code point you see, 65533, the "Replacement Character".
The reason String.fromCharCode(55296).charCodeAt(0) works I believe is because it doesn't need to do this String -> USVString conversion.
As to why TextEncoder.encode() was designed this way, I don't understand the unicode details well enough to attempt to explain, but I suspect it's to simplify implementation since the only output encoding it supports seems to be UTF-8, in an Uint8Array. I'm guessing requiring a USVString argument without surrogates (instead of a native UTF-16 String possibly with surrogates) simplifies the encoding to UTF-8, or maybe makes some encoding/decoding use cases simpler?
For those (like me) who aren't sure what "unicode surrogates" are:
The problem
The character code 55296 is not a valid character by itself. So this part of the code is already a problem:
String.fromCharCode(55296)
Since there is no valid character at that charCode, the .fromCharCode function returns the error character "�" instead, which happens to have the code 65533.
Codes like 55296 are only valid as the first element of a pair of codes. Pairs of codes are used to represent the characters that didn't fit in Unicode's Basic Multilingual Plane. (There are a lot of characters outside the Basic Multilingual Plane, so they need two 16-bit numbers to encode them.)
For example, here is a valid use of the code 55296:
console.log(String.fromCharCode(55296, 57091)
It returns the character "𐌃", from the ancient Etruscan alphabet.
The solution
This code will round-trip correctly:
const code = new TextEncoder().encode(String.fromCharCode(55296, 57091));
console.log(new TextDecoder().decode(code).charCodeAt(0)); // Returns 55296
But beware: .charCodeAt only returns the first part of the pair. A safer option might be to use String.codePointAt to convert the character into a single 32-bit code:
const code = new TextEncoder().encode(String.fromCharCode(55296, 57091));
console.log(new TextDecoder().decode(code).codePointAt(0)); // Returns 66307
Two questions in one, not sure if that's allowed, but they're directly related to the same code.
I retrieve a CSV string as a HTTP response in Javascript - this string comes in UTF-16 encoding it seems, as it has for example ' € ' instead of '€'.
a) How can I convert this to UTF-8 in vanilla Javascript?
Once that's done, how do I
b) transform the multi-line CSV into a 2D array in vanilla Javascript?
Thanks!
[UPDATE]
Based on anqooqie's pointers, I take the following approach to re-encode the string:
OK, clear - so to be honest, I went a slightly different way (as the reencode function didn't work for me and it threw a generic error code) and now do the below;
var O = new ActiveXObject('ADODB.Stream');
O.Type = 2;
O.Open;
O.Charset = 'ISO-8859-1';
O.LineSeparator = 10;
O.WriteText (csvStr);
O.Position = 0;
O.Charset = 'UTF-8';
And this works fine and in pretty much a split second (even though it's a 35K row CSV). Now if I want to put it back into the csvStr, I would do
csvStr = O.ReadText
but this takes ages - is that expected or am I doing something wrong?
For putting it into a 2D array, I split on the LineSeparator and then loop using a regex, which seems to work.
var A = new Array
A.push(csvStr[0].match(/"[^"]*"|[^,]+/g))
The vast delay on the readText is bothering me though, especially as the WriteText is so quick. Any help is appreciated.
Looks like you are confused about the terms of character encoding, so let's reconfirm that.
String is just a string.
There is no "UTF-16 string", nor "UTF-8 string".
Character encoding is a protocol which converts between a string and a byte array.
UTF-16 is one of the character encodings.
Also, both of UTF-8 and ISO-8859-1 are character encodings.
In UTF-16, the string '€' can be encoded to a byte array 20 AC.
In UTF-8, the string '€' can be encoded to a byte array E2 82 AC.
In ISO-8859-1, the byte array E2 82 AC can be decoded to a string 'â¬'.
Now, you may find that 'â¬' is not a "UTF-16 string".
It is '€' encoded as UTF-8 and mistakenly decoded as ISO-8859-1.
a) How can I convert this to UTF-8 in vanilla Javascript?
What you should do is to fix the code to retrieve a CSV file.
I cannot tell you how to fix it since I do not know your code, but I believe that it now decodes a CSV file as ISO-8859-1.
You should fix the character encoding from ISO-8859-1 to UTF-8.
If the code is not yours and you cannot fix it, you can use a workaround.
In other words, you can 1) re-encode a mistakenly decoded string as ISO-8859-1, and 2) re-decode it as UTF-8.
1)
// Note: This code requires ES5 or later.
function reencode(inputString) {
return Array.apply(null, Array(inputString.length)).map(function (x, i) { return inputString.charCodeAt(i); });
}
2)
See this answer.
b) How do I transform the multi-line CSV into a 2D array in vanilla Javascript?
See this answer.
I have a problem URL-encoding a text with javascript.
I am in Germany, where we have these "Umlaute" (ÄÖÜ), and these letters make some problems.
An online encoder/decoder returned the following results for the word "Äpfel" (apples).
Äpfel >>> url-encode >>> %C3%84pfel
%C3%84pfel >>> url-decode >>> Äpfel
For testing, I created the following php.file (poc.php) with no php-content, just the javascript:
<script type="text/javascript">
var t = "Äpfel";
t = encodeURI(t);
alert(t);
t = decodeURI(t);
alert(t);
</script>
The first alert returns "%EF%BF%BDpfel", which differs from the result of the online encoder.
The second alert returns "�pfel" (yes, the diamond with the "?").
It seems that javascript cannot decode the text it just encoded.
I guess the cause of this behaviour is somewhere in the PHP settings. When I just rename the file from "poc.php" to "poc.html" the encoding is correct and the alerts return the same results as the online encoder/decoder.
When I check the current encoding, javascript and php return "utf-8".
In my "real" project I have a ".js" file included in my php-file (with the same problem).
<script type="text/javascript" src="scripts/functions.js"></script>
Has anybody an idea what causes this behaviour?
The weird byte stream %EF%BF%BD you're receiving is utf-8 version of the Unicode replacement character, that is, literally the � symbol.
The Javascript portion can url-decode the text it just url-encoded, it was just asked to encode the symbol for a missing symbol.
So: some part of your system is not using utf-8, but some other character set instead, and there's an unnecessary conversion done. My guess is that the file is encoded in latin-1, aka. ISO 8859-1, and PHP tries to read it as if it was UTF-8, converting the unrecognized character 0xc4 ('Ä' in latin-1) to the replacament character symbol.
I'm trying to find a simple way to record and temporarily obfuscate answers to "quiz" questions I'm writing in Markdown. (I'll tell the students the quiz answers during the presentation, so I'm not looking for any kind of secure encryption.)
I thought I could use atob('message I want to obfuscate') then tell students they can use btoa() in their developer tools panel to reverse the process. However the following does not return 'one':
btoa( atob('one') )
Does anyone know why this doesn't return 'one'? Are there other methods built into JavaScript that will allow one to loosely encrypt and decrypt a message? (I'm working with absolute beginners who might be confused by functions and who would be very confused trying to add libraries to a page).
That is the reason.
In Base64 encoding, the length of output encoded String must be a
multiple of 3. If it's not, the output will be padded with additional
pad characters (=). On decoding, these extra padding characters will
be discarded.
var string1 = "one",
string2 = "one2";
console.log("Value of string1", string1)
console.log("Decoded string1", atob(string1))
console.log("Encoded string1", btoa(atob(string1)))
console.log("-------------------------------------")
console.log("Value of string2", string2)
console.log("Decoded string2", atob(string2))
console.log("Encoded string2", btoa(atob(string2)))
As #george pointed out, one must use btoa() before using atob():
atob( btoa( 'hello' ) )
btoa means binary to ascii: input is Binary=any kind of data: text, images, audio. Output is Ascii=its base64 encoding, which is an ascii subset, i.e. a text string containing only upper and lowercase letters, numbers, comma, plus, slash, equal sign (only for padding at end).
atob means ascii to binary: input MUST be a subset of Ascii, i.e. the result of a base64 encoded string. Output is Binary=any type of data (text, image, audio, ...).
I have a UTF-8 encoded file "myFile.aaa" with non-printable char represented by hexadecimal x80 (decimal 128).
I need to develop a Javascript function that will read this char from myFile.aaa and return its decimal value, 128.
Is it possible to do that? How?
If I copy myFile.aaa content to "var data", and do "data[0].charCodeAt(0)" I get value 8364 instead of 128.
Thanks
I don't think your UTF-8 encoding makes sense, so I'm going to tell you the best way I've found of dealing with dodgy AJAX data. Set the content type as user-defined:
var req = new XMLHttpRequest();
req.overrideMimeType('text/plain; charset=x-user-defined')
You can then just read the file as plain bytes instead of encoded characters.