Develop Reference

Why won't window.btoa work on – ” characters in Javascript?

So I'm converting a string to BASE64 as shown in the code below...
var str = "Hello World";
var enc = window.btoa(str);
This yields SGVsbG8gV29ybGQ=. However if I add these characters – ” such as the code shown below, the conversion doesn't happen. What is the reason behind this? Thank you so much.
var str = "Hello – World”";
var enc = window.btoa(str);

btoa is an exotic function in that it requires a "Binary String", which is an 8-bit clean string format. It doesn't work with unicode values above charcode 255, such as used by your em dash and "fancy" quote symbol.
You'll either have to turn your string into a new string that conforms to single byte packing (and then manually reconstitute the result of the associated atob), or you can uri encode the data first, making it safe:
> var str = `Hello – World`;
> window.btoa(encodeURIComponent(str));
"SGVsbG8lMjAlRTIlODAlOTMlMjBXb3JsZA=="
And then remember to decode it again when unpacking:
> var base64= "SGVsbG8lMjAlRTIlODAlOTMlMjBXb3JsZA==";
> decodeURIComponent(window.atob(base64));
"Hello – World"

The Problem is the character ” lies outside of Latin1 range.
For this you can use unescape (now deprecated)
var str = "Hello – World”";
var enc = btoa(unescape(encodeURIComponent(str)));
alert(enc);
And to decode:
var encStr = "SGVsbG8g4oCTIFdvcmxk4oCd";
var dec = decodeURIComponent(escape(window.atob(encStr)))
alert(dec);

This ultimately owes to a deficiency in the JavaScript type system.
JavaScript strings are strings of 16-bit code units, which are customarily interpreted as UTF-16. The Base64 encoding is a method of transforming an 8-bit byte stream into a string of digits, by taking each three bytes and mapping them into four digits, each covering 6 bits: 3 × 8 = 4 × 6. As we see, this is crucially dependent on the bit width of each symbol.
At the time the btoa function was defined, JavaScript had no type for 8-bit byte streams, so the API was defined to take the ordinary 16-bit string type as input, with the restriction that each code unit was supposed to be confined to the range [U+0000, U+00FF]; when encoded into ISO-8859-1, such a string would reproduce the intended byte stream exactly.
The character – is U+2013, while ” is U+201D; neither of those characters fits into the above-mentioned range, so the function rejects it.
If you want to convert Unicode text into Base64, you need to pick a character encoding and convert it into a byte string first, and encode that. Asking for a Base64 encoding of a Unicode string itself is meaningless.

The most bullet proof way is to work on binary data directly.
For this, you can encode your string to an ArrayBuffer object representing the UTF-8 version of your string.
Then a FileReader instance will be able to give you the base64 quite easily.
var str = "Hello – World”";
var buf = new TextEncoder().encode( str );
var reader = new FileReader();
reader.onload = evt => { console.log( reader.result.split(',')[1] ); };
reader.readAsDataURL( new Blob([buf]) );
And since the Blob() constructor does automagically encode DOMString instances to UTF-8, we could even get rid of the TextEncoder object:
var str = "Hello – World”";
var reader = new FileReader();
reader.onload = evt => { console.log( reader.result.split(',')[1] ); };
reader.readAsDataURL( new Blob([str]) );

Java and javascript generates different output for Base64 Hmac SHA256

When I tried to create similar function in Java and javascript which outputs a Base64 string of a Hmac SHA 256 encryption, the output given is not the same.
Javascript
var dataToSign = "message";
var secret = "secret";
function generateAuthHeader(dataToSign){
var hash = CryptoJS.HmacSHA256(dataToSign,secret);
return hash.toString(CryptoJS.enc.Base64);
which outputs
+eZuF5tnR65UEI+C+K3os8Jddv0wr95sOVgixTAZYWk=
Java
String key="secret";
String dataToSign = "message";
Mac sha256_HMAC = Mac.getInstance("HmacSHA256");
SecretKeySpec secret_key = new SecretKeySpec(key.getBytes("UTF-8"), "HmacSHA256");
sha256_HMAC.init(secret_key);
System.out.println(Base64.getEncoder().encodeToString(sha256_HMAC.doFinal(dataToSign.getBytes("UTF-8"))));
which outputs
i19IcCmVwVmMVz2x4hhmqbgl1KeU0WnXBgoDYFeWNgs=
is there errors in my code?

What's the value of hash in the JavaScript version and what's the value of sha256_HMAC.doFinal(dataToSign.getBytes("UTF-8")) in the Java version?
Also, can you please try getting the Base64 repr in JavaScript:
CryptoJS.enc.Base64.stringify(hash);
instead of hash.toString ?
Also UTF-8 shouldn't make a difference in the Java version, but I'd try to remove those and simply do getBytes() everywhere.

Converting string containing decimal code to plain string

I am getting strings from an API endpoint like: Senol Güneş Spor Kompleksi. ş is a Turkish letter, ş. I want my string to be Senol Güneş Spor Kompleksi. I tried JSON.Parse and utf8 package without success. Can I parse this string to a plain string?

I solved the problem by using html-entities package.
const Entities = require('html-entities').AllHtmlEntities;
const entities = new Entities();
const decodedString = entities.decode(originalString);

How to decode String which contains characters like 'Total\x20Value' my actual value is 'Total Value'

How to decode String which contains characters like 'Total\x20Value' my actual value is 'Total Value'
Using javascript it is getting decoded by the browser like:
if I write on browser console:
var a = 'Total\x20Value';
then I print a then it will print 'Total Value' mean browser decoded this string automatically
Now my question is how can I do this in Java code, I want this string to be decoded in java code but I am not getting a way to decode it in Java.
One more thing I can not go for string replace solution for this case because the given string only contains a space char but at run time I will get different characters so I need a generic solution in which I can decode any string without any replace operation.
One more string example is :
"DIMENSION\x5f13420895086619127059036175667828\x7e\x24\x7e1\x7e\x24\x7e1"
its real string is :
"DIMENSION_13420895086619127059036175667828~$~1~$~1".
Suggest something If it can be achieved in Java using some predefined class I have gone through with many solutions but nothing worked for me.

I suspect that a better way to address the problem you have is to fix the way these strings are created, so they don't have substrings such as \x20 or \x7e to start off with.
However, these strings could well be coming from a third-party API which you might not have any control over. If that's the case, the following method should help. It takes the string value you want to decode, containing such substrings, and replaces them with the appropriate characters:
import java.util.regex.*;
// ...
private static String decode(String input) {
Pattern p = Pattern.compile("\\\\x[0-9A-Fa-f]{2}");
Matcher m = p.matcher(input);
StringBuffer sb = new StringBuffer();
while (m.find()) {
String matchedText = m.group(0);
int characterCode = Integer.parseInt(matchedText.substring(2), 16);
m.appendReplacement(sb,
Matcher.quoteReplacement(Character.toString((char)characterCode)));
}
m.appendTail(sb);
return sb.toString();
}
There are a few things to note about it:
The overall structure of this code is based on example code in the Matcher documentation.
A regexp to match a substring of the form \x24 or \x7e is \\x[0-9A-Fa-f]{2}. Note that we have to double the backslash \ because \ has special meaning in regular expressions and we want to match an actual \ character. However, \ also has a special meaning in Java string literals so we need to double it again.
We need to use Matcher.quoteReplacement to ensure that the string we are replacing with is interpreted as that string and nothing else. In the replacement string, $1 for example will be interpreted as the first matched group, and $ on its own will cause an exception to be thrown. (Fortunately, your second example string contained $ characters - without those I may well have missed this.)
You may want to consider moving the Pattern to a static final constant somewhere, to avoid the regular expression being compiled every time the method is called.

Those \xNN substrings are just the hexadecimal ASCII code of the encoded character. You can find such an ASCII table here.
You can create your own map which holds the mapping hexadecimal to character and use it to manipulate your strings. Example:
import java.util.HashMap;
import java.util.Map;
public class NewClass {
public static void main(String[] args){
String str1 = "Total\\x20Value";
String str2 = "DIMENSION\\x5f13420895086619127059036175667828\\x7e\\x24\\x7e1\\x7e\\x24\\x7e1";
System.out.println(decode(str1));
System.out.println(decode(str2));
}
public static String decode(String str){
Map<String,String> map = new HashMap<>();
//you can extend this to x<256 if you expect your strings to contain special characters like (Ã,Ç,Æ,§,¾ ...)
for(int i = 0; i< 128; i++){
map.put((i<16?"\\x0":"\\x")+Integer.toHexString(i), Character.toString((char)i));
}
for(String key: map.keySet()){
if(str.contains(key)){
str = str.replace(key, map.get(key));
}
}
return str;
}
}

This solution involves conversion of Regular expression to Hexadecimal String and then getting the equivalent ASCII value representing the Hexadecimal String. Finally replacing the Hex string with equivalent ASCII character.
public static void main(String[] args) {
String input = "Total\\x20Value\\x7e";
String modifiedInput = input.replace("\\", "0");
for (int i = 0 ; i<modifiedInput.length() ; i++)
{
if(i<=modifiedInput.length()-3){
if (modifiedInput.charAt(i) == '0' && modifiedInput.charAt(i+1)=='x')
{
String subString = modifiedInput.substring(i, i+4) ;
String ascii = convert(subString);
modifiedInput = modifiedInput.replace(subString.toString(), ascii);
}
}
}
System.out.println(modifiedInput);
}
public static String convert(String hexDigits){
// byte[] bytes = new byte[hexDigits.length];
byte[] bytes = new byte[1];
bytes[0] = Integer.decode(hexDigits).byteValue();
String result;
result = new String(bytes);
return result;
}
}

CryptoJS not decrypting non-Latin characters faithfully

I am trying to use CryptoJS AES, like so:
var msg = "café";
var key = "something";
var c = CryptoJS.AES.encrypt(msg, key).toString();
CryptoJS.AES.decrypt(c, key).toString(CryptoJS.enc.Latin1);
Unfortunately this returns café, not café. Clearly Latin1 is not the right encoding to use, but I can't find a better one. Is there a solution?
Thanks.

You are just missing the format
The proper way is using CryptoJS.enc.Utf8
So, Please try:
CryptoJS.AES.decrypt(c, key).toString(CryptoJS.enc.Utf8);

https://code.google.com/p/crypto-js/#The_Hasher_Input
The hash algorithms accept either strings or instances of CryptoJS.lib.WordArray [...] an array of 32-bit words. When you pass a string, it's automatically converted to a WordArray encoded as UTF-8.
So, when you pass a string (and don't use CryptoJS.enc.* to generate a WordArray) it automatically converts the string (message) to a utf8 WordArray.
See here for sample roundtrip encrypt/decrypt:
https://code.google.com/p/crypto-js/#The_Cipher_Output
Here's a jsfiddle to play with CryptoJS
https://jsfiddle.net/8qbf4746/4/
var message = "café";
var key = "something";
var encrypted = CryptoJS.AES.encrypt(message, key);
//equivalent to CryptoJS.AES.encrypt(CryptoJS.enc.Utf8.parse(message), key);
var decrypted = CryptoJS.AES.decrypt(encrypted, key);
$('#1').text("Encrypted: "+encrypted);
$('#2').text("Decrypted: "+decrypted.toString(CryptoJS.enc.Utf8));
To emphasize my point here is the same thing using Latin1 encoding:
https://jsfiddle.net/3a8tf48f/2/
var message = "café";
var key = "something";
var encrypted = CryptoJS.AES.encrypt(CryptoJS.enc.Latin1.parse(message), key);
var decrypted = CryptoJS.AES.decrypt(encrypted, key);
$('#1').text("Encrypted: " + encrypted);
$('#2').text("Decrypted: " + decrypted.toString(CryptoJS.enc.Latin1));
On a side note, the API would probably be better if it only accepted WordArray and didn't overload the toString method (which is just a convenience interface to CryptoJS.enc.*.stringify). The string conversion magic is a little misleading.

You are trying to decrypt your data as a Latin1 string, even though your input string is not in Latin1. The encoding used by CryptoJS internally is not the same as the encoding you use to write the input file.
You need to specify the same encoding both when encrypting (for the string -> byte array conversion) and when decrypting (for the byte array -> string conversion).