Some background information, you can skip this part for the actual question
this is my third question about this topic here at stackoverflow. To be complete, these are the other questions AES with crypt-js and PyCrypto and Match AES de/encryption in python and javascript. Unfortunately my last attempt got two downvots for the original question. The problem was, even I did not know what my real question was. I just dug around to find the real question I was looking for. With the feedback in the comments, and reading some additional information, I updated my question. I excavate the right question, I think. But my problem didn't get any more views after my updates. So I really hope, that this question is now more clear and understandable - even I know what my Problem is now :D
Thank you all for making stackoverflow to this cool community - I often found here solutions for my problems. Please keep giving feedback to bad questions, so they can be improved and updated, which increases this huge knowledge and solutions database.
And feel free to correct my english grammar and spelling.
The Problem
AES in Javascript
I have an encrypted String which I can decrypt with this this Javascript Implementation of AES 256 CTR Mode
password = "myPassphrase"
ciphertext = "bQJdJ1F2Y0+uILADqEv+/SCDV1jAb7jwUBWk"
origtext = Aes.Ctr.decrypt(ciphertext, password, 256);
alert(origtext)
This decrypts my string and the alert box with This is a test Text pops up.
AES with PyCrypto
Now I want to decrypt this string with python and PyCrypto
password = 'myPassphrase'
ciphertext = "bQJdJ1F2Y0+uILADqEv+/SCDV1jAb7jwUBWk"
ctr = Counter.new(nbits=128)
encryptor = AES.new(key, AES.MODE_CTR, counter=ctr)
origtext = encryptor.decrypt(base64.b64decode(ciphertext))
print origtext
This code does not run. I get an ValueError: AES key must be either 16, 24, or 32 bytes long. When I recognized, that I have to do more in PyCrypto then just call a decrypt method, I started to investigate end try to figure out what I have to do.
Investigation
The basic things I figured out first, were:
AES 256 Bit (?). But AES standard is 128 bit. Does increasing the passphrase to 32 Byte is enough?
Counter Mode. Easily to set in PyCrypto with AES.MODE_CTR. But I have to specify a counter() Method. So I used the basic binary Counter provided by PyCrypto. Is this compatible with the Javascript Implementation? I can't figure out what they are doing.
The string is base64 encoded. Not a big problem.
Padding in general. Both passphrase and the encrypted string.
For the passphrase they do this:
for (var i=0; i<nBytes; i++) {
pwBytes[i] = isNaN(password.charCodeAt(i)) ? 0 : password.charCodeAt(i);
}
Then I did this in python
l = 32
key = key + (chr(0)*(l-len(key)%l))
But this did not help. I still get a weird string ?
A???B??d9= ,?h????' with the following code
l = 32
key = 'myPassphrase'
key = key + (chr(0)*(l-len(key)%l))
ciphertext = "bQJdJ1F2Y0+uILADqEv+/SCDV1jAb7jwUBWk"
ctr = Counter.new(nbits=128)
encryptor = AES.new(key, AES.MODE_CTR, counter=ctr)
origtext = encryptor.decrypt(base64.b64decode(ciphertext))
print origtext
Then I read more about the Javascript Implementation and it says
[...] In this implementation, the initial block holds the nonce in the first 8 bytes, and the block count in the second 8 bytes. [...]
I think this could be the key to the solution. So I tested what happens when I encrypt an empty string in Javascript:
origtext = ""
var ciphertext =Aes.Ctr.encrypt(origtext, password, 256);
alert(ciphertext)
The alert box shows /gEKb+N3Y08= (12 characters). But why 12? Shouldn't it be 8+8 = 16Bytes? Well anyway, I tried a bruteforce method on the python decryption by testing the decryption with for i in xrange(0,20): and ciphertext[i:] or base64.b64decode(ciphertext)[i:]. I know this is a very embarrassing try, but I got more and more desperate. And it didn't work either.
The future prospects are also to implement the encryption in the same way.
additional information
The encrypted string was not originally encrypted with this Javascript implementation, It's from another source. I just recognized, that the Javascript code does the right thing. So I affirm that this kind of implementation is something like a "standard".
The question
What can I do, that the encryption and decryption from a string with PyCrypto is the same like in the Javascript implementation, so that I can exchange data between Javascript and Python?
I also would switch to another crypto library in python, if you can suggest another one.
Furthermore I'm happy with any kind of tips and feedback.
And I think, all comes down to How can I include the nonce and block count to the encrypted string? and How can I extract this information for decryption?
We are still dealing with a bunch of questions here.
How can I extract the nonce and the counter for decryption?
This is easy. In the Javascript implementation (which does not follow a particular
standard in this respect) the 8-byte nonce is prepended to the encrypted result.
In Python, you extract it with:
import base64
from_js_bin = base64.decode(from_js)
nonce = from_js_bin[:8]
ciphertext = from_js_bin[8:]
Where from_js is a binary string you received.
The counter cannot be extracted because the JS implementation does not transmit it.
However, the initial value is (as typically happens) 0.
How can I use the nonce and counter to decrypt the string in Python?
First, it must be established how nonce and counter are combined to get the counter block.
It seems that the JS implementation follows the NIST 800-38A standard, where the
left half is the nonce, and the right half is the counter. More precisely, the counter
is in big endian format (LSB is the rightmost byte). This is also what Wikipedia shows:
.
Unfortunately, CTR mode is poorly documented in PyCrypto (a well-known problem).
Basically, the counter parameter must be a callable object that returns
the correct 16-byte (for AES) counter block, for each subsequent call.
Crypto.Util.Counter does that, but in an obscure way.
It is there only for performance purposes though. You can easily implement it yourself like this:
from Crypto.Cipher import AES
import struct
class MyCounter:
def __init__(self, nonce):
"""Initialize the counter object.
#nonce An 8 byte binary string.
"""
assert(len(nonce)==8)
self.nonce = nonce
self.cnt = 0
def __call__(self):
"""Return the next 16 byte counter, as binary string."""
righthalf = struct.pack('>Q',self.cnt)
self.cnt += 1
return self.nonce + righthalf
cipher_ctr = AES.new(key, mode=AES.MODE_CTR, counter=MyCounter(nonce))
plaintext = cipher_ctr.decrypt(ciphertext)
How long is the key for AES?
The key length for AES-128 is 16 bytes.
The key length for AES-192 is 24 bytes.
The key length for AES-256 is 32 bytes.
Each algorithm is different, but much of the implementation is shared.
In all cases, the algorithm operate on 16 byte blocks of data.
For simplicity, stick to AES-128 (nBits=128).
Will your code work?
I have the feeling it won't, because the way you compute the AES key seems incorrect.
The JS code encodes the password in UTF-8 and encrypts it with itself.
The result is the actual key material. It is 16 byte long, so for AES-192 and -256, the implementation copies part of it at the back. Additionally, the plaintext is also UTF-8 encoded before encryption.
In general, I suggest you follow this approach:
Make your JS implementation reproducible (right now encryption depends on the current time, which changes quite often ;-) ).
Print the value of the keys and data at each step (or use a debugger).
Try to reproduce the same algorithm in Python, and print the values too.
Investigate where they start to differ.
Once you have duplicated the encryption algorithm in Python, the decryption one should be easy.
Related
I have an app that use crypto-Js AES. The simulate working code is:
var ciphertext = CryptoJS.AES.encrypt('My_message', 'My_secret_key');
console.log(ciphertext.toString());
the answer is:
U2FsdGVkX1/Dd3uAr/mdw5lVoBvq0UX5LHnNoX24JAM=
when I try to reproduce it server side I never get the same answer:
$passphrase='My_secret_key';
$value='My_message';
$salt = openssl_random_pseudo_bytes(8);
$salt ='';
$salted = '';
$dx = '';
while (strlen($salted) < 48) {
$dx = md5($dx.$passphrase.$salt, true);
$salted .= $dx;
}
$key = substr($salted, 0, 32);
$iv = substr($salted, 32,16);
$encrypted_data = openssl_encrypt($value, 'aes-256-cbc', $key, true, $iv);
echo base64_encode($encrypted_data);
server side answer:
3jSTl1yR55lfTbz7f0o3Yw==
I must miss something but can't point out what. Local side can't touched.
All help is welcome
If the second parameter in CryptoJS.AES.encrypt is passed as string, it is interpreted as a passphrase from which the actual key and IV are derived, [1]. This is achieved by using the functionality of the OpenSSL-function EVP_BytesToKey with an iteration count of 1 and the MD5-digest, [2] [3] (note that CryptoJS doesn't consider the switch of the default digest from MD5 to SHA256 from OpenSSL version 1.1.0c on, [4]).
CryptoJS.AES.encrypt returns a CipherParams-object that encapsulates ciphertext, key, IV, and salt, [5]. In addition CipherParams#toString() returns the result in OpenSSL-format as Base64-encoded string. The OpenSSL-format consists of a 16-byte header and the subsequent ciphertext. The header starts with the ASCII-encoded string Salted__ followed by an 8-byte salt. The salt is randomly generated each time and used together with the password to derive the key / IV. This creates a different key / IV each time.
The PHP-code is functionally identical: Key and IV are derived with an analog logic from a passphrase using a freshly generated salt each time (for the proof, see below). However, some minor changes are necessary:
The following line must be removed: $salt ='';
In the current code, only the Base64-encoded ciphertext is displayed. For a Base64-encoded output of the result in OpenSSL-format the code must be instead:
echo base64_encode('Salted__'.$salt.$encrypted_data);
The 4th parameter in openssl_encrypt should be changed from true to OPENSSL_RAW_DATA. Both are functionally identical, but the use of OPENSSL_RAW_DATA is more transparent.
The JavaScript- and PHP-code generate a new salt each time and thus a different key and IV, which changes the ciphertext each time. That's the way it should be. Since the salt is stored together with the ciphertext, it is possible to decrypt the ciphertext at any time using the passphrase.
Proof that both codes use the same logic to derive key and IV: The new salt / ciphertext generated each time prevents a direct comparison of the results of both codes. In order to perform this comparison without much effort, it is best to use the salt generated in the JavaScript-code in the PHP-code as well. The salt in the JavaScript-code can be determined as hexadecimal string with:
console.log(ciphertext.salt.toString(CryptoJS.enc.Hex));
This salt is to be used in the PHP-code instead of the randomly generated salt (of course only for this one comparison):
$salt = hex2bin('<Salt from JavaScript-Code as hexadecimal string>');
A comparison of both outputs now proves that they are equal, showing that both codes are functionally identical.
I've been in your exact situation before. We had extreme difficulty getting the same results on PHP and Java (for Android). Many developers at 2 companies over many days. No luck.
We eventually ended up calling CryptoJS from PHP. If I remember correctly, there is something non-standard in Crypto JS. I could be wrong it was a while back.
Calling CryptoJS with node through PHP can be a more robust solution. Then you can use the same library which ensures compatibility.
$result = shell_exec('node yourCryptoJSprogram.js');
We did reach a limit to how much data can be passed as arguments this way. I would recommend writing to a file with PHP and reading with NodeJS again.
If performance becomes problematic, consider running an Express server and making REST calls from PHP.
If that answer doesn't satisfy you, consider using or copying this simple OpenSSL based PHP library I wrote to figure it out:
https://github.com/io-digital/php-crypto
I'm doing some work on AES and I have been seeing lots of pseudocode where, for example, if encryption is done in 10 rounds, decryption is done in 9. Primarily this one, to be exact:
http://people.eku.edu/styere/Encrypt/JS-AES.html
Is this normal? Is there something I'm missing? Is it actually 10 rounds of decryption but I'm reading the code wrong?
The real question is why people trust these sites publishing some crappy JavaScript version of a cipher.
This is the official NIST pseudo code from FIPS 197:
Cipher(byte in[4*Nb], byte out[4*Nb], word w[Nb*(Nr+1)])
begin
byte state[4,Nb]
state = in
AddRoundKey(state, w[0, Nb-1])
for round = 1 step 1 to Nr–1
SubBytes(state)
ShiftRows(state)
MixColumns(state)
AddRoundKey(state, w[round*Nb, (round+1)*Nb-1]) end for
SubBytes(state)
ShiftRows(state)
AddRoundKey(state, w[Nr*Nb, (Nr+1)*Nb-1])
out = state
end
InvCipher(byte in[4*Nb], byte out[4*Nb], word w[Nb*(Nr+1)])
begin
byte state[4,Nb]
state = in
AddRoundKey(state, w[Nr*Nb, (Nr+1)*Nb-1])
for round = Nr-1 step -1 downto 1
InvShiftRows(state)
InvSubBytes(state)
AddRoundKey(state, w[round*Nb, (round+1)*Nb-1])
InvMixColumns(state) end for
InvShiftRows(state)
InvSubBytes(state)
AddRoundKey(state, w[0, Nb-1])
out = state
end
And bang, gone is the difference. The site you pointed to made a mistake, taking it up to 11 in the encryption routine.
When looking for code, test vectors use the original or original documents and specifications.
Here is a JavaScript part which decodes a string with AES encryption
var p = 'some large string'
var s = 'Q05WTmhPSjlXM1BmeFd0UEtiOGg='
var y = CryptoJS.AES.decrypt({
ciphertext: CryptoJS.enc.Base64.parse(p)
}, CryptoJS.enc.Base64.parse(s), {
iv CryptoJS.enc.Hex.parse("random")
});
var v = y.toString(CryptoJS.enc.Utf8)
I am trying to code a similar decoding function in python with importing AES.
Could anyone help me with this one. I can't figure out all equivalent code for js to python.
I looked up this page
Python AES Decryption Routine (Code Help)
and
AES - Encryption with Crypto (node-js) / decryption with Pycrypto (python)
Not sure if they have the code similar to the js I have here
"y.toString(CryptoJS.enc.Utf8)"
This in python what it means
I have tried something like this from another source
from base64 import b64decode
from Crypto.Cipher import AES
iv = 'random'
key = 'Q05WTmhPSjlXM1BmeFd0UEtiOGg='
encoded = b64decode('some large string')
dec = AES.new(key=key, mode=AES.MODE_CBC, IV=iv)
value = dec.decrypt(encoded)
There are multiple problems with your CryptoJS code and Python code.
Wrong key size
Your key s contains only 20 bytes (160 bit) which doesn't constitute any of the valid key sizes for AES which are 128 (10), 192 (12) and 256 bit (14 rounds). CryptoJS will silently run the key schedule for a 160 bit key with 11 rounds which PyCrypto doesn't support (see AES.c).
You can reduce the key to 128 bit like this in CryptoJS:
var key = CryptoJS.enc.Base64.parse('Q05WTmhPSjlXM1BmeFd0UEtiOGg=');
key.sigBytes = 16;
key.clamp();
or in Python:
key = b64decode('Q05WTmhPSjlXM1BmeFd0UEtiOGg=')[:16]
Wrong character encoding
You forgot to decode the key from a Base64 string in Python and you forgot to decode the IV from hex. The character '0' and the byte 0x00 are entirely different. There's an easier way to define an all zero IV:
iv = "\0"*16
No unpadding
CryptoJS uses PKCS#7 padding by default, but PyCrypto doesn't implement any padding and only handles data as a multiple of the block size. After you decrypt something, you need to remove the padding yourself in Python:
value = value[:value[-1]]
(the last byte determines how many bytes are padding bytes). More on that here.
Other considerations:
You really shouldn't be setting the IV to a static value. The IV should be randomly generated for every encryption using the same key. Otherwise, you will lose semantic security. Since the IV doesn't have to be secret, you can put it in front of the ciphertext and slice it off before decryption.
I'm using CryptoJS to encrypt some usernames and passwords, it's working well enough I think.
But I have some questions regarding the encrypted data plaintext.
No matter what the key or data is it always starts with "U2FsdGVkX1...".
The encrypted data changes constantly even if the input data remains the same as shown below:
U2FsdGVkX1/BshMm2v/DcA6fkBQGPss6xKa9BTyC8g0=
U2FsdGVkX1/uc5OTSD7CfumdgqK1vN2LU4ISwaQsTQE=
U2FsdGVkX1/8OOLOTZlfunN4snEVUdF2ugiL7SeAluE=
U2FsdGVkX1+c8j3l1NRBJDb1byHwOmmNSmbTci22vsA=
username_encrypted = CryptoJS.AES.encrypt(username, key);
password_encrypted = CryptoJS.AES.encrypt(password, key);
console.log(username_encrypted.toString());
console.log(password_encrypted.toString());
console.log(CryptoJS.AES.decrypt(username_encrypted, key).toString(CryptoJS.enc.Utf8));
console.log(CryptoJS.AES.decrypt(password_encrypted, key).toString(CryptoJS.enc.Utf8));
Is this the way it is supposed to work or am I doing something wrong? Because on some online AES encryption sites I get very different results, encrypted data not changing all the time for one.
That's correct. CryptoJS uses the OpenSSL proprietary format. If it uses a salted password to derive a key, it has a magic value in front. E.g. your first base64 translates into
53616C7465645F5F C1B21326DAFFC3700E9F9014063ECB3AC4A6BD053C82F20D
in hex. Connoisseurs will immediately recognize the first 8 bytes as being ASCII,
Salted__ | C1B21326DAFFC370 0E9F9014063ECB3AC4A6BD053C82F20D
So what you have is first a 8 byte magic, then an 8 byte salt, then the ciphertext.
The actual key and IV are derived from the key in your code (which is actually interpreted as being a passphrase). As the salt is always random, so are the derived key and IV. This is how it should be as otherwise you could distinguish plaintext that start with (or is fully identical) to other plaintext.
I have the following JSON:
[{"hashcode": 4830991188237466859},{...}]
I have the following Angular/JS code:
var res = $resource('<something>');
...
res.query({}, function(json) {hashcode = json[0].hashcode;};
...
Surprisingly (to me, I'm no JS expert), I find that something (?) is rounding the value to the precision of 1000 (rounding the last 3 digits). This is a problem, since this is a hash code of something.
If, on the other hand I write the value as a String to the JSON, e.g -
[{"hashcode": "4830991188237466859"},{...}]
this does not happen. But this causes a different problem for me (with JMeter/JSON Path, which extracts the value ["4830991188237466859"] by running my query $.hashcode - which I can't use as a HTTP request parameter (I need to add ?hashcode=... to the query, but I end up with ?hashcode=["..."]
So I appreciate help with:
Understanding who and why -- is rounding my hash, and how to avoid it
Help with JMeter/JSON Path
Thanks!
Each system architecture has a maximum number it can represent. See Number.MAX_VALUE or paste your number into the console. You'll see it happens at the JavaScript level, nothing to do with angular. Since the hash doesn't represent the amount of something, it's perfectly natural for it to be a string. Which leads me to
Nothing wrong with site.com/page?hashcode=4830991188237466859 - it's treated as a string there and you should keep treating it as such.
The javascript Number type is floating point based, and can only represent all integers in the range between -253 and 253. Some integers outside this range are therefore subject to "rounding" as you experience.
In regards to JMeter JSON Path Extractor plugin, the correct JSON Path query for your hashcode will look like
$..hashcode[0]
See Parsing JSON chapter of the guide for XPath to JSON Path mappings and more details.