I am implementing decoding of BER-compressed integers and recently I've found a weird JavaScript behavior related to bitwise operations with big integers.
E.g.:
var a = 17516032; // has 25 bits
alert(a << 7) // outputs -2052915200
alert(a * 128) // outputs 2242052096
alert(2242052096 >> 16) // outputs -31325
alert(2242052096 / 65536) // outputs 34211
While the first workaround (multiplication instead of left shift) is acceptable, the second isn't.
Why it happens? How to bear with it?
17516032 in binary is 00000001000010110100011000000000. Shifting to the left by 7 gives you 10000101101000110000000000000000. This is equal to -2052915200 in two's complement (which is how almost all computers represent negative numbers).
>> is a signed right shift. That means that the leftmost bit (which determines the sign of a number) will be shifted into the left side.
e.g.
1100 >> 2 == 1111
0111 >> 2 == 0001
If you want to do an unsigned shift (which ignores the sign bit), use >>> which will zero-fill the left end of the bitstring.
Bitwise operators work on 32 bit integers, while multiplication and division works on floating point numbers.
When you shift a number, it's converted from a floating point number to a 32 bit integer before the operations, and converted back to a floating point number after the operation. The number 2242052096 has the 32nd bit set, so it is a negative number when converted to and from a 32 bit integer.
The >> right shift operator doesn't change the sign of the value, i.e. the bits that are shifted in from the left have the same value as the sign bit. Use the >>> right shift operator to shift in zero bits instead.
Reference: MDN: Bitwise operators
(2242052096 / 65536) == (2242052096 >>> 16)
Note the different shift.
Javascript normally represents numbers as (double-precision) floating point.
Almost all bitwise operations convert to a signed 32-bit integer, do whatever they're going to do, then treat the result as a signed 32-bit integer when converting back.
The exception is >>> which treats the result as an unsigned 32-bit integer when converting back.
So:
right shifts can be made to work simply by using >>> instead of >> ;
a * 128 gives the expected answer because it's never converted to a signed 32-bit integer in the first place - it's just a floating-point multiplication;
a << 7 gives an unexpected answer because it's converted to a signed 32-bit integer, and then you shift a 1 into the sign bit, resulting in a negative signed 32-bit value.
There isn't a <<<, but if you want to write your left shift as a shift, you can use
(a << 7) >>> 0
to get the expected answer (the >>> 0 effectively casts the signed 32-bit value to an unsigned 32-bit value).
Related
EDIT: Explanation at the end.
I was trying to implement a 64bit integer class using a Uint32Array and have bitwise operations performed under the hood on two uint32 members. I quickly found out that, as to my understanding of the specification, bitwise operations return a signed 32bit integer. Initially I was hoping that the Uint32Array would just take care of the sign bit, but it doesn't.
I tried coding around the sign issue, but I am stuck at something I simply can't make sense of at all.
var a = (Math.pow(2, 32)-1); //set a to uint32 max value
So far, so good.
a.toString(2);// gives "11111111111111111111111111111111", as expected
However:
(a << 0); // gives "-1"
(a >> 1); // gives "-1"
(a << 0) == (a >> 1); // evaluates to true
Even if JS bitwise operations turn numbers into signed 32bit integers, 32 set bits shifted to the right by 1 should never be -1. Or should they? Should a non-zero number shifted by 0 bits equal itself shifted 1 bit? Is this a bug? Am I running into undefined behaviour?
Usually the answer to similar questions has to do with the signed 32bit conversion but I can't see how that should cause this behaviour.
EDIT2, explanation: The cause of my confusion was a fundamental misunderstanding of how negative numbers are represented in binary. While the first bit is in fact the sign bit, 1 indicating a negative, 0 a positive number, the remaining bits aren't just used to store the abs(), as I assumed.
Signed 4bit example:
0111 equals +7. 1111 does not equal -7, it equals -1. How do we end up with negative one? Because the two's complement of 1111 is 0001. To get a number's two's complement, flip all bits and add one:
1111 -> 0000 -> 0001.
Now that I know that, making sense of 11..11 << 0 being -1 is easy. It's perfectly similar to my 4bit example. 11..11 >> 1 being -1 is also completely expected now. The signed right shift >> is 1 filling, so 11..11 >> 1 is still 11..11 which is still -1.
I will leave this as is for now, because I'm certainly not the only one misunderstanding binary signed integer representation. Thanks for everyone's time.
Even if JS bitwise operations turn numbers into signed 32bit integers, 32 set bits shifted to the right by 1 should never be -1. Or should they? Should a non-zero number shifted by 0 bits equal itself shifted 1 bit? Is this a bug? Am I running into undefined behaviour?
That's normal, expected and defined. And yes, they should.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Right_shift is what you use, and its description is this:
The right shift operator (>>) shifts the first operand the specified number of bits to the right. Excess bits shifted off to the right are discarded. Copies of the leftmost bit are shifted in from the left. Since the new leftmost bit has the same value as the previous leftmost bit, the sign bit (the leftmost bit) does not change. Hence the name "sign-propagating".
So if you have 32 bits of 1, after applying right shift by 1 you will have 32 bits of 1.
The fact that it's 32 bits wide is in the specs, https://tc39.es/ecma262/
6.1.6.1.10 Number::signedRightShift ( x, y )
[...]
4. Return the result of performing a sign-extending right shift of lnum by shiftCount bits. The most significant bit is propagated. The result is a signed 32-bit integer.
(Similarly, << produces 32-bit signed integer)
Doing some tests with bitwise operations / shifting with JavaScript
0x80000000 >> 1 // returns -1073741824 (-0x40000000)
I would expect that to return 0x40000000 since
0x40000000 >> 1 // returns 0x20000000
0x20000000 >> 1 // returns 0x10000000
Its an arithmetic shift that's why the sign is preserved, to do a logical shift use >>>
0x80000000 >>> 1 // returns 1073741824 (0x40000000)
In Javascript, >> is the Signed Right Shift Operator. In §11.7.2 of the language specification it says:
Performs a sign-filling bitwise right shift operation on the left operand by the amount specified by the right operand.
Before the shifting is done, the left operand is converted to a signed 32-bit integer (step 5 of the algorithm given in the specification). In your case this conversion turns the given positive number into a negative value. After that, the actual shifting is done, preserving the negative sign of the value.
When I try to set 31 bit 0 | 1 << 31 I get the following result:
console.log(0 | 1 << 31); // -2147483648
Which is actualy:
console.log((-2147483648).toString(2)) // -10000000000000000000000000000000
Is it correct to set 31 bit or should I restrict to 30 to prevent negative values?
Refer to ECMA5 that the bitwise operators and shift operators operate on 32-bit ints, so in that case, the max safe integer is 2^31-1, or 2147483647.
Here is one explanation.
The << operator is defined as working on signed 32-bit integers (converted from the native Number storage of double-precision float). So 1<<31 must result in a negative number.
The only JavaScript operator that works using unsigned 32-bit integers is >>>. You can exploit this to convert a signed-integer-in-Number you've been working on with the other bitwise operators to an unsigned-integer-in-Number:
(1<<31)>>>0
Most bitwise operations are specified as converting their operands to signed 32-bit integers. It is perfectly correct to use bit 31, but yes, you'll get negative values. Usually it doesn't matter if you're doing bitwise operations anyway, since all you (should) care about is the bit pattern, not the decimal value of the number.
If you do want a positive value back, you can convert it back with >>> 0, because >>> is specified to convert its operands to unsigned 32-bit integers.
console.log((0 | 1 << 31) >>> 0);
So I was playing around with shifts in the console, and the results have me stumped.
Input:
a = -1
a >>> 100
Output:
268435455
I looked on the Mozilla reference page about it, but it mentions nothing about the behavior of >>> when you shift by large amounts. I assumed that shifting all the bits to the right with zero-fill would result in a zero.
Is this a bug in Firefox or something?
It seems you can only shift by a maximum of 31.
From the site you linked in your post (MDN):
Shifts a in binary representation b (< 32) bits to the right,
discarding bits shifted off, and shifting in zeros from the left.
From the actual spec (Page 77)
Let shiftCount be the result of masking out all but the least
significant 5 bits of rnum, that is, compute rnum & 0x1F.
What's actually happening is when you shift by 100 it shifts by (100 & 0x1F) or 4.
-1 >>> 100 === -1 >>> 4
If you were to split it up into multiple shifts then it will work:
-1 >>> 25 >>> 25 >>> 25 >>> 25 === 0
Any bitwise operator on a Number in JavaScript will convert its operand to a 32 bit big-endian signed number.
This means that if the number is larger than what can be stored by 32 bits, it will be truncated. Big-endian means that number are stored in natural order when reading it from left to right, i.e. more significant numbers are stored first, so if the number is stored over one byte, its first byte is the more significant.
This means that -1's binary representation will be...
11111111 11111111 11111111 11111111
(This is -1 in two's complement. This is performed by calculating the number's value in binary, and then flipping each bit and adding one.)
When you shift by 100, you will find it only shifts by 4, leaving you with...
00001111 11111111 11111111 11111111
As you can see, the high bit is no longer set, so it's not negative, and it is in fact 268435455 (the number from your question).
Doing some tests with bitwise operations / shifting with JavaScript
0x80000000 >> 1 // returns -1073741824 (-0x40000000)
I would expect that to return 0x40000000 since
0x40000000 >> 1 // returns 0x20000000
0x20000000 >> 1 // returns 0x10000000
Its an arithmetic shift that's why the sign is preserved, to do a logical shift use >>>
0x80000000 >>> 1 // returns 1073741824 (0x40000000)
In Javascript, >> is the Signed Right Shift Operator. In §11.7.2 of the language specification it says:
Performs a sign-filling bitwise right shift operation on the left operand by the amount specified by the right operand.
Before the shifting is done, the left operand is converted to a signed 32-bit integer (step 5 of the algorithm given in the specification). In your case this conversion turns the given positive number into a negative value. After that, the actual shifting is done, preserving the negative sign of the value.