I have a img field whose value is base64 encoded. I checked the value in a decoder and it is bmp. when i put the value like it is not showing. I saw an example where the encoded string is trimmed with (base64 encoded string).substr(104). Image start rendering. I did not get a proper reason what is the reason behind trimming it.. Please suggest me with the exact reason. Thanks.
If it works after trimming first 104 characters, then if 1 character is 6 bits of information, then 104 characters == 624 bits == 78 bytes. If it works after trimming, then those first 78 bytes are redundant, they are probably some information, ex. a header.
It would be a lot harder if you'd have to remove number of bytes non-evenly divisible by 3. As 78 is evenly divisible by 3, it corresponds exactly to 104 characters.
Related
I have a class project where we're converting a bmp file to a buffer, manipulating the buffer, and writing it to a new file.
https://en.wikipedia.org/wiki/BMP_file_format
The first two bytes return the characters "BM" when converted to utf-8 using
buffer.toString('utf-8', 0, 2);
And sure enough the octets "42 4d" return "BM" when converted.
In class we were told that:
buffer.readUInt32LE(2) would return the size of the file (according to the file header).
In this case it's 4 octets:
"66 75 00 00"
Which using the readUInt32LE returns the number "30054" and sure enough when I checked the number of bytes, this was the case.
What I'm trying to figure out is the series of operations I need to make in order to for "66 75 00 00" to become "30054"
Any clarification would be helpful!
EDIT: I needed to calculated the hex values in the other direction! "75 66" and not "66 75". Thanks Guido :)
I am working with Northwind service in SAP Web IDE. Images in this service is stored in base64 String format:FRwvAAIAAAAN.....
I found out that I can't use these images in my app directly, using given base64 String value, because Northwind DB is old and made in MS Access and there are 78 redundant bytes which represent OLE header. So I would like to remove these 78 bytes from base64 String.
Can you please help me, using JavaScript language (I am new in this language). I hope for you experts. Here is what I have done:
I created function:
photo : function (value) {
var str = "";
for (var p in value) {
if (value.hasOwnProperty(p)) {
str += value[p];
}
}
..........
With this function I am taking base64 Sting as import parameter. I converted that import parameter from object to string.
So what should I do next? Create Array or something else? How can I remove 78 BYTES from String?
In base64 each character contains six bits of information, so four characters contains 24 bits of information, which is three bytes.
You are in luck. As 78 happens to be evenly divisble by three, the first 78 bytes corresponds exactly to the first 104 characters (78 bytes = 624 bits = 104 characters).
So, to remove the first 78 bytes of a base64 string, you remove the first 104 characters:
s = s.substr(104);
(If you hadn't been so lucky, you would have had to decode the entire string into bytes, remove the first 78 bytes, the encode the bytes into a string again.)
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
If I say 2 bytes can hold 510 characters of data will I be wrong? based on the fact that 1 byte can hold 0-255 max character
One byte is :
8 bits, each one can be either 0 or 1
something that can represent 256 distinct values
Two bytes are ..two bytes.
16 bits
something that can represent 65536 distinct values
There is no meaning in what a byte (or two bytes) is, if you don't know the encoding used, what each single one of the 256 (or 65536) values are supposed to be/mean.
If you're talking about Char, you can't either say it's one or two or fifty chars...
ASCII encoding holds 128 distinct characters (95 chars can be displayed while the remaining are control chars) ranging from code 0 to 127 (Byte value expressed in decimal literal)
Unicode encoding (v7) is a generic encoding. You have the UTF-8, the UTF-16 Little Endian or Big Endian, and the UTF-32 Little Endian or Big Endian.
UTF-8 requires either 1, 2, 3 or 4 bytes to represent one single character.
UTF-16 is a fixed-size character encoding : each char requires 2 bytes.
UTF-32 is also a fixed size character encoding that requires 4 bytes per character.
There are hundreds of different Encodings that can represent one character for each of the 256 unique values a single Byte can represent. Like ANSI.
So I tend to say, yes, you're wrong thinking two bytes can hold 510 characters of data, assuming you're using one of the above encoding or similar.
But again, a Byte is a Byte, not a Char !
Let's imagine a (new) custom encoding with specific parser and formatter where each bit [0 or 1] define the selection of one word/text/string stored in a dictionary, and following words/text/string selection depends on the previous selected word (previous bit value)
The purpose of such type of encoding is somewhat useless, but hey ! Because you used a dictionary, you can affirm one single byte can represent exactly 510 characters of data (or even more) because of the use of this specific encoding/decoding..!
Again, a byte is a byte, saying it holds one, two, zero or 510 characters doesn't mean anything if you don't define first what is the encoding used.
EDIT !
And while it's out of the scope of the question, compression is even more evil - and generally uses dictionary ;) - But compression are only effective from a certain amount of bytes....
A character is a graphical representation of a concept and may occupy an arbitrary number of bytes. For example, character "S" (capital letter 'S') occupies 1 byte whereas character 💋 (kissing lips) occupies 3 bytes.
I think your answer is wrong. byte is 1 character. a character in binary is a series of 8 on or offs or 0 or 1s. one of those is a bit and 8 bits make a byte so 1 byte is one character.so 2 bytes hold two characters.
It depends on the format of the string. 1 byte per character in ASCII and 2 bytes per character in Unicode. so 2 byte can hold only single Unicode character or 2 ASCII character.
The following code will explain my answer
MsgBox(System.Text.ASCIIEncoding.Unicode.GetByteCount("h")) '<--- displays 2
MsgBox(System.Text.ASCIIEncoding.ASCII.GetByteCount("h")) '<--- displays 1
I need to convert the following to a binary format (and later recoup) in the smallest amount of data possible.
my_arr = [
[128,32 ,22,23],
[104,53 ,21,25],
[150,55 ,79,23],
[104,101,23,8 ],
[57 ,117,13,21],
[37 ,135,21,20],
[81 ,132,23,6 ],
[81 ,138,7 ,8 ],
[97 ,138,7 ,8 ]...
the numbers don't exceed 399
If I use a 0 for each digit (8 0's in a row = 8) and a 1 as separator, the first line looks like this:
010010000000011000100110010011001000
This is really long for numbers like 99
If I pad each number to three digits and convert each in turn to actual binary the first line looks like this:
000100101000000000110010000000100010000000100011
This works out as 12 chars per number.
As the first char won't ever be a 4 or above I can save two digts by treating 0 as 00, 1 as 01, 2 as 10 and 3 as 11. Hence 10 chars per number
On the whole this reduces the size down to about 90% of the first option (on average) but is there a shorter way?
edit: yes as a string of 1's and 0's... and it doesn’t need to be shorter than the original integers... just the shortest possible way of writing it using only 2 symbols
If the values are evenly distributed between 0 and 399, then a pretty good encoding would be to take three values and encode them as a base 400 three-digit integer. I.e. val1 + 400*val2 + 400*400*val3. Then that integer will fit nicely in 26 bits. Four successive 26-bit values will fit in 13 bytes. Then you get an average of 13/12 bytes per value.
That's about as good as you're going to be able to do, unless the distribution of values is biased or if there is repetition or correlation, in which case you would be able to compress them more.
To deal with the details, you can use the number of bytes in the encoded sequence to determine the number of values, which may not be a multiple of three. If it is not a multiple of three, then there will be one or two values on the end, coded simply as nine bits each. Since it takes eight bits to go from 18 to 26 bits to add a value, there is no ambiguity in the count.
A good starting point would be to create constant-length blocks of ones and zeroes, which gives you easy to decode strings.
400 in binary is 110010000, which requires 9 characters to encode each number as its binary representation zero-padded to constant length.
encoding the first row:
var padTo9 = function( bin ){
while( bin.length<9 ){ bin = "0" + bin; }
return bin;
}
[128,32 ,22,23].map( function(i){ return padTo9( i.toString(2) ) }).join('');
/* result:
"010000000000100000000010110000010111"
*/
decoding
"010000000000100000000010110000010111".match(/[0-1]{9}/g).map( function(i){ return parseInt( i, 2 ) });
/* result:
[128, 32, 22, 23]
*/
I think the only way to get shorter string is using variable block length, which would require adding some control symbols to tell the decoder that following numbers are encoded in a specific number of characters. But these symbols have to be in >400 and still 9 characters long, so I think it wouldn't help given random distribution of data.
max 399:
2**9 is the smallest instance of (2**n)>=399, each number can be stored as 9 bits;
convert each to binary, and concat
I have a string exactly 53 characters long that contains a limited set of possible characters.
[A-Za-z0-9\.\-~_+]{53}
I need to reduce this to length 50 without loss of information and using the same set of characters.
I think it should be possible to compress most strings down to 50 length, but is it possible for all possible length 53 strings? We know that in the worst case 14 characters from the possible set will be unused. Can we use this information at all?
Thanks for reading.
If, as you stated, your output strings have to use the same set of characters as the input string, and if you don't know anything special about the requirements of the input string, then no, it's not possible to compress every possible 53-character string down to 50 characters. This is a simple application of the pigeonhole principle.
Your input strings can be represented as a 53-digit number in base 67, i.e., an integer from 0 to 6753 - 1 ≅ 6*1096.
You want to map those numbers to an integer from 0 to 6750 - 1 ≅ 2*1091.
So by the pigeonhole principle, you're guaranteed that 673 = 300,763 different inputs will map to each possible output -- which means that, when you go to decompress, you have no way to know which of those 300,763 originals you're supposed to map back to.
To make this work, you have to change your requirements. You could use a larger set of characters to encode the output (you could get it down to 50 characters if each one had 87 possible values, instead of the 67 in the input). Or you could identify redundancy in the input -- perhaps the first character can only be a '3' or a '5', the nineteenth and twentieth are a state abbreviation that can only have 62 different possible values, that sort of thing.
If you can't do either of those things, you'll have to use a compression algorithm, like Huffman coding, and accept the fact that some strings will be compressible (and get shorter) and others will not (and will get longer).
What you ask is not possible in the most general case, which can be proven very simply.
Say it was possible to encode an arbitrary 53 character string to 50 chars in the same set. Do that, then add three random characters to the encoded string. Then you have another arbitrary, 53 character string. How do you compress that?
So what you want can not be guaranteed to work for any possible data. However, it is possible that all your real data has low enough entropy that you can devise a scheme that will work.
In that case, you will probably want to do some variant of Huffman coding, which basically allocates variable-bit-length encodings for the characters in your set, using the shortest encodings for the most commonly used characters. You can analyze all your data to come up with a set of encodings. After Huffman coding, your string will be a (hopefully shorter) bitstream, which you encode to your character set at 6 bits per character. It may be short enough for all your real data.
A library-based encoding like Smaz (referenced in another answer) may work as well. Again, it is impossible to guarantee that it will work for all possible data.
One byte (character) can encode 256 values (0-255) but your set of valid characters uses only 67 values, which can be represented in 7 bits (alas, 6 bits gets you only 64) and none of your characters uses the high bit of the byte.
Given that, you can throw away the high bit and store only 7 bits, running the initial bits of the next character into the "spare" space of the first character. This would require only 47 bytes of space to store. (53 x 7 = 371 bits, 371 / 8 = 46.4 == 47)
This is not really considered compression, but rather a change in encoding.
For example "ABC" is 0x41 0x42 0x43
0x41 0x42 0x43 // hex values
0100 0001 0100 0010 0100 0011 // binary
100 0001 100 0010 100 0011 // drop high bit
// run it all together
100000110000101000011
// split as 8 bits (and pad to 8)
10000011 00001010 00011[000]
0x83 0x0A 0x18
As an example these 3 characters won't save any space, but your 53 characters will always come out as 47, guaranteed.
Note, however, that the output will not be in your original character set, if that is important to you.
The process becomes:
original-text --> encode --> store output-text (in database?)
retrieve --> decode --> original-text restored
If I remember correctly Huffman coding is going to be the most compact way to store the data. It has been too long since I used it to write the algorithm quickly, but the general idea is covered here, but if I remember correctly what you do is:
get the count for each character that is used
prioritize them based on how frequently they occurred
build a tree based off the prioritization
get the compressed bit representation of each character by traversing the tree (start at the root, left = 0 right = 1)
replace each character with the bits from the tree
Smaz is a simple compression library suitable for compressing very short strings.