Say I have a single-page application that uses a third party API for content. The app’s logic is in-browser only; there is no backend I can write to.
To allow deep-linking into the state of the app, I use pushState() to keep track of a few variables that determine the state of the app. (Note that Ubersicht’s public version doesn’t do this yet.)
Variables: repos, labels, milestones, username, show_open (bool), with_comments (bool), and without_comments (bool).
URL format: ?label=label_1,label_2,label_3&repos=repo_1….
Values: the usual suspects. Roughly, [a-zA-Z][a-zA-Z0-9_-], or any boolean indicator.
So far so good.
Now, since the query string can be a bit long and unwieldy and I would like to be able to pass around URLs like http://espy.github.io/ubersicht/?state=SOMOPAQUETOKENTHATLOSSLESSLYDECOMPRESSESINTOTHEORIGINALVALUES#hoodiehq, the shorter the better.
My first attempt was going to be using some zlib-like algorithm for this. Then #flipzagging pointed to antirez/smaz, which looks more suitable for short strings. (JavaScript version here.)
Since = and & are not specifically handled in the Javascript version (see line 9 of the main lib file), we might be able to tweak things a little there.
Furthermore, there is an option for encoding the values in a fixed table. With this option, the order of arguments is pre-defined and all we need to keep track of is the actual value. Example: turn a=hamster&b=cat into 7hamster3cat (length+chars) or hamster|cat (value + |), potentially before the smaz compression.
Is there anything else I should be looking for?
A working solution putting various bits of good (or so I think) ideas together
I did this for fun, mainly because it gave me an opportunity to implement an Huffman encoder in PHP and I could not find a satisfactory existing implementation.
However, this might save you some time if you plan to explore a similar path.
Burrows-Wheeler+move-to-front+Huffman transform
I'm not quite sure BWT would be best suited for your kind of input.
This is no regular text, so recurring patterns would probably not occur as often as in source code or plain English.
Besides, a dynamic Huffman code would have to be passed along with the encoded data which, for very short input strings, would harm the compression gain badly.
I might well be wrong, in which case I would gladly see someone prove me to be.
Anyway, I decided to try another approach.
General principle
1) define a structure for your URL parameters and strip the constant part
for instance, starting from:
repos=aaa,bbb,ccc&
labels=ddd,eee,fff&
milestones=ggg,hhh,iii&
username=kkk&
show_open=0&
show_closed=1&
show_commented=1&
show_uncommented=0
extract:
aaa,bbb,ccc|ddd,eee,fff|ggg,hhh,iii|kkk|0110
where , and | act as string and/or field terminators, while boolean values don't need any.
2) define a static repartition of symbols based on the expected average input and derive a static Huffman code
Since transmitting a dynamic table would take more space than your initial string, I think the only way to achhieve any compression at all is to have a static huffman table.
However, you can use the structure of your data to your advantage to compute reasonable probabilities.
You can start with the repartition of letters in English or other languages and throw in a certain percentage of numbers and other punctuation signs.
Testing with a dynamic Huffman coding, I saw compression rates of 30 to 50%.
This means with a static table you can expect maybe a .6 compression factor (reducing the lenght of your data by 1/3), not much more.
3) convert this binary Huffmann code into something an URI can handle
The 70 regular ASCII 7 bits chars in that list
!'()*-.0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz
would give you an expansion factor of about 30%, practically no better than a base64 encode.
A 30% expansion would ruin the gain from a static Huffman compression, so this is hardly an option!
However, since you control the encoding client and server side, you can use about anything that is not an URI reserved character.
An interesting possiblity would be to complete the above set up to 256 with whatever unicode glyphs, which would allow to encode your binary data with the same number of URI-compliant characters, thus replacing a painful and slow bunch of long integer divisions with a lightning fast table lookup.
Structure description
The codec is meant to be used both client and server side, so it is essential that server and clients share a common data structure definition.
Since the interface is likely to evolve, it seems wise to store a version number for upward compatibility.
The interface definition will use a very minimalistic description language, like so:
v 1 // version number (between 0 and 63)
a en // alphabet used (English)
o 10 // 10% of digits and other punctuation characters
f 1 // 1% of uncompressed "foreign" characters
s 15:3 repos // list of expeced 3 strings of average length 15
s 10:3 labels
s 8:3 milestones
s 10 username // single string of average length 10
b show_open // boolean value
b show_closed
b show_commented
b show_uncommented
Each language supported will have a frequency table for all its used letters
digits and other computerish symbols like -, . or _ will have a global frequency, regardless of languages
separators (, and |) frequencies will be computed according to the number of lists and fields present in the structure.
All other "foreign" characters will be escaped with a specific code and encoded as plain UTF-8.
Implementation
The bidirectional conversion path is as follows:
list of fields <-> UTF-8 data stream <-> huffman codes <-> URI
Here is the main codec
include ('class.huffman.codec.php');
class IRI_prm_codec
{
// available characters for IRI translation
static private $translator = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöùúûüýþÿĀāĂ㥹ĆćĈĉĊċČčĎďĐđĒēĔĕĖėĘęĚěĜĝĞğĠġĢģĤĥĦħĨĩĪīĬĭĮįİıIJijĴĵĶķĸĹĺĻļĽľĿŀŁłŃńŅņŇňʼnŊŋŌōŎŏŐőŒœŔŕŖŗŘřŚśŜŝŞşŠšŢţŤťŦŧŨũŪūŬŭŮůŰűŲųŴŵŶŷŸŹźŻżŽžſƀƁƂƃƄƅ";
const VERSION_LEN = 6; // version number between 0 and 63
// ========================================================================
// constructs an encoder
// ========================================================================
public function __construct ($config)
{
$num_record_terminators = 0;
$num_record_separators = 0;
$num_text_sym = 0;
// parse config file
$lines = file($config, FILE_IGNORE_NEW_LINES|FILE_SKIP_EMPTY_LINES);
foreach ($lines as $line)
{
list ($code, $val) = preg_split('/\s+/', $line, 2);
switch ($code)
{
case 'v': $this->version = intval($val); break;
case 'a': $alphabet = $val; break;
case 'o': $percent_others = $val; break;
case 'f': $percent_foreign = $val; break;
case 'b':
$this->type[$val] = 'b';
break;
case 's':
list ($val, $field) = preg_split('/\s+/u', $val, 2);
#list ($len,$num) = explode (':', $val);
if (!$num) $num=1;
$this->type[$field] = 's';
$num_record_terminators++;
$num_record_separators+=$num-1;
$num_text_sym += $num*$len;
break;
default: throw new Exception ("Invalid config parameter $code");
}
}
// compute symbol frequencies
$total = $num_record_terminators + $num_record_separators + $num_text_sym + 1;
$num_chars = $num_text_sym * (100-($percent_others+$percent_foreign))/100;
$num_sym = $num_text_sym * $percent_others/100;
$num_foreign = $num_text_sym * $percent_foreign/100;
$this->get_frequencies ($alphabet, $num_chars/$total);
$this->set_frequencies (" .-_0123456789", $num_sym/$total);
$this->set_frequencies ("|", $num_record_terminators/$total);
$this->set_frequencies (",", $num_record_separators/$total);
$this->set_frequencies ("\1", $num_foreign/$total);
$this->set_frequencies ("\0", 1/$total);
// create Huffman codec
$this->huffman = new Huffman_codec();
$this->huffman->make_code ($this->frequency);
}
// ------------------------------------------------------------------------
// grab letter frequencies for a given language
// ------------------------------------------------------------------------
private function get_frequencies ($lang, $coef)
{
$coef /= 100;
$frequs = file("$lang.dat", FILE_IGNORE_NEW_LINES|FILE_SKIP_EMPTY_LINES);
foreach ($frequs as $line)
{
$vals = explode (" ", $line);
$this->frequency[$vals[0]] = floatval ($vals[1]) * $coef;
}
}
// ------------------------------------------------------------------------
// set a given frequency for a group of symbols
// ------------------------------------------------------------------------
private function set_frequencies ($symbols, $coef)
{
$coef /= strlen ($symbols);
for ($i = 0 ; $i != strlen($symbols) ; $i++) $this->frequency[$symbols[$i]] = $coef;
}
// ========================================================================
// encodes a parameter block
// ========================================================================
public function encode($input)
{
// get back input values
$bools = '';
foreach (get_object_vars($input) as $prop => $val)
{
if (!isset ($this->type[$prop])) throw new Exception ("unknown property $prop");
switch ($this->type[$prop])
{
case 'b': $bools .= $val ? '1' : '0'; break;
case 's': $strings[] = $val; break;
default: throw new Exception ("Uh oh... type ".$this->type[$prop]." not handled ?!?");
}
}
// set version number and boolean values in front
$prefix = sprintf ("%0".self::VERSION_LEN."b$bools", $this->version);
// pass strings to our Huffman encoder
$strings = implode ("|", $strings);
$huff = $this->huffman->encode ($strings, $prefix, "UTF-8");
// translate into IRI characters
mb_internal_encoding("UTF-8");
$res = '';
for ($i = 0 ; $i != strlen($huff) ; $i++) $res .= mb_substr (self::$translator, ord($huff[$i]), 1);
// done
return $res;
}
// ========================================================================
// decodes an IRI string into a lambda object
// ========================================================================
public function decode($input)
{
// convert IRI characters to binary
mb_internal_encoding("UTF-8");
$raw = '';
$len = mb_strlen ($input);
for ($i = 0 ; $i != $len ; $i++)
{
$c = mb_substr ($input, 0, 1);
$input = mb_substr ($input, 1);
$raw .= chr(mb_strpos (self::$translator, $c));
}
$this->bin = '';
// check version
$version = $this->read_bits ($raw, self::VERSION_LEN);
if ($version != $this->version) throw new Exception ("Version mismatch: expected {$this->version}, found $version");
// read booleans
foreach ($this->type as $field => $type)
if ($type == 'b')
$res->$field = $this->read_bits ($raw, 1) != 0;
// decode strings
$strings = explode ('|', $this->huffman->decode ($raw, $this->bin));
$i = 0;
foreach ($this->type as $field => $type)
if ($type == 's')
$res->$field = $strings[$i++];
// done
return $res;
}
// ------------------------------------------------------------------------
// reads raw bit blocks from a binary string
// ------------------------------------------------------------------------
private function read_bits (&$raw, $len)
{
while (strlen($this->bin) < $len)
{
if ($raw == '') throw new Exception ("premature end of input");
$this->bin .= sprintf ("%08b", ord($raw[0]));
$raw = substr($raw, 1);
}
$res = bindec (substr($this->bin, 0, $len));
$this->bin = substr ($this->bin, $len);
return $res;
}
}
The underlying Huffman codec
include ('class.huffman.dict.php');
class Huffman_codec
{
public $dict = null;
// ========================================================================
// encodes a string in a given string encoding (default: UTF-8)
// ========================================================================
public function encode($input, $prefix='', $encoding="UTF-8")
{
mb_internal_encoding($encoding);
$bin = $prefix;
$res = '';
$input .= "\0";
$len = mb_strlen ($input);
while ($len--)
{
// get next input character
$c = mb_substr ($input, 0, 1);
$input = substr($input, strlen($c)); // avoid playing Schlemiel the painter
// check for foreign characters
if (isset($this->dict->code[$c]))
{
// output huffman code
$bin .= $this->dict->code[$c];
}
else // foreign character
{
// escape sequence
$lc = strlen($c);
$bin .= $this->dict->code["\1"]
. sprintf("%02b", $lc-1); // character length (1 to 4)
// output plain character
for ($i=0 ; $i != $lc ; $i++) $bin .= sprintf("%08b", ord($c[$i]));
}
// convert code to binary
while (strlen($bin) >= 8)
{
$res .= chr(bindec(substr ($bin, 0, 8)));
$bin = substr($bin, 8);
}
}
// output last byte if needed
if (strlen($bin) > 0)
{
$bin .= str_repeat ('0', 8-strlen($bin));
$res .= chr(bindec($bin));
}
// done
return $res;
}
// ========================================================================
// decodes a string (will be in the string encoding used during encoding)
// ========================================================================
public function decode($input, $prefix='')
{
$bin = $prefix;
$res = '';
$len = strlen($input);
for ($i=0 ;;)
{
$c = $this->dict->symbol($bin);
switch ((string)$c)
{
case "\0": // end of input
break 2;
case "\1": // plain character
// get char byte size
if (strlen($bin) < 2)
{
if ($i == $len) throw new Exception ("incomplete escape sequence");
$bin .= sprintf ("%08b", ord($input[$i++]));
}
$lc = 1 + bindec(substr($bin,0,2));
$bin = substr($bin,2);
// get char bytes
while ($lc--)
{
if ($i == $len) throw new Exception ("incomplete escape sequence");
$bin .= sprintf ("%08b", ord($input[$i++]));
$res .= chr(bindec(substr($bin, 0, 8)));
$bin = substr ($bin, 8);
}
break;
case null: // not enough bits do decode further
// get more input
if ($i == $len) throw new Exception ("no end of input mark found");
$bin .= sprintf ("%08b", ord($input[$i++]));
break;
default: // huffman encoded
$res .= $c;
break;
}
}
if (bindec ($bin) != 0) throw new Exception ("trailing bits in input");
return $res;
}
// ========================================================================
// builds a huffman code from an input string or frequency table
// ========================================================================
public function make_code ($input, $encoding="UTF-8")
{
if (is_string ($input))
{
// make dynamic table from the input message
mb_internal_encoding($encoding);
$frequency = array();
while ($input != '')
{
$c = mb_substr ($input, 0, 1);
$input = mb_substr ($input, 1);
if (isset ($frequency[$c])) $frequency[$c]++; else $frequency[$c]=1;
}
$this->dict = new Huffman_dict ($frequency);
}
else // assume $input is an array of symbol-indexed frequencies
{
$this->dict = new Huffman_dict ($input);
}
}
}
And the huffman dictionary
class Huffman_dict
{
public $code = array();
// ========================================================================
// constructs a dictionnary from an array of frequencies indexed by symbols
// ========================================================================
public function __construct ($frequency = array())
{
// add terminator and escape symbols
if (!isset ($frequency["\0"])) $frequency["\0"] = 1e-100;
if (!isset ($frequency["\1"])) $frequency["\1"] = 1e-100;
// sort symbols by increasing frequencies
asort ($frequency);
// create an initial array of (frequency, symbol) pairs
foreach ($frequency as $symbol => $frequence) $occurences[] = array ($frequence, $symbol);
while (count($occurences) > 1)
{
$leaf1 = array_shift($occurences);
$leaf2 = array_shift($occurences);
$occurences[] = array($leaf1[0] + $leaf2[0], array($leaf1, $leaf2));
sort($occurences);
}
$this->tree = $this->build($occurences[0], '');
}
// -----------------------------------------------------------
// recursive build of lookup tree and symbol[code] table
// -----------------------------------------------------------
private function build ($node, $prefix)
{
if (is_array($node[1]))
{
return array (
'0' => $this->build ($node[1][0], $prefix.'0'),
'1' => $this->build ($node[1][1], $prefix.'1'));
}
else
{
$this->code[$node[1]] = $prefix;
return $node[1];
}
}
// ===========================================================
// extracts a symbol from a code stream
// if found : updates code stream and returns symbol
// if not found : returns null and leave stream intact
// ===========================================================
public function symbol(&$code_stream)
{
list ($symbol, $code) = $this->get_symbol ($this->tree, $code_stream);
if ($symbol !== null) $code_stream = $code;
return $symbol;
}
// -----------------------------------------------------------
// recursive search for a symbol from an huffman code
// -----------------------------------------------------------
private function get_symbol ($node, $code)
{
if (is_array($node))
{
if ($code == '') return null;
return $this->get_symbol ($node[$code[0]], substr($code, 1));
}
return array ($node, $code);
}
}
Example
include ('class.iriprm.codec.php');
$iri = new IRI_prm_codec ("config.txt");
foreach (array (
'repos' => "discussion,documentation,hoodie-cli",
'labels' => "enhancement,release-0.3.0,starter",
'milestones' => "1.0.0,1.1.0,v0.7",
'username' => "mklappstuhl",
'show_open' => false,
'show_closed' => true,
'show_commented' => true,
'show_uncommented' => false
) as $prop => $val) $iri_prm->$prop = $val;
$encoded = $iri->encode ($iri_prm);
echo "encoded as $encoded\n";
$decoded = $iri->decode ($encoded);
var_dump($decoded);
output:
encoded as 5ĶůťÊĕCOĔƀŪļŤłmĄZEÇŽÉįóšüÿjħũÅìÇēOĪäŖÏŅíŻÉĒQmìFOyäŖĞqæŠŹōÍĘÆŤŅËĦ
object(stdClass)#7 (8) {
["show_open"]=>
bool(false)
["show_closed"]=>
bool(true)
["show_commented"]=>
bool(true)
["show_uncommented"]=>
bool(false)
["repos"]=>
string(35) "discussion,documentation,hoodie-cli"
["labels"]=>
string(33) "enhancement,release-0.3.0,starter"
["milestones"]=>
string(16) "1.0.0,1.1.0,v0.7"
["username"]=>
string(11) "mklappstuhl"
}
In that example, the input got packed into 64 unicode characters, for an input length of about 100, yielding a 1/3 reduction.
An equivalent string:
discussion,documentation,hoodie-cli|enhancement,release-0.3.0,starter|
1.0.0,1.1.0,v0.7|mklappstuhl|0110
Would be compressed by a dynamic Huffman table to 59 characters. Not much of a difference.
No doubt smart data reordering would reduce that, but then you would need to pass the dynamic table along...
Chinese to the rescue?
Drawing on ttepasse's idea, one could take advantage of the huge number of Asian characters to find a range of 0x4000 (12 bits) contiguous values, to code 3 bytes into 2 CJK characters, like so:
// translate into IRI characters
$res = '';
$len = strlen ($huff);
for ($i = 0 ; $i != $len ; $i++)
{
$byte = ord($huff[$i]);
$quartet[2*$i ] = $byte >> 4;
$quartet[2*$i+1] = $byte &0xF;
}
$len *= 2;
while ($len%3 != 0) $quartet[$len++] = 0;
$len /= 3;
for ($i = 0 ; $i != $len ; $i++)
{
$utf16 = 0x4E00 // CJK page base, enough range for 2**12 (0x4000) values
+ ($quartet[3*$i+0] << 8)
+ ($quartet[3*$i+1] << 4)
+ ($quartet[3*$i+2] << 0);
$c = chr ($utf16 >> 8) . chr ($utf16 & 0xFF);
$res .= $c;
}
$res = mb_convert_encoding ($res, "UTF-8", "UTF-16");
and back:
// convert IRI characters to binary
$input = mb_convert_encoding ($input, "UTF-16", "UTF-8");
$len = strlen ($input)/2;
for ($i = 0 ; $i != $len ; $i++)
{
$val = (ord($input[2*$i ]) << 8) + ord ($input[2*$i+1]) - 0x4E00;
$quartet[3*$i+0] = ($val >> 8) &0xF;
$quartet[3*$i+1] = ($val >> 4) &0xF;
$quartet[3*$i+2] = ($val >> 0) &0xF;
}
$len *= 3;
while ($len %2) $quartet[$len++] = 0;
$len /= 2;
$raw = '';
for ($i = 0 ; $i != $len ; $i++)
{
$raw .= chr (($quartet[2*$i+0] << 4) + $quartet[2*$i+1]);
}
The previous output of 64 Latin chars
5ĶůťÊĕCOĔƀŪļŤłmĄZEÇŽÉįóšüÿjħũÅìÇēOĪäŖÏŅíŻÉĒQmìFOyäŖĞqæŠŹōÍĘÆŤŅËĦ
would "shrink" to 42 Asian characters:
乙堽孴峴勀垧壩坸冫嚘佰嫚凲咩俇噱刵巋娜奾埵峼圔奌夑啝啯嶼勲婒婅凋凋伓傊厷侖咥匄冯塱僌
However, as you can see, the sheer bulk of your average ideogram makes the string actually longer (pixel-wise), so even if the idea was promising, the outcome is rather disappointing.
Picking thinner glyphs
On the other hand, you can try to pick "thin" characters as a base for URI encoding. For instance:
█ᑊᵄ′ӏᶟⱦᵋᵎiïᵃᶾ᛬ţᶫꞌᶩ᠇܂اlᶨᶾᛁ⁚ᵉʇȋʇίן᠙ۃῗᥣᵋĭꞌ៲ᛧ༚ƫܙ۔ˀȷˁʇʹĭ∕ٱ;łᶥյ;ᴶ⁚ĩi⁄ʈ█
instead of
█5ĶůťÊĕCOĔƀŪļŤłmĄZEÇŽÉįóšüÿjħũÅìÇēOĪäŖÏŅíŻÉĒQmìFOyäŖĞqæŠŹōÍĘÆŤŅËĦ█
That will shrink the length by half with proportional fonts, including in a browser address bar.
My best candidate set of 256 "thin" glyphs so far:
᠊།ᑊʲ་༌ᵎᵢᶤᶩᶪᶦᶧˡ ⁄∕เ'Ꞌꞌ꡶ᶥᵗᶵᶨ|¦ǀᴵ ᐧᶠᶡ༴ˢᶳ⁏ᶴʳʴʵ։᛬⍮ʹ′ ⁚⁝ᵣ⍘༔⍿ᠵᥣᵋᵌᶟᴶǂˀˁˤ༑,. ∙Ɩ៲᠙ᵉᵊᵓᶜᶝₑₔյⵏⵑ༝༎՛ᵞᵧᚽᛁᛂᛌᛍᛙᛧᶢᶾ৷⍳ɩΐίιϊᵼἰἱἲἳἴἵἶἷὶίῐῑῒΐῖῗ⎰⎱᠆ᶿ՝ᵟᶫᵃᵄᶻᶼₐ∫ª౹᠔/:;\ijltìíîïĩīĭįıĵĺļłţŧſƚƫƭǐǰȉȋțȴȷɉɨɪɫɬɭʇʈʝːˑ˸;·ϳіїјӏ᠇ᴉᵵᵻᶅᶖḭḯḷḹḻḽṫṭṯṱẗẛỉị⁞⎺⎻⎼⎽ⱡⱦ꞉༈ǁ‖༅༚ᵑᵝᵡᵦᵪา᠑⫶ᶞᚁᚆᚋᚐᚕᵒᵔᵕᶱₒⵗˣₓᶹๅʶˠ᛫ᵛᵥᶺᴊ
Conclusion
This implementation should be ported to JavaScript to allow client-server exchange.
You should also provide a way to share the structure and Huffman codes with the clients.
It is not difficult and rather fun to do, but that means even more work :).
The Huffman gain in term of characters is around 30%.
Of course these characters are multibyte for the most part, but if you aim for the shortest URI it does not matter.
Except for the booleans that can easily be packed to 1 bit, those pesky strings seem rather reluctant to be compressed.
It might be possible to better tune the frequencies, but I doubt you will get above 50% compression rate.
On the other hand, picking thin glyphs does actually more to shrink the string.
So all in all the combination of both might indeed achieve something, though it's a lot of work for a modest result.
Just as you yourself propose, I would first get rid of all the characters that are not carrying any information, because they are part of the "format".
E.g. turn "labels=open,ssl,cypher&repository=275643&username=ryanbrg&milestones=&with_comment=yes" to
"open,ssl,cyper|275643|ryanbrg||yes".
Then use a Huffmann encoding with a fixed probability vector (resulting in a fixed mapping from characters to variable length bitstrings - with the most probable characters mapped to shorter bitstrings and less probable characters mapped to longer bitstrings).
You could even use different probability vectors for the different parameters. For example in the parameter "labels" the alpha characters will have high probability, but in the "repository" parameter the numeric characters will have the highest probability. If you do this, you should consider the separator "|" a part of the preceeding parameter.
And finally turn the long bitstring (which is the concatenation all the bitstrings to which the characters were mapped) into something you can put into an URL by base64url encoding it.
If you could send me a set of representative parameter lists, I could run them through a Huffmann coder to see how well they compress.
The probability vector (or equivalently the mapping from characters to bitstrings) should be encoded as constant arrays into the Javascript function that is sent to the browser.
Of course you could go even further and - for example - try to get a list of possible lables with their probabilities. Then you could map entire lables to bitstrings with a Huffmann encoding. This will give you better compression, but you will have extra work for those labels that are new (e.g. falling back to the single character encoding), and of course the mapping (which - as mentioned above - is a constant array in the Javascript function) will be much larger.
Why not using protocol-buffers?
Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. You can even update your data structure without breaking deployed programs that are compiled against the "old" format.
ProtoBuf.js converts objects to protocol buffer messages and vice vera.
The following object converts to: CgFhCgFiCgFjEgFkEgFlEgFmGgFnGgFoGgFpIgNqZ2I=
{
repos : ['a', 'b', 'c'],
labels: ['d', 'e', 'f'],
milestones : ['g', 'h', 'i'],
username : 'jgb'
}
Example
The following example is built using require.js. Give it a try on this jsfiddle.
require.config({
paths : {
'Math/Long' : '//rawgithub.com/dcodeIO/Long.js/master/Long.min',
'ByteBuffer' : '//rawgithub.com/dcodeIO/ByteBuffer.js/master/ByteBuffer.min',
'ProtoBuf' : '//rawgithub.com/dcodeIO/ProtoBuf.js/master/ProtoBuf.min'
}
})
require(['message'], function(message) {
var data = {
repos : ['a', 'b', 'c'],
labels: ['d', 'e', 'f'],
milestones : ['g', 'h', 'i'],
username : 'jgb'
}
var request = new message.arguments(data);
// Convert request data to base64
var base64String = request.toBase64();
console.log(base64String);
// Convert base64 back
var decodedRequest = message.arguments.decode64(base64String);
console.log(decodedRequest);
});
// Protobuf message definition
// Message definition could also be stored in a .proto definition file
// See: https://github.com/dcodeIO/ProtoBuf.js/wiki
define('message', ['ProtoBuf'], function(ProtoBuf) {
var proto = {
package : 'message',
messages : [
{
name : 'arguments',
fields : [
{
rule : 'repeated',
type : 'string',
name : 'repos',
id : 1
},
{
rule : 'repeated',
type : 'string',
name : 'labels',
id : 2
},
{
rule : 'repeated',
type : 'string',
name : 'milestones',
id : 3
},
{
rule : 'required',
type : 'string',
name : 'username',
id : 4
},
{
rule : 'optional',
type : 'bool',
name : 'with_comments',
id : 5
},
{
rule : 'optional',
type : 'bool',
name : 'without_comments',
id : 6
}
],
}
]
};
return ProtoBuf.loadJson(proto).build('message')
});
I have a cunning plan! (And a drink of gin tonic)
You doesn't seem to care about the length of the bytestream but of the length of the resulting glyphs, e.g. what the string which is displayed to the user.
Browser are pretty good in converting an IRI to the underlying [URI][2] while still displaying the IRI in the address bar. IRIs have a greater repertoire of possible characters while your set of possible chars is rather limited.
That means you can encode bigrams of your chars (aa, ab, ac, …, zz & special chars) into one char of the full unicode spectrum. Say you've got 80 possible ASCII chars: the number of possible combinations of two chars is 6400. Which are easy findable in Unicodes assigned chars, e.g. in the han unified CJK spectrum:
aa → 一
ab → 丁
ac → 丂
ad → 七
…
I picked CJK because this is only (slighty) reasonable if the target chars are assigned in unicode and have assigned glyphs on the major browser and operating systems. For that reason the private use area is out and the more efficient version using trigrams (whose possible combinations could use all of Unicodes 1114112 possible code points) are out.
To recap: the underlying bytes are still there and – given UTF-8 encoding – possible even longer, but the string of displayed characters the user sees and copies is 50% shorter.
Ok, Ok, reasons, why this solution is insane:
IRIs are not perfect. A lot of lesser tools than modern browser have their problems.
The algorithm needs obviously a lot of more work. You'll need a function which maps the bigrams to the target chars and back. And it should preferable work arithmetically to avoid big hash tables in memory.
The target chars should be checked if they are assigned and if they are simple chars and not fancy unicodian things like combining chars or stuff that got lost somewhere in Unicode normalization. Also if the target area is an continuous span of assigned chars with glyphs.
Browser are sometimes wary of IRIs. For good reason, given the IDN homograph attacks. Are they OK with all these non-ASCII-chars in their address bar?
And the biggest: people are notoriously bad at remembering characters in scripts they don't know. They are even worse at trying to (re)-type these chars. And copy'n'paste can go wrong in many different clicks. There is a reason URL shorteners use Base64 and even smaller alphabets.
… speaking of which: That would be my solution. Offloading the work of shortening links either to the user or integrating goo.gl or bit.ly via their APIs.
Small tip: Both parseInt and Number#toString support radix arguments. Try using a radix of 36 to encode numbers (or indexes into lists) in URLs.
Update: I released an NPM package with some more optimizations, see https://www.npmjs.com/package/#yaska-eu/jsurl2
Some more tips:
Base64 encodes with a..zA..Z0..9+/=, and un-encoded URI characters are a..zA..Z0..9-_.~. So Base64 results only need to swap +/= for -_. and it won't expand URIs.
You could keep an array of key names, so that objects could be represented with the first character being the offset in the array, e.g. {foo:3,bar:{g:'hi'}} becomes a3,b{c'hi'} given key array ['foo','bar','g']
Interesting libraries:
JSUrl specifically encodes JSON so it can be put in a URL without changes, even though it uses more characters than specified in the RFC. {"name":"John Doe","age":42,"children":["Mary","Bill"]} becomes ~(name~'John*20Doe~age~42~children~(~'Mary~'Bill)) and with a key dictionary ['name','age','children'] that could be ~(0~'John*20Doe~1~42~2~(~'Mary~'Bill)), thus going from 101 bytes URI encoded to 38.
Small footprint, fast, reasonable compression.
lz-string uses an LZW-based algorithm to compress strings to UTF16 for storing in localStorage. It also has a compressToEncodedURIComponent() function to produce URI-safe output.
Still only a few KB of code, pretty fast, good/great compression.
So basically I'd recommend picking one of these two libraries and consider the problem solved.
There are two main aspects to the problem: encoding and compression.
General purpose compression doesn’t seem to work well on small strings. As browsers don’t provide any API to compress strings, you also need to load the source, which can be huge.
But a lot of characters can be saved by using an efficient encoding. I have written a library named μ to handle the encoding and decoding part.
The idea is to specify as much as information available about the structure and domain of the URL parameters as a specification. This specification can be then used to drive the encoding and decoding. For example:
booleans can be encoded using just one bit;
integers can be converted to base64 (thereby reducing the number of characters required);
object keys need not be encoded (because they can be inferred from the specification);
enums can be encoded using log2(numberOfAllowedValues) bits.
Perhaps you can find a url shortener with a jsonp API, that way you could make all the URLs really short automatically.
http://yourls.org/ even has jsonp support.
It looks like the Github APIs have numeric IDs for many things (looks like repos and users have them, but labels don't) under the covers. It might be possible to use those numbers instead of names wherever advantageous. You then have to figure out how to best encode those in something that'll survive in a query string, e.g. something like base64(url).
For example, your hoodie.js repository has ID 4780572.
Packing that into a big-endian unsigned int (as many bytes as we need) gets us \x00H\xf2\x1c.
We'll just toss the leading zero, we can always restore that later, now we have H\xf2\x1c.
Encode as URL-safe base64, and you have SPIc (toss any padding you might get).
Going from hoodiehq/hoodie.js to SPIc seems like a good-sized win!
More generally, if you're willing to invest the time, you can try to exploit a bunch of redudancies in your query strings. Other ideas are along the lines of packing the two boolean params into a single character, possibly along with other state (like what fields are included). If you use base64-encoding (which seems the best option here due to the URL-safe version -- I looked at base85, but it has a bunch of characters that won't survive in a URL), that gets you 6 bits of entropy per character... there's a lot you can do with that.
To add to Thomas Fuchs' note, yes, if there's some kind of inherent, immutable ordering in some of things you're encoding, than that would obviously also help. However, that seems hard for both the labels and the milestones.
Maybe any simple JS minifier will help you. You'll need only to integrate it on serialization and deserialization points only. I think it'd be the easiest solution.
Why not use a third party link shortener?
(I am assuming you don't have a problem with URI length limits since you mentioned this is an existing application.)
It looks like you're writing a Greasemonkey script or thereabouts, so perhaps you have access to GM_xmlhttpRequest(), which would allow use of a third party link shortener.
Otherwise, you'd need to use XMLHttpRequest() and host your own link shortening service on the same server to avoid crossing the same-origin policy boundary. A quick online search for hosting your own shorteners supplied me with a list of 7 free/open source PHP link shortener scripts and one more on GitHub, though the question likely excludes this kind of approach since "The app’s logic is in-browser only, and there is no backend I can write to."
You can see example code implementing this kind of thing in the URL Shortener UserScript (for Greasemonkey), which pops up a shortened version of the current page's URL when you press SHIFT+T.
Of course, shorteners will redirect users to the long form URL, but this would be a problem in any non-server-side solution. At least a shortener can theoretically proxy (like Apache's RewriteRule with [P]) or use a <frame> tag.
Short
Use a URL packing scheme such as my own, starting only from the params section of your URL.
Longer
As other's here have pointed out, typical compression systems don't work for short strings. But, it's important to recognise that URLs and Params are a serialization format of a data model: a text human-readable format with specific sections - we know that the scheme is first, the host is found directly after, the port is implied but can be overridden, etc...
With the underlying conceptual data model, one can serialize with a more bit-efficient serialization scheme. In fact, I have created such a serialization myself which archives around 50% compression: see http://blog.alivate.com.au/packed-url/
Conceptually, my scheme was written with the conceptual data model in mind, it doesn't deserialize the URL into that conceptual model as a distinct step. However, that's possible, and that formal approach might yield greater efficiencies, where the bits don't need to be in the same order as what a string URL might be.
Related
I need to create an internet-less password reset feature. My hardware uses a web (JS/HTML) interface and a PHP back-end. I've seen this in the wild before: the user gets a short key which they read to the IT person, and the IT person generates a short private response key. The user then uses this key and is able to get to a password change screen.
Yes, I have a "security questions" feature already. This is going to be for last-ditch recovery, like if your last Admin user dies or something. Also, it cannot be time-sensitive like standard keyfob RSA. Some of these units are literally on top of mountains, so a person might have a considerable lag time between generating the public key, receiving the private one, and setting it on their device.
I've considered just taking something like the unit's serial and the date and running them through some sort of encryption, but I wonder if you guys have some simpler way? Hell, I could encrypt it the same way I do the passwords, but the strings that generates can be... unwieldy. Ideally this would be maybe a 10-12 character string.
I found a pretty slick way to do it. I took a set of data ($serial, hard-coded $seed, $date, and $type for account type) md5'd it, and encrypted that with openssl using an encrypt function I found on another answer, then did sort of the first half of a checksum to condense the long string to something 1/3 the length. You just run genkey($serial,$seed,$date,$type) to get a nice short string, such as "URMYVAFYNE". The person on the other end does the same thing, then compares the strings to see if they match. Then they generate a second string using the first one as the $seed, and the client enters that to have it verified... you get the idea.
function genkey($serial,$seed,$date,$type) {
$fullstring = $serial.$seed.$date.$type;
$md5 = md5($fullstring);
$encrypted = encrypt_decrypt('encrypt',$md5);
return condense(condense($encrypted));
}
function encrypt_decrypt($action, $string)
{
$output = false;
$encrypt_method = "AES-256-CBC";
$secret_key = '[REDACTED]'; //32 byte
$secret_iv = '[REDACTED]'; //16 byte
// hash
$key = hash('sha256', $secret_key);
// iv - encrypt method AES-256-CBC expects 16 bytes
$iv = substr(hash('sha256', $secret_iv), 0, 16);
if ( $action == 'encrypt' ) {
$output = openssl_encrypt($string, $encrypt_method, $key, 0, $iv);
$output = base64_encode($output);
} else if( $action == 'decrypt' ) {
$output = openssl_decrypt(base64_decode($string), $encrypt_method, $key, 0, $iv);
}
return $output;
}
function condense($str) {
$arr = str_split($str, 3);
$retstr = '';
for($i=0;$i<count($arr);$i++) {
$sum=WordSum($arr[$i]);
$letternum = $sum%26 + 65; //ascii code for capital letter
$letter = chr($letternum);
$retstr .= $letter;
}
return $retstr;
}
I'm building a table who have a button. When the user click this button the stat of row changes, in the table this represents a field with 0 or 1. Fine i'd made the button but i get an error in some rows, the error is Uncaught SyntaxError: missing ) after argument list, and i get very confused because the code works on like 80% of the time.
The problem wasn't with the database because i'd already searched for some field with problems.
function enviaAssociado(CTITULAR) {
if (CTITULAR) {
$("#CTITULAR").remove();
$.ajax({
url: 'php_action/enviaAssociado.php',
type: 'post',
data: {
CTITULAR: CTITULAR
},
success: function() {
manageTable.ajax.reload();
//$("#CTITULAR").val(response.CTITULAR);
}
});
}
manageTable.ajax.reload();
}
PHP
`
$CTITULAR = $_POST['CTITULAR'];
$sql = "UPDATE importsmiles4e.tb_conv
SET STAT = CASE
WHEN STAT = 1 THEN 0
WHEN STAT = 0 THEN 1
ELSE STAT
END
WHERE substring(CTITULAR,2,6) = '$CTITULAR'
";
$query = $connect->query($sql);
$CTITULAR = $_POST['CTITULAR'];
$sql = "UPDATE importsmiles4e.tb_conv3
SET STAT = CASE
WHEN STAT = 1 THEN 0
WHEN STAT = 0 THEN 1
ELSE STAT
END
WHERE substring(CTITULAR,2,6) = '$CTITULAR'";
$query = $connect->query($sql);
// close the database connection
$connect->close();
echo json_encode($sql);
`
The Table
while ($row = $query->fetch_assoc()) {
$active = '';
if($row['EMAIL'] != '') {
$active = '<label class="label label-success">Cadastrado</label>';
} else {
$active = '<label class="label label-danger">Cadastrar</label>';
}
$botao = '<a type="button" class="btn btn-default" onclick="enviaAssociado('.$row['CTITULAR'].') ">Alterar</a>';
$status = '';
if($row['STAT'] == '0'){
$status ='<label class="label label-warning">Não</label>';
}else{
$status ='<label class="label label-success">Sim</label>';
}
$output['data'][] = array(
$row['NOME'],
$row['CPF'],
$row['CEPRES'],
$row['NROPROPOSTA'],
$row['DTADMISSAO'],
$row['DEPENDENTES'],
$row['VLSMENS'],
$status,
$botao,
);
https://i.stack.imgur.com/KnD7w.png => error Log
https://i.stack.imgur.com/MhsDh.png => Return when error
The return when success is the same the only change is the value key and i've already check if the values are broken.
The problem is buried away in one of your screenshots:
enviaAssociado(0516542.00)
That leading 0 combined with the fact that there are no 8 or 9 digits in it (yes, really) makes that a legacy octal integer literal in loose mode. Since it's an integer literal, it can't have a decimal point, so as of the . it's a syntax error.
If that's meant to be a number, have your PHP code strip any leading 0s from the number. If it's meant to be a string, have your PHP code output it in quotes.
If you output data to JavaScript code from PHP, you're usually best off doing so via json_encode. If you're doing it within an attribute (which is not best practice), you also want htmlspecialchars. So:
$botao = '<a type="button" class="btn btn-default" onclick="enviaAssociado('.htmlspecialchars(json_encode($row['CTITULAR'])).') ">Alterar</a>';
Sometimes, the number will contain an 8 or a 9 and so it'll work (that's why the example you gave in a deleted comment on this answer, 0655178.00, worked). This works in loose mode, for instance:
enviaAssociado(0655178.00)
Notice the 8. (A 9 would do the same thing.) It works because the 8 is in the part of the number prior to the ., so the JavaScript parser knows that it's not a legacy octal integer literal, instead it's a (try not to laugh here) non-octal decimal integer literal, which is a specific kind of a decimal integer literal, which means it can be the first part of a decimal literal, so the . isn't a syntax error. This is one of JavaScript's deep dark corners. (And a good reason to use strict mode, which disallows non-octal decimal integer literals.)
But this is just a specific case of where the real problem is not outputting from PHP to JavaScript code correctly (and not properly escaping something being put in an HTML context, specifically in this case an attribute). So although this legacy octal integer literal thing is a deep dark corner of JavaScript, that's not the real issue. The real issue is ensuring you encode things properly when outputting from PHP to JavaScript like that, via json_encode (and in this case, htmlspecialchars).
Itroduction
I'm currently working on John Conway's Game of Life in js. I have the game working (view here) and i'm working on extra functionalities such as sharing your "grid / game" to your friends. To do this i'm extracting the value's of the grid (if the cell is alive or dead) into a long string of 0's and 1's.
This string has a variable length since the grid is not always the same size. for example:
grid 1 has a length and width of 30 => so the string's length is 900
grid 2 has a length and width of 50 => so the string's length is 2500
The problem
As you can see these string's of 0's and 1's are way too long to copy around and share.
However hard i try I don't seem to be able to come up with a code that would compress a string this long to a easy to handle one.
Any ideas on how to compress (and decompress) this?
I have considered simply writing down every possible grid option for the gird sizes 1x1 to 100x100 and giving them a key/reference to use as sharable code. Doing that by hand would be madness but maybe any of you has an idea on how to create an algorithm that can do this?
GitHub repository
In case it wasn't already obvious, the string you're trying to store looks like a binary string.
Counting systems
Binary is a number in base-2. This essentially means that there are two characters being used to keep count. Normally we are used to count with base-10 (decimal characters). In computer science the hexadecimal system (base-16) is also widely being used.
Since you're not storing the bits as bits but as bytes (use var a = 0b1100001; if you ever wish to store them like bits) the 'binary' you wish to store just takes as much space as any other random string with the same length.
Since you're using the binary system each position just has 2 possible values. When using the hexadecimal value a single position can hold up to 16 possible values. This is already a big improvement when it comes to storing the data compactly. As an example 0b11111111 and 0xff both represents the decimal number 255.
In your situation that'd shave 6 bytes of every 8 bytes you have to store. In the end you'd be stuck with a string just 1/4th of the length of the original string.
Javascript implementation
Essentially what we want to do is to interpret the string you store as binary and retrieve the hexadecimal value. Luckily JavaScript has built in functionality to achieve stuff like this:
var bin =
'1110101110100011' +
'0000101111100001' +
'1010010101011010' +
'0000110111011111' +
'1111111001010101' +
'0111000011100001' +
'1011010100110001' +
'0111111110010100' +
'0111110110100101' +
'0000111101100111' +
'1100001111011100' +
'0101011100001111' +
'0110011011001101' +
'1000110010001001' +
'1010100010000011' +
'0011110000000000';
var returnValue = '';
for (var i = 0; i < parseInt(bin.length / 8); i++) {
returnValue += parseInt(bin.substr(i*8, 8), 2).toString(16);
}
console.log(bin.length); // Will return 265
console.log(returnValue.length); // Will return 64
We're saying "parse this string and interpret it like a base-2 number and store it as a hexadecimal string".
Decoding is practically the same. Replace all occurrences of the number 8 in the example above with 2 and vice versa.
Please note
A prerequisite for this code to work correctly is that the binary length is dividable by 8. See the following example:
parseInt('00011110', 2).toString(16); // returns '1e'
parseInt('1e', 16).toString(2); // returns '11110'
// Technically both representations still have the same decimal value
When decoding you should add leading zeros until you have a full byte (8 bits).
In case the positions you have to store are not dividable by 8 you can, for example, add padding and add a number to the front of the output string to identify how much positions to strip.
Wait, there's more
To get even shorter strings you can build a lookup table with 265 characters in which you search for the character associated with the specific position. (This works because you're still storing the hexadecimal value as a string.) Sadly neither the ASCII nor the UTF-8 encodings are suited for this as there are blocks with values which have no characters defined.
It may look like:
// Go fill this array until you have 265 values within it.
var lookup = ['A', 'B', 'C', 'D'];
var smallerValue = lookup[0x00];
This way you can have 265 possible values at a single position, AND you have used your byte to the fullest.
Please note that no real compression is happening here. We're rather utilising data types to be used more efficiently for your current use case.
If we make the assumption than the grid contains much more 0's than 1's, you may want to try this simple compression scheme:
convert the binary string to an hexadecimal string
convert '00' sub-strings to 'z' symbol
convert 'zz' sub-strings to 'Z' symbol
we could go further, but let's stop here for the demo
Below is an example with a 16x16 grid:
var bin =
'0000000000000000' +
'0000001000000000' +
'0000011100000000' +
'0000001000000000' +
'0000000000000000' +
'0000000000111000' +
'0000100000111000' +
'0000000000111000' +
'0000000000000000' +
'0000000000000000' +
'0000000010000000' +
'0000000101000000' +
'0000000010000000' +
'0000000000000000' +
'0000100000000000' +
'0000000000000000';
var packed = bin
.match(/(.{4})/g)
.map(function(x) {
return parseInt(x, 2).toString(16);
})
.join('')
.replace(/00/g, 'z')
.replace(/zz/g, 'Z');
This will produce the string "Z02z07z02ZZ380838z38ZZz8z14z08Zz8Zz".
The unpacking process is doing the exact opposite:
var bin = packed
.replace(/Z/g, 'zz')
.replace(/z/g, '00')
.split('')
.map(function(x) {
return ('000' + parseInt(x, 16).toString(2)).substr(-4, 4);
})
.join('');
Note that this code will only work correctly if the length of the input string is a multiple of 4. If it's not the case, you'll have to pad the input and crop the output.
EDIT : 2nd method
If the input is completely random -- with roughly as many 0's as 1's and no specific repeating patterns -- the best you can do is probably to convert the binary string to a BASE64 string. It will be significantly shorter (this time with a fixed compression ratio of about 17%) and can still be copied/pasted by the user.
Packing:
var bin =
'1110101110100011' +
'0000101111100001' +
'1010010101011010' +
'0000110111011111' +
'1111111001010101' +
'0111000011100001' +
'1011010100110001' +
'0111111110010100' +
'0111110110100101' +
'0000111101100111' +
'1100001111011100' +
'0101011100001111' +
'0110011011001101' +
'1000110010001001' +
'1010100010000011' +
'0011110000000000';
var packed =
btoa(
bin
.match(/(.{8})/g)
.map(function(x) {
return String.fromCharCode(parseInt(x, 2));
})
.join('')
);
Will produce the string "66ML4aVaDd/+VXDhtTF/lH2lD2fD3FcPZs2MiaiDPAA=".
Unpacking:
var bin =
atob(packed)
.split('')
.map(function(x) {
return ('0000000' + x.charCodeAt(0).toString(2)).substr(-8, 8);
})
.join('');
Or if you want to go a step further, you can consider using something like base91 instead, for a reduced encoding overhead.
LZ-string
Using LZ-string I was able to compress the "code" quite a bit.
By simply compressing it to base64 like this:
var compressed = LZString.compressToBase64(string)
Decompressing is also just as simple as this:
var decompressed = LZString.decompressFromBase64(compressed)
However the length of this compressed string is still pretty long given that you have about as many 0s as 1s (not given in the example)
example
But the compression does work.
ANSWER
For any of you who are wondering how exactly I ended up doing it, here's how:
First I made sure every string passed in would be padded with leading 0s untill it was devidable by 8. (saving the amount of 0s used to pad, since they're needed while decompressing)
I used Corstian's answer and functions to compress my string (interpreted as binary) into a hexadecimal string. Although i had to make one slight alteration.
Not every binary substring with a lenght of 8 will return exactly 2 hex characters. so for those cases i ended up just adding a 0 in front of the substring. The hex substring will have the same value but it's length will now be 2.
Next up i used a functionality from Arnaulds answer. Taking every double character and replacing it with a single character (one not used in the hexadecimal alphabet to avoid conflict). I did this twice for every hexadecimal character.
For example:
the hex string 11 will become h and hh will become H
01101111 will become 0h0H
Since most grids are gonna have more dead cells then alive ones, I made sure the 0s would be able to compress even further, using Arnaulds method again but going a step further.
00 -> g | gg -> G | GG -> w | ww -> W | WW -> x | xx -> X | XX-> y | yy -> Y | YY -> z | zz -> Z
This resulted in Z representing 4096 (binary) 0s
The last step of the compression was adding the amount of leading 0s in front of the compressed string, so we can shave those off at the end of decompressing.
This is how the returned string looks in the end.
amount of leading 0s-compressed string so a 64*64 empty grid, will result in 0-Z
Decompressing is practically doing everything the other way around.
Firstly splitting the number that represents how many leading 0s we've used as padding from the compressed string.
Then using Arnaulds functionality, turning the further "compressed" characters back into hexadecimal code.
Taking this hex string and turning it back into binary code. Making sure, as Corstian pointed out, that every binary substring will have a length of 8. (ifnot we pad the substrings with leading 0s untill the do, exactly, have a length of 8)
And then the last step is to shave off the leading 0s we've used as padding to make the begin string devidable by 8.
The functions
Function I use to compress:
/**
* Compresses the a binary string into a compressed string.
* Returns the compressed string.
*/
Codes.compress = function(bin) {
bin = bin.toString(); // To make sure the binary is a string;
var returnValue = ''; // Empty string to add our data to later on.
// If the lenght of the binary string is not devidable by 8 the compression
// won't work correctly. So we add leading 0s to the string and store the amount
// of leading 0s in a variable.
// Determining the amount of 'padding' needed.
var padding = ((Math.ceil(bin.length/8))*8)-bin.length;
// Adding the leading 0s to the binary string.
for (var i = 0; i < padding; i++) {
bin = '0'+bin;
}
for (var i = 0; i < parseInt(bin.length / 8); i++) {
// Determining the substring.
var substring = bin.substr(i*8, 8)
// Determining the hexValue of this binary substring.
var hexValue = parseInt(substring, 2).toString(16);
// Not all binary values produce two hex numbers. For example:
// '00000011' gives just a '3' while what we wand would be '03'. So we add a 0 in front.
if(hexValue.length == 1) hexValue = '0'+hexValue;
// Adding this hexValue to the end string which we will return.
returnValue += hexValue;
}
// Compressing the hex string even further.
// If there's any double hex chars in the string it will take those and compress those into 1 char.
// Then if we have multiple of those chars these are compressed into 1 char again.
// For example: the hex string "ff will result in a "v" and "ffff" will result in a "V".
// Also: "11" will result in a "h" and "1111" will result in a "H"
// For the 0s this process is repeated a few times.
// (string with 4096 0s) (this would represent a 64*64 EMPTY grid)
// will result in a "Z".
var returnValue = returnValue.replace(/00/g, 'g')
.replace(/gg/g, 'G')
// Since 0s are probably more likely to exist in our binary and hex, we go a step further compressing them like this:
.replace(/GG/g, 'w')
.replace(/ww/g, 'W')
.replace(/WW/g, 'x')
.replace(/xx/g, 'X')
.replace(/XX/g, 'y')
.replace(/yy/g, 'Y')
.replace(/YY/g, 'z')
.replace(/zz/g, 'Z')
//Rest of the chars...
.replace(/11/g, 'h')
.replace(/hh/g, 'H')
.replace(/22/g, 'i')
.replace(/ii/g, 'I')
.replace(/33/g, 'j')
.replace(/jj/g, 'J')
.replace(/44/g, 'k')
.replace(/kk/g, 'K')
.replace(/55/g, 'l')
.replace(/ll/g, 'L')
.replace(/66/g, 'm')
.replace(/mm/g, 'M')
.replace(/77/g, 'n')
.replace(/nn/g, 'N')
.replace(/88/g, 'o')
.replace(/oo/g, 'O')
.replace(/99/g, 'p')
.replace(/pp/g, 'P')
.replace(/aa/g, 'q')
.replace(/qq/g, 'Q')
.replace(/bb/g, 'r')
.replace(/rr/g, 'R')
.replace(/cc/g, 's')
.replace(/ss/g, 'S')
.replace(/dd/g, 't')
.replace(/tt/g, 'T')
.replace(/ee/g, 'u')
.replace(/uu/g, 'U')
.replace(/ff/g, 'v')
.replace(/vv/g, 'V');
// Adding the number of leading 0s that need to be ignored when decompressing to the string.
returnValue = padding+'-'+returnValue;
// Returning the compressed string.
return returnValue;
}
The function I use to decompress:
/**
* Decompresses the compressed string back into a binary string.
* Returns the decompressed string.
*/
Codes.decompress = function(compressed) {
var returnValue = ''; // Empty string to add our data to later on.
// Splitting the input on '-' to seperate the number of paddin 0s and the actual hex code.
var compressedArr = compressed.split('-');
var paddingAmount = compressedArr[0]; // Setting a variable equal to the amount of leading 0s used while compressing.
compressed = compressedArr[1]; // Setting the compressed variable to the actual hex code.
// Decompressing further compressed characters.
compressed = compressed// Decompressing the further compressed 0s. (even further then the rest of the chars.)
.replace(/Z/g, 'zz')
.replace(/z/g, 'YY')
.replace(/Y/g, 'yy')
.replace(/y/g, 'XX')
.replace(/X/g, 'xx')
.replace(/x/g, 'WW')
.replace(/W/g, 'ww')
.replace(/w/g, 'GG')
.replace(/G/g, 'gg')
.replace(/g/g, '00')
// Rest of chars...
.replace(/H/g, 'hh')
.replace(/h/g, '11')
.replace(/I/g, 'ii')
.replace(/i/g, '22')
.replace(/J/g, 'jj')
.replace(/j/g, '33')
.replace(/K/g, 'kk')
.replace(/k/g, '44')
.replace(/L/g, 'll')
.replace(/l/g, '55')
.replace(/M/g, 'mm')
.replace(/m/g, '66')
.replace(/N/g, 'nn')
.replace(/n/g, '77')
.replace(/O/g, 'oo')
.replace(/o/g, '88')
.replace(/P/g, 'pp')
.replace(/p/g, '99')
.replace(/Q/g, 'qq')
.replace(/q/g, 'aa')
.replace(/R/g, 'rr')
.replace(/r/g, 'bb')
.replace(/S/g, 'ss')
.replace(/s/g, 'cc')
.replace(/T/g, 'tt')
.replace(/t/g, 'dd')
.replace(/U/g, 'uu')
.replace(/u/g, 'ee')
.replace(/V/g, 'vv')
.replace(/v/g, 'ff');
for (var i = 0; i < parseInt(compressed.length / 2); i++) {
// Determining the substring.
var substring = compressed.substr(i*2, 2);
// Determining the binValue of this hex substring.
var binValue = parseInt(substring, 16).toString(2);
// If the length of the binary value is not equal to 8 we add leading 0s (js deletes the leading 0s)
// For instance the binary number 00011110 is equal to the hex number 1e,
// but simply running the code above will return 11110. So we have to add the leading 0s back.
if (binValue.length != 8) {
// Determining how many 0s to add:
var diffrence = 8 - binValue.length;
// Adding the 0s:
for (var j = 0; j < diffrence; j++) {
binValue = '0'+binValue;
}
}
// Adding the binValue to the end string which we will return.
returnValue += binValue
}
var decompressedArr = returnValue.split('');
returnValue = ''; // Emptying the return variable.
// Deleting the not needed leading 0s used as padding.
for (var i = paddingAmount; i < decompressedArr.length; i++) {
returnValue += decompressedArr[i];
}
// Returning the decompressed string.
return returnValue;
}
URL shortener
I still found the "compressed" strings a little long for sharing / pasting around. So i used a simple URL shortener (view here) to make this process a little easier for the user.
Now you might ask, then why did you need to compress this string anyway?
Here's why:
First of all, my project is hosted on github pages (gh-pages). The info page of gh-pages tells us that the url can't be any longer than 2000 chars. This would mean that the max grid size would be the square root of 2000 - length of the base url, which isn't that big. By using this "compression" we are able to share much larger grids.
Now the second reason why is that, it's a challange. I find dealing with problems like these fun and also helpfull since you learn a lot.
Live
You can view the live version of my project here. and/or find the github repository here.
Thankyou
I want to thank everyone who helped me with this problem. Especially Corstian and Arnauld, since i ended up using their answers to reach my final functions.
Sooooo.... thanks guys! apriciate it!
In the Game of Life there is a board of ones and zeros. I want to back up to previous generation - size 4800 - save each 16 cells as hexadecimal = 1/4 the size. http://innerbeing.epizy.com/cwebgl/gameoflife.html [g = Go] [b = Backup]
function drawGen(n) {
stop(); var i = clamp(n,0,brw*brh-1), hex = gensave[i].toString();
echo(":",i, n,nGEN); nGEN = i; var str = '';
for (var i = 0; i < parseInt(hex.length / 4); i++)
str = str + pad(parseInt(hex.substr(i*4,4), 16).toString(2),16,'0');
for (var j=0;j<Board.length;j++) Board[j] = intr(str.substr(j,1));
drawBoard();
}
function Bin2Hex(n) {
var i = n.indexOf("1"); /// leading Zeros = NAN
if (i == -1) return "0000";
i = right(n,i*-1);
return pad(parseInt(i,2).toString(16),4,'0');
}
function saveGen(n) {
var b = Board.join(''), str = ''; /// concat array to string 10101
for (var i = 0; i < parseInt(b.length / 16); i++)
str = str + Bin2Hex(b.substr(i*16,16));
gensave[n] = str;
}
function right(st,n) {
var s = st.toString();
if (!n) return s;
if (n < 0) return s.substr(n * -1,s.length + n);
return s.substr(s.length - n,n);
}
function pad(str, l, padwith) {
var s = str;
while (s.length < l) s = padwith + s;
return s;
}
My app generates links, which contain hex string like: 37c1fbcabbc31f2f8d2ad31ceb91cd8d0d189ca5963dc6d353188d3d5e75b8b3e401d4e74e9b3e02efbff0792cda5c4620cb3b1f84aeb47b8d2225cd40e761a5. I would really like to make them shorter, like the solution mentioned for Ruby in Compressing a hex string in Ruby/Rails.
Is there a way to do this in JavaScript/NodeJS?
node int-encoder does this, using the strategy already mentioned.
it also supports large numbers
npm install int-encoder
var en = require('int-encoder');
//simple integer conversion
en.encode(12345678); // "ZXP0"
en.decode('ZXP0'); // 12345678
//convert big hex number using optional base argument
en.encode('e6c6b53d3c8160b22dad35a0f705ec09', 16); // 'hbDcW9aE89tzLYjDgyzajJ'
en.decode('hbDcW9aE89tzLYjDgyzajJ', 16); // 'e6c6b53d3c8160b22dad35a0f705ec09'
You could use toString and parseInt method, that basically are doing the same thing of the methods you mentioned in the link:
var hexString = "4b3fc1400";
var b36 = parseInt(hexString, 16).toString(36); // "9a29mgw"
And to convert it back, you just need to do the opposite:
hexString = parseInt(b36, 36).toString(16); // "4b3fc1400"
The only problem with your string, is that is too big to be threat as number in JavaScript. You should split them in chunk. JavaScript's numbers are accurate up to 2^53 (plus sign), so the max positive number you can handle is 0x20000000000000 (in hexadecimal, that is 9007199254740992 in decimal); you can use the accuracy to handle the chunk:
var hexString = "37c1fbcabbc31f2f8d2ad31ceb91cd8d0d189ca5963dc6d353188d3d5e75b8b3e401d4e74e9b3e02efbff0792cda5c4620cb3b1f84aeb47b8d2225cd40e761a5"
var b36 = "", b16 = "";
var chunk, intChunk;
// 14 is the length of 0x20000000000000 (2^53 in base 16)
for (var i = 0, max = 14; i < hexString.length; i += max) {
chunk = hexString.substr(i, max);
intChunk = parseInt(chunk, 16);
if (intChunk.toString(16) !== chunk) {
intChunk = parseInt(hexString.substr(i, max - 1), 16);
i -= 1;
}
b36 += intChunk.toString(36)
}
// 11 is the length of 2gosa7pa2gv (2^53 in base 36)
for (var i = 0, max = 11; i < b36.length; i += max ) {
chunk = b36.substr(i, max);
intChunk = parseInt(chunk, 36);
if (intChunk.toString(36) !== chunk) {
intChunk = parseInt(b36.substr(i, max - 1), 36);
i -= 1;
}
b16 += intChunk.toString(16)
}
console.log(hexString);
console.log(b36);
console.log(b16);
Update: You could also use a base 62 instead of 36 to compress more, but notice that JS supports up to base 36, so you need to implement that personal notation manually (I believe there are already some implementation around).
The simplest and fastest thing to do is define a set of 64 safe characters for use in the URL, such as A-Z, a-z, 0-9, _, and $. Then encode every three hex digits (4 bits each) into two safe characters (6 bits each). This requires no multiplication and division, and it can be used on arbitrarily long strings.
You will need to pick a 65th character to use at the end of the string to indicate if the last four-bit piece is used or not. Otherwise you will have an ambiguity for character strings with an even number of characters. Let's call it 2n. Then there are either 3n-1 or 3n hex digits encoded within, but there is no way to tell which. You can follow the sequence with a special character to indicate one of those cases. E.g. a '.' (period).
Note: The last few characters picked here for the set differ from Base64 encoding, since URLs have their own definition of safe punctuation characters. See RFC 1738.
I'm working on a twitter app and just stumbled into the world of utf-8(16). It seems the majority of javascript string functions are as blind to surrogate pairs as I was. I've got to recode some stuff to make it wide character aware.
I've got this function to parse strings into arrays while preserving the surrogate pairs. Then I'll recode several functions to deal with the arrays rather than strings.
function sortSurrogates(str){
var cp = []; // array to hold code points
while(str.length){ // loop till we've done the whole string
if(/[\uD800-\uDFFF]/.test(str.substr(0,1))){ // test the first character
// High surrogate found low surrogate follows
cp.push(str.substr(0,2)); // push the two onto array
str = str.substr(2); // clip the two off the string
}else{ // else BMP code point
cp.push(str.substr(0,1)); // push one onto array
str = str.substr(1); // clip one from string
}
} // loop
return cp; // return the array
}
My question is, is there something simpler I'm missing? I see so many people reiterating that javascript deals with utf-16 natively, yet my testing leads me to believe, that may be the data format, but the functions don't know it yet. Am I missing something simple?
EDIT:
To help illustrate the issue:
var a = "0123456789"; // U+0030 - U+0039 2 bytes each
var b = "𝟘𝟙𝟚𝟛𝟜𝟝𝟞𝟟𝟠𝟡"; // U+1D7D8 - U+1D7E1 4 bytes each
alert(a.length); // javascript shows 10
alert(b.length); // javascript shows 20
Twitter sees and counts both of those as being 10 characters long.
Javascript uses UCS-2 internally, which is not UTF-16. It is very difficult to handle Unicode in Javascript because of this, and I do not suggest attempting to do so.
As for what Twitter does, you seem to be saying that it is sanely counting by code point not insanely by code unit.
Unless you have no choice, you should use a programming language that actually supports Unicode, and which has a code-point interface, not a code-unit interface. Javascript isn't good enough for that as you have discovered.
It has The UCS-2 Curse, which is even worse than The UTF-16 Curse, which is already bad enough. I talk about all this in OSCON talk, 🔫 Unicode Support Shootout: 👍 The Good, the Bad, & the (mostly) Ugly 👎.
Due to its horrible Curse, you have to hand-simulate UTF-16 with UCS-2 in Javascript, which is simply nuts.
Javascript suffers from all kinds of other terrible Unicode troubles, too. It has no support for graphemes or normalization or collation, all of which you really need. And its regexes are broken, sometimes due to the Curse, sometimes just because people got it wrong. For example, Javascript is incapable of expressing regexes like [𝒜-𝒵]. Javascript doesn’t even support casefolding, so you can’t write a pattern like /ΣΤΙΓΜΑΣ/i and have it correctly match στιγμας.
You can try to use the XRegEXp plugin, but you won’t banish the Curse that way. Only changing to a language with Unicode support will do that, and 𝒥𝒶𝓋𝒶𝓈𝒸𝓇𝒾𝓅𝓉 just isn’t one of those.
I've knocked together the starting point for a Unicode string handling object. It creates a function called UnicodeString() that accepts either a JavaScript string or an array of integers representing Unicode code points and provides length and codePoints properties and toString() and slice() methods. Adding regular expression support would be very complicated, but things like indexOf() and split() (without regex support) should be pretty easy to implement.
var UnicodeString = (function() {
function surrogatePairToCodePoint(charCode1, charCode2) {
return ((charCode1 & 0x3FF) << 10) + (charCode2 & 0x3FF) + 0x10000;
}
function stringToCodePointArray(str) {
var codePoints = [], i = 0, charCode;
while (i < str.length) {
charCode = str.charCodeAt(i);
if ((charCode & 0xF800) == 0xD800) {
codePoints.push(surrogatePairToCodePoint(charCode, str.charCodeAt(++i)));
} else {
codePoints.push(charCode);
}
++i;
}
return codePoints;
}
function codePointArrayToString(codePoints) {
var stringParts = [];
for (var i = 0, len = codePoints.length, codePoint, offset, codePointCharCodes; i < len; ++i) {
codePoint = codePoints[i];
if (codePoint > 0xFFFF) {
offset = codePoint - 0x10000;
codePointCharCodes = [0xD800 + (offset >> 10), 0xDC00 + (offset & 0x3FF)];
} else {
codePointCharCodes = [codePoint];
}
stringParts.push(String.fromCharCode.apply(String, codePointCharCodes));
}
return stringParts.join("");
}
function UnicodeString(arg) {
if (this instanceof UnicodeString) {
this.codePoints = (typeof arg == "string") ? stringToCodePointArray(arg) : arg;
this.length = this.codePoints.length;
} else {
return new UnicodeString(arg);
}
}
UnicodeString.prototype = {
slice: function(start, end) {
return new UnicodeString(this.codePoints.slice(start, end));
},
toString: function() {
return codePointArrayToString(this.codePoints);
}
};
return UnicodeString;
})();
var ustr = UnicodeString("f𝌆𝌆bar");
document.getElementById("output").textContent = "String: '" + ustr + "', length: " + ustr.length + ", slice(2, 4): " + ustr.slice(2, 4);
<div id="output"></div>
Here are a couple scripts that might be helpful when dealing with surrogate pairs in JavaScript:
ES6 Unicode shims for ES3+ adds the String.fromCodePoint and String.prototype.codePointAt methods from ECMAScript 6. The ES3/5 fromCharCode and charCodeAt methods do not account for surrogate pairs and therefore give wrong results.
Full 21-bit Unicode code point matching in XRegExp with \u{10FFFF} allows matching any individual code point in XRegExp regexes.
Javascript string iterators can give you the actual characters instead of the surrogate code points:
>>> [..."0123456789"]
["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"]
>>> [..."𝟘𝟙𝟚𝟛𝟜𝟝𝟞𝟟𝟠𝟡"]
["𝟘", "𝟙", "𝟚", "𝟛", "𝟜", "𝟝", "𝟞", "𝟟", "𝟠", "𝟡"]
>>> [..."0123456789"].length
10
>>> [..."𝟘𝟙𝟚𝟛𝟜𝟝𝟞𝟟𝟠𝟡"].length
10
This is along the lines of what I was looking for. It needs better support for the different string functions. As I add to it I will update this answer.
function wString(str){
var T = this; //makes 'this' visible in functions
T.cp = []; //code point array
T.length = 0; //length attribute
T.wString = true; // (item.wString) tests for wString object
//member functions
sortSurrogates = function(s){ //returns array of utf-16 code points
var chrs = [];
while(s.length){ // loop till we've done the whole string
if(/[\uD800-\uDFFF]/.test(s.substr(0,1))){ // test the first character
// High surrogate found low surrogate follows
chrs.push(s.substr(0,2)); // push the two onto array
s = s.substr(2); // clip the two off the string
}else{ // else BMP code point
chrs.push(s.substr(0,1)); // push one onto array
s = s.substr(1); // clip one from string
}
} // loop
return chrs;
};
//end member functions
//prototype functions
T.substr = function(start,len){
if(len){
return T.cp.slice(start,start+len).join('');
}else{
return T.cp.slice(start).join('');
}
};
T.substring = function(start,end){
return T.cp.slice(start,end).join('');
};
T.replace = function(target,str){
//allow wStrings as parameters
if(str.wString) str = str.cp.join('');
if(target.wString) target = target.cp.join('');
return T.toString().replace(target,str);
};
T.equals = function(s){
if(!s.wString){
s = sortSurrogates(s);
T.cp = s;
}else{
T.cp = s.cp;
}
T.length = T.cp.length;
};
T.toString = function(){return T.cp.join('');};
//end prototype functions
T.equals(str)
};
Test results:
// plain string
var x = "0123456789";
alert(x); // 0123456789
alert(x.substr(4,5)) // 45678
alert(x.substring(2,4)) // 23
alert(x.replace("456","x")); // 0123x789
alert(x.length); // 10
// wString object
x = new wString("𝟘𝟙𝟚𝟛𝟜𝟝𝟞𝟟𝟠𝟡");
alert(x); // 𝟘𝟙𝟚𝟛𝟜𝟝𝟞𝟟𝟠𝟡
alert(x.substr(4,5)) // 𝟜𝟝𝟞𝟟𝟠
alert(x.substring(2,4)) // 𝟚𝟛
alert(x.replace("𝟜𝟝𝟞","x")); // 𝟘𝟙𝟚𝟛x𝟟𝟠𝟡
alert(x.length); // 10