How to serialize a numpy array of floats to a string? - javascript

I have a huge numpy array of floats, say ~1500*2500px and I want to
convert this array to a list (like javascript) (e.g. [[0.1,0.3,0.2],[0.1,0.3,0.2]])
serialize it to a string for a POST request to a server.
I don't know how to do (1). For (2) I took a look at numpy.array_str(), array2string() and array_repr() functions and they return representations of the array but not the full array.
How should I do this?

I'm not sure why you want it "this array to [look?] like a JavaScript array, so I am presuming (as I am at liberty to do in the absence of information to the contrary) that you wish to communicate the array to some unfortunate front-end process: almost four million elements is still a significant amount of data to squirt across network pipes. So, as always, some background to the problem would be helpful (and you can edit your question to provide it).
Assuming you want to serialize the data for transmission or storage then the simplest way to render it as a string comprehensible to JavaScript (I didn't rally know what "[look?] like" meant) is using the json standard library. Since this can't natively encode anything but list and dicts of ints, floats, truth values and strings, you are still faced with the problem of how best to represent the matrix as a list of lists.
Small example, but you have to accept this is a random shot in the dark. First
let's create a manageable data set to work with:
a = np.random.randn(4, 5)
This cannot be directly represented in JSON:
import json
try:
json.dumps(a)
except Exception as e:
print "Exception", e
resulting in the rather verbose (it's probably just calling the object's repr) but comprehensible and true message
Exception array([[ 1.24064541, 0.97989932, -0.8469167 , -0.27318908, 1.21954134],
[-1.30172725, 0.41261504, 1.39895842, 0.75260258, -1.34749298],
[-0.38415007, -0.56925321, -1.59202204, 1.29900292, 1.91357277],
[ 1.06254537, 2.75700739, -0.66371951, 1.36906192, -0.3973517 ]]) is not JSON serializable
If we ask the interpreter to convert the array to a list it does a half-hearted job, converting it into a list of array objects:
list(a)
shows as its result
[array([ 1.24064541, 0.97989932, -0.8469167 , -0.27318908, 1.21954134]),
array([-1.30172725, 0.41261504, 1.39895842, 0.75260258, -1.34749298]),
array([-0.38415007, -0.56925321, -1.59202204, 1.29900292, 1.91357277]),
array([ 1.06254537, 2.75700739, -0.66371951, 1.36906192, -0.3973517 ])]
Using the same function to convert those arrays into lists yields a usable list of lists:
list(list(r) for r in a)
evaluating to
[[1.2406454087805279,
0.97989932000522928,
-0.84691669720415574,
-0.27318907894171163,
1.219541337120247],
[-1.3017272505660062,
0.41261503624079976,
1.3989584188044133,
0.75260257672408482,
-1.3474929807527067],
[-0.38415007296182629,
-0.56925320938196644,
-1.5920220380072485,
1.2990029230603588,
1.9135727724853433],
[1.0625453748520415,
2.7570073901625185,
-0.66371950666590918,
1.3690619178580901,
-0.39735169991907082]]
This is eminently convertible to JSON, which I do here by converting it into a string:
json.dumps(list(list(r) for r in a))
which gives the (string) result
'[[1.2406454087805279, 0.97989932000522928, -0.84691669720415574, -0.27318907894171163, 1.219541337120247], [-1.3017272505660062, 0.41261503624079976, 1.3989584188044133, 0.75260257672408482, -1.3474929807527067], [-0.38415007296182629, -0.56925320938196644, -1.5920220380072485, 1.2990029230603588, 1.9135727724853433], [1.0625453748520415, 2.7570073901625185, -0.66371950666590918, 1.3690619178580901, -0.39735169991907082]]'
You can check that the result is correct by reconstituting the list of lists and comparing it with the array (since one of the arguments is a numpy array, the comparison is done elementwise):
s = json.dumps(list(list(r) for r in a))
lofls = json.loads(s)
lofls == a
array([[ True, True, True, True, True],
[ True, True, True, True, True],
[ True, True, True, True, True],
[ True, True, True, True, True]], dtype=bool)
Did I understand your question correctly?

the following method is intuitive.
np.ndarray -> list -> json_str -> list -> np.array(list_obj)
code sample:
import json
import numpy as np
np_arr = np.array([[1, 2], [7, 8]])
json_str = json.dumps(np_arr.tolist())
arr = json.loads(json_str)
restore_np = np.array(arr)
# True
print((np_arr == restore_np).all())

You can convert it to a normal python list and then to a string
arr = np.random.rand((10,10))
final_string = str(arr.tolist())
resulting in
[[0.7998950511604668, 0.3504357174428122, 0.4516363276829708, 0.42090556177992977], [0.5151195486975273, 0.7101183117731774, 0.9530575343271824, 0.39869760958795464], [0.20318293100519536, 0.17244659329654555, 0.3530236209359401, 0.2081303162461341], [0.9186758779272243, 0.9300730012004015, 0.14121513893149895, 0.39315493832613735]]

Related

Papers.js exportJSON (Options) how to specify the Options?

I am using Paper.js, in Javascript, because I need to debug my code.
I want to generate a Json String of my drawings, which works well.
But I need to reduce the precision.
0 is 0.0003. 510.05 = 510.05005..such things.
Documentation mentions:
————
exportJSON([options])
Exports (serializes) the project with all its layers and child items to a JSON data object or string.
Options:
options.asString: Boolean — whether the JSON is returned as a Object or a String — default: true.
options.precision: Number — the amount of fractional digits in numbers used in JSON data — default: 5.
Parameters:
options: Object — the serialization options — optional
Returns:
String — the exported JSON data
I do not understand what this means. How specify these options? Whatever I try endup in a crash.
I am programming Javascript since about 3 month, I am coming from C and Assembler languages.
Maybe my question is too simple for this forum?
I did try:
json_vect_string = layer_wall_vector.exportJSON(true, 2);
json_vect_string = layer_wall_vector.exportJSON(asString = true, precision = 2);
json_vect_string = layer_wall_vector.exportJSON(options.asString = true, options.precision = 2);

Algorithm to turn JavaScript object into efficient query string/urls

Context: I have been building an application that creates and uses magnet links. I've been trying to find an efficient way to transfer a javascript object in the Querystring so that On the other side I can deserialize it into an object keeping the same types. with efficient i mean using as little characters as possible/transferring as much data as possible. I've found my application has a max of +-1500 characters in url.
At first I used original Querystring npm packages but these can change types on deserialize and also very inefficient on deeper objects.
eg:
var input = { age: 12, name:'piet'};
var qs = querystring.encode(input); // ?age=12&name=piet
var output querystring.decode(qs); // {age: '12', name: 'piet'
Then I've tried using json stringifying with and without base64 for Querystrings. But this left me most of the time with much bigger strings for simple objects.
var input = { age: 12, name:'piet'};
var qs = encodeURIComponent(JSON.stringify(input)); // "%7B%22age%22%3A12%2C%22name%22%3A%22piet%22%7D"
But this leaves me with rediculous long querystring because half of the characters get encoded and become 3x as long which almost doubles the length.
base64 encoding in this case is a much better solution:
var input = { age: 12, name:'piet'};
var qs = btoa(JSON.stringify(input)); // eyJhZ2UiOjEyLCJuYW1lIjoicGlldCJ9
I've been trying to Google for an efficient algorithm it but haven't really found a good solution. I've been looking into msgPack binary serialisation by then I would also have to base64 which probably ends with a longer string.
Is there a known more efficient algorithm for object to Querystring serialisation with static types? or would i have to create my own?
I've been thinking on on a simple query string algorithm that works as follows:
Query string order is imporotant, for next point:
Keys starts with . shows depth: ?obj&.property="test" = { obj : {property: "test" }}
First character in string defines its type: b=boolean, s=string,n=number, (etc if needed)
This would lead to much more efficient query string i think. but am i not building something that has already been made before?
Surely sticking the stringified object into a single URI value should work?
var test = { string: 'string', integer: 8, intarray: [1,2,3] };
encodeURIComponent(JSON.stringify(test))
// "%7B%22string%22%3A%22string%22%2C%22integer%22%3A8%2C%22intarray%22%3A%5B1%2C2%2C3%5D%7D"
JSON.parse(decodeURIComponent("%7B%22string%22%3A%22string%22%2C%22integer%22%3A8%2C%22intarray%22%3A%5B1%2C2%2C3%5D%7D")
// {string: "string", integer: 8, intarray: [1, 2, 3]}
The object parsed at the end has the same types as the object inputted at the start.
Just stick your object into a single key:
var url = 'http://example.com/query?key=' + encodeURIComponent(JSON.stringify(object));
Right? And on the server you just parse that single value into an object.

How to pass an array of integers as a parameter from javascript to python?

I have a javascript code that obtains the value of several checked boxes and inserts their value (integers) into an array:
var my_list = $('.item:checked').map(function(){return $(this).attr('name');}).get();
I want to pass this javascript array to a python function as a paramter. This is how I am doing it right now, with ajax and jQuery:
$.ajax({
url : "{{tg.url('/foo/bar')}}",
data : {
my_list: JSON.stringify(my_list),
...
},
On the python side, I have this code, to get the parameters:
item_list = get_paramw(kw, 'my_list', unicode)
This works, but I am not receiving the array as an array of integers, but as a single string containing the "[", "]", "," and """ symbols, which I would have to parse and isn't the most elegant way of dealing with this whole situation I think.
How should it be done to receive a javascript array of integers as an array of integers in python (not as a string)?
The easiest way to send simple arbitrarily-structured data from Javascript to Python is JSON. You've already got the Javascript side of it, in that JSON.stringify(my_list). All you need is the Python side. And it's one line of code:
item_list_json = get_paramw(kw, 'my_list', unicode)
item_list = json.loads(item_list_json)
If item_list in your JS code were the Javascript array of numbers [1, 2, 3, 4], item_list in your Python code would be the Python list of ints [1, 2, 3, 4].
(Well, you also have to import json at the top of your script, so I guess it's two lines.)
The proper way to send multiple values for one key is
data: {
my_list: my_list,
},
traditional: true
...
Suppose my_list = [ 1, 2, 3 ], this results a query string / POST data
my_list=1&my_list=2&my_list=3
Without the traditional flag, the result would be
my_list[]=1&my_list[]=2&my_list[]=3
Which of course works, but in TurboGears you need to access it as my_list[].
On the other hand, I am not sure if the TG keyword argument passing work with multiple values; if you get a single value in kw, you can use request.getall('foo') to return all the foo keys as a list (possibly an empty list).

Traversing 'JSON Array of arrays' in python 'List of lists' format

For reasons I don't understand, the JSON format data I need to parse that comes out of a webservice isn't in nme value pairs. 'For simplicity and to reduce overhead' the returned JSON seems to be in a format that works with python eval but as far as I can see not with javascript (caveat, my javascript is very poor so I could be wrong - php etc, fine. js, not so much!)
So the data is returned as:
[[0, 'OK'],
[['ITEM10314', ['ITEM10397']],
['ITEM10315', ['cornflower']],
['ITEM10397', ['ITEM10315']],
['ITEM10514', ['ITEM10397']],
['ITEM003', []],
['ITEM004', []],
['servertest', ['ITEM004', 'ITEM003']],
['serverroot', []]]]
(in case you are interested, it is a reply from a MKLiveStatus for Nagios LQL host query)
The first array is the status, then subsequent arrays consist of the host monitored in nagios and that host's parents (in an inner array).
Nice, isn't it. But I need to get it into decent key/value pairs and there must be a better way than writing my own parser for this (not least because this is one data output but there are several more in similar formats).
I'm trying to keep this all in native js but if there is a jQuery easy way then I'm easily led to lazyness. No need to worry about old browsers, I don't care, this project ends up using d3.js which won't work on old browsers anyway.
Any suggestions? The depth in this case won't go below what it is here, so that at least is a known. However, I can't just flatten it, I need to know which parents a host has after this.
I have seen a few python-js links here but not arbitrary unknown sized lists in lists.
Something like this should do it
var data = [
[0, "OK"],
[
["ITEM10314", ["ITEM10397"]],
["ITEM10315", ["cornflower"]],
["ITEM10397", ["ITEM10315"]],
["ITEM10514", ["ITEM10397"]],
["ITEM003", []],
["ITEM004", []],
["servertest", ["ITEM004", "ITEM003"]],
["serverroot", []]
]
];
function parse(array) {
var object = {
ok: 1
};
if (!Array.isArray(array) && array[0][0] !== 0 && array[0][1] !== "OK") {
return object;
}
object.ok = 0;
array[1].forEach(function (element) {
object[element[0]] = element[1];
});
return object;
}
console.log(parse(data));
On jsfiddle

Is the order of elements in a JSON list preserved?

I've noticed the order of elements in a JSON object not being the original order.
What about the elements of JSON lists? Is their order maintained?
Yes, the order of elements in JSON arrays is preserved. From RFC 7159 -The JavaScript Object Notation (JSON) Data Interchange Format
(emphasis mine):
An object is an unordered collection of zero or more name/value
pairs, where a name is a string and a value is a string, number,
boolean, null, object, or array.
An array is an ordered sequence of zero or more values.
The terms "object" and "array" come from the conventions of
JavaScript.
Some implementations do also preserve the order of JSON objects as well, but this is not guaranteed.
The order of elements in an array ([]) is maintained. The order of elements (name:value pairs) in an "object" ({}) is not, and it's usual for them to be "jumbled", if not by the JSON formatter/parser itself then by the language-specific objects (Dictionary, NSDictionary, Hashtable, etc) that are used as an internal representation.
Practically speaking, if the keys were of type NaN, the browser will not change the order.
The following script will output "One", "Two", "Three":
var foo={"3":"Three", "1":"One", "2":"Two"};
for(bar in foo) {
alert(foo[bar]);
}
Whereas the following script will output "Three", "One", "Two":
var foo={"#3":"Three", "#1":"One", "#2":"Two"};
for(bar in foo) {
alert(foo[bar]);
}
Some JavaScript engines keep keys in insertion order. V8, for instance, keeps all keys in insertion order except for keys that can be parsed as unsigned 32-bit integers.
This means that if you run either of the following:
var animals = {};
animals['dog'] = true;
animals['bear'] = true;
animals['monkey'] = true;
for (var animal in animals) {
if (animals.hasOwnProperty(animal)) {
$('<li>').text(animal).appendTo('#animals');
}
}
var animals = JSON.parse('{ "dog": true, "bear": true, "monkey": true }');
for (var animal in animals) {
$('<li>').text(animal).appendTo('#animals');
}
You'll consistently get dog, bear, and monkey in that order, on Chrome, which uses V8. Node.js also uses V8. This will hold true even if you have thousands of items. YMMV with other JavaScript engines.
Demo here and here.
"Is the order of elements in a JSON list maintained?" is not a good question. You need to ask "Is the order of elements in a JSON list maintained when doing [...] ?"
As Felix King pointed out, JSON is a textual data format. It doesn't mutate without a reason. Do not confuse a JSON string with a (JavaScript) object.
You're probably talking about operations like JSON.stringify(JSON.parse(...)). Now the answer is: It depends on the implementation. 99%* of JSON parsers do not maintain the order of objects, and do maintain the order of arrays, but you might as well use JSON to store something like
{
"son": "David",
"daughter": "Julia",
"son": "Tom",
"daughter": "Clara"
}
and use a parser that maintains order of objects.
*probably even more :)

Categories

Resources