Develop Reference

Scrapy: Converting Javascript array to Json on Python

I have been struggling with a site I am scrapping using scrappy.
This site, returns a series of Javascript variables (array) with the products data.
Example:
datos[0] = ["12345","3M YELLOW CAT5E CABLE","6.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1,"",\'\'];
datos[1] = ["12346","3M GREEN CAT5E CABLE","7.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1,"",\'\'];
...
So on...
Fetching the array into a string with scrapy was easy, since the site response prints the variables.
The problem is I want to transform it into Json so I can process it and store it in a database table.
Normally I would use Javascript's function Json.stringify to convert it to Json and post it in PHP.
However when using Python's json.loads and even StringIO I am unable to load the array into json.
Probably is a format error, but I am unable to identify it, since I am not expert in Json nor Python.
EDIT:
I just realize since scrapy is unable to execute Javascript probably the main issue is that the data is just a string. I should format it into a Json format.
Any help is more than welcome.
Thank you.

If you wanted to take an array and create a json object, you could do something like this.
values = ["12345","3M YELLOW CAT5E CABLE","6.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1]
keys = [x for x in range(len(values))]
d = dict(zip(keys, values))
x = json.dumps(d)

There is a section in the scrapy doc to find various ways to parse the JavaScript code. For your case, if you just need to have it in an array, you can use the regex to get the data.
Since the website you are scraping is not present in the question, I am assuming this would be a more straightforward way to get it, but you could use whichever way seems suitable.

How to convert a JSON string to a JSON string with a different structure

I am building an application where data is retrieved from a third party system as a JSON string. I need to convert this JSON string to another JSON string with a different structure such that it can be used with pre-existing functions defined in a internal Javascript library.
Ideally I want to be able to perform this conversion on the client machine using Javascript.
I have looked at JSONT as a means of achieving this but that project does not appear to be actively maintained:
http://goessner.net/articles/jsont/
Is there a de facto way of achieving this? Or do I have to roll my own mapping code?

You shouldn't be passing JSON into an internal JavaScript library. You should parse the JSON into a JS object, then iterate over it, transforming it into the new format
Example
var json = '[{"a": 1:, "b": 2}, {"a": 4:, "b": 5}]';
var jsObj = JSON.parse(json);
// Transform property a into aa and property b into bb
var transformed = jsObj.map(function(obj){
return {
aa: obj.a,
bb: obj.b
}
});
// transformed = [{aa:1, bb:2},{aa:4, bb:5}]
If you really want JSON you'd just call JSON.stringify(transformed)
https://developer.mozilla.org/en-US/docs/JavaScript/Reference/Global_Objects/Array/map
Here's another answer with an even more complicated transformation How to make a jquery Datatable array out of standard json?

From what I can tell from the home page, the JSONT project is about transforming JSON into entirely different formats anyway (i.e. JSON => HTML).
It's going to be a lot simpler to write your own mapping code, possibly just as a from_json() method on the object you're creating (so YourSpecialObject.from_json(input); returns an instance of that object generated from the JSON data).
From your question, I'm not sure if this fits your use case, but hopefully someone else will have a better answer soon.

Another option is using XSLT. As there are SAX readers and writers for JSON, you can write happily use XSLT with JSON. There's no horrific JSON to XML and back conversion needs to go on. See: http://www.gerixsoft.com/blog/json/xslt4json
I can definitely see the irony in using a XML based language to tranform JSON - but it seems like a good option.
Otherwise you're probably best of writing your own mapping code.

regular expression to filter out part of key value in a json string

I have the following JSON string as part of a log line.
cells : {"Lac":"7824","CntryISO":"us","NetTyp":"GSM","NetOp":"310260","Cid":"11983"}
I want to filter out to the following format: {"Lac":"7824","Cid":"11983"}.
How can do this using regular expression ? in Javascript or Python ?
the keys are constant strings(Lac, CntryISO, ...), but the value strings are varying.

Why don't you just delete them in JavaScript?
var myJson = {"Lac":"7824","CntryISO":"us","NetTyp":"GSM","NetOp":"310260","Cid":"11983"};
delete myJson.Lac;
delete myJson.cId;

To expand and explain #alex answer:
JSON is a nested multi dimensional structure. Simply filtering the "string-ifiyed form of a Javascript object" (aka JSON) will work in very simple cases, but will rapidly fail when the structure is no longer flat or it starts to get complex with escaped fields, etc.
At that point you will need proper parsing logic. This is nicely supplied by Javascript itself, to quote #alexes code:
var myJson = {"Lac":"7824","CntryISO":"us","NetTyp":"GSM","NetOp":"310260","Cid":"11983"};
delete myJson.Lac;
delete myJson.cId;
Or, if you want to use python, the json module will work just fine:
http://docs.python.org/library/json.html
Good luck! :)

Why would you want to use regex for this when you can just use a JSON parser/serializer? Try cjson in Python if you are concerned about speed, it is faster than 'json' module in the Python standard library.

Best practice for writing ARRAYS

I've got an array with about 250 entries in it, each their own array of values. Each entry is a point on a map, and each array holds info for:
name,
another array for points this point can connect to,
latitude,
longitude,
short for of name,
a boolean,
and another boolean
The array has been written by another developer in my team, and he has written it as such:
names[0]=new Array;
names[0][0]="Campus Ice Centre";
names[0][1]= new Array(0,1,2);
names[0][2]=43.95081811364498;
names[0][3]=-78.89848709106445;
names[0][4]="CIC";
names[0][5]=false;
names[0][6]=false;
names[1]=new Array;
names[1][0]="Shagwell's";
names[1][1]= new Array(0,1);
names[1][2]=43.95090307839151;
names[1][3]=-78.89815986156464;
names[1][4]="shg";
names[1][5]=false;
names[1][6]=false;
Where I would probably have personally written it like this:
var names = []
names[0] = new Array("Campus Ice Centre", new Array[0,1,2], 43.95081811364498, -78.89848709106445, "CIC", false, false);
names[1] = new Array("Shagwell's", new Array[0,1], 43.95090307839151, -78.89815986156464, 'shg", false, false);
They both work perfectly fine of course, but what I'm wondering is:
1) does one take longer than the other to actually process?
2) am I incorrect in assuming there is a benefit to the compactness of my version of the same thing?
I'm just a little worried about his 3000 lines of code versus my 3-400 to get the same result.
Thanks in advance for any guidance.

What you really want to do here is define a custom data type which represents your data more accurately. I'm not sure what language you are using so here is some psuedocode:
class Location
{
double latitude;
double longitude;
String Name;
String Abbreviation;
bool flag1;//you should use a better name
bool flag2;
}
Then you can just create an array to hold all the Location objects and it would be much more readable and maintainable.
Locations = new Array;
Locations[0] = new Location("Shagwell's",...);
....
===EDIT===
Because you said you are using javascript then the best practise would probably be to store your data in a json text file, this has the benefit of removing the data from the code file and having a very easily editable data source if you want to make changes.
your JSON file would look like this
[{"lat":"23.2323", "long":"-72.3", "name":"Shagwell's" ...},
{"lat":"26.2323", "long":"-77.3", "name":"loc2" ...},
...]
You could then store the json text in an accesible place on your webserver say "data.json", then if you are using jquery you can load it in by doing something like this:
$.getJSON("data.json", function(data) { //do something with the data});

With structured data, like your example, both you and your co-worker are relatively "wrong". From the looks of things, you should have implemented an array of structures, assuming of course that the data you are presenting is truly unordered, which I would be willing to guess it probably isn't. Arrays are used too often, because they are amongst the first data structures we learn, but very often aren't the best choice.
As to performance, that more often comes down to the data access code than the data type itself. Frankly too, unless you are dealing with gigantic datasets or literally real time applications, performance should be a non issue.
As to the two examples you have posted, after the compiler is done with them, they will be virtually identical.

I personally find the former much more readable. From a performance perspective, the difference is probably minimal.

Leaving the other answers aside (although I agree with the others that you need structs here) your co-workers way seems better to me. Like Serapth says, the compiler will optimize away the differences and the original code has better readability.

How Do You Fix A Parameter Names Mismatch - DOJO and PL/SQL

How do you fix a names mismatch problem, if the client-side names are keywords or reserved words in the server-side language you are using?
The DOJO JavaScript toolkit has a QueryReadStore class that you can subclass to submit REST patterned queries to the server. I'm using this in conjunction w/ the FilteringSelect Dijit.
I can subclass the QueryReadStore and specify the parameters and arguments getting passed to the server. But somewhere along the way, a "start" and "count" parameter are being passed from the client to the server. I went into the API and discovered that the QueryReadStore.js is sending those parameter names.
I'm using Fiddler to confirm what's actually being sent and brought back. The server response is telling me I have a parameter names mismatch, because of the "start" and "count" parameters. The problem is, I can't use "start" and "count" in PL/SQL.
Workaround or correct implementation advice would be appreciated...thx.
//I tried putting the code snippet in here, but since it's largely HTML, that didn't work so well.

While it feels like the wrong thing to do, because I'm hacking at a well tested, nicely written JavaScript toolkit, this is how I fixed the problem:
I went into the DOJOX QueryReadStore.js and replaced the "start" and "count" references with acceptable (to the server-side language) parameter names.
I would have like to handled the issue via my PL/SQL (but I don't know how to get around reserved words) or client-side code (subclassing did not do the trick)...without getting into the internals of the library. But it works, and I can move on.

As opposed to removing it from the API, as you mentioned, you can actually create a subclass with your own fetch, and remove start/count parameters (theoretically). Have a look at this URL for guidance:
http://www.sitepen.com/blog/2008/06/25/web-service-data-store/
Start and count are actually very useful because they allow you to pass params for the query that you can use to filter massive data sets, and it helps to manage client-side paging. I would try to subclass instead, intercept, and remove.

Is your pl/sql program accessed via a URL and mod_plsql? If so, then you can use "flexible parameter passing" and the variables get assigned to an array of name/value pairs.
Define your package spec like this...
create or replace package pkg_name
TYPE plsqltable
IS
TABLE OF VARCHAR2 (32000)
INDEX BY BINARY_INTEGER;
empty plsqltable;
PROCEDURE api (name_array IN plsqltable DEFAULT empty ,
value_array IN plsqltable DEFAULT empty
);
END pkg_name;
Then the body:
CREATE OR REPLACE PACKAGE BODY pkg_name AS
l_count_value number;
l_start_value number;
PROCEDURE proc_name (name_array IN plsqltable DEFAULT empty,
value_array IN plsqltable DEFAULT empty) is
------------
FUNCTION get_value (p_name IN VARCHAR) RETURN VARCHAR2 IS
BEGIN
FOR i IN 1..name_array.COUNT LOOP
IF UPPER(name_array(i)) = UPPER(p_name) THEN
RETURN value_array(i);
END IF;
END LOOP;
RETURN NULL;
END get_value;
----------------------
begin
l_count_value := get_value('count');
l_start_value := get_value('start');
end api;
end pkg_name;
Then you can call pkg_name.api using
http://server/dad/!pkg_name.api?start=3&count=3