Develop Reference

Scrapy: Converting Javascript array to Json on Python

I have been struggling with a site I am scrapping using scrappy.
This site, returns a series of Javascript variables (array) with the products data.
Example:
datos[0] = ["12345","3M YELLOW CAT5E CABLE","6.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1,"",\'\'];
datos[1] = ["12346","3M GREEN CAT5E CABLE","7.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1,"",\'\'];
...
So on...
Fetching the array into a string with scrapy was easy, since the site response prints the variables.
The problem is I want to transform it into Json so I can process it and store it in a database table.
Normally I would use Javascript's function Json.stringify to convert it to Json and post it in PHP.
However when using Python's json.loads and even StringIO I am unable to load the array into json.
Probably is a format error, but I am unable to identify it, since I am not expert in Json nor Python.
EDIT:
I just realize since scrapy is unable to execute Javascript probably the main issue is that the data is just a string. I should format it into a Json format.
Any help is more than welcome.
Thank you.

If you wanted to take an array and create a json object, you could do something like this.
values = ["12345","3M YELLOW CAT5E CABLE","6.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1]
keys = [x for x in range(len(values))]
d = dict(zip(keys, values))
x = json.dumps(d)

There is a section in the scrapy doc to find various ways to parse the JavaScript code. For your case, if you just need to have it in an array, you can use the regex to get the data.
Since the website you are scraping is not present in the question, I am assuming this would be a more straightforward way to get it, but you could use whichever way seems suitable.

JS: convert string into object

I have code
data = "{isShowLoginPopup:true,newFavOfferId:1486882}";
I want to convert it into JS object (not in JSON) and use it in this way:
data.newFavOfferId = ...
How can I do this?

If your source is trusted, the simplest solution is to use eval :
data = eval('('+data+')');
If you don't trust the source, then you'd better specify what you can have and parse the string manually (not terribly hard if you have only one level of properties for example).
Another solution (depending on your real data) would be to change your data into JSON by inserting the missing quotes :
data = JSON.parse(datareplace(/({|,)\s*([^:,}{]+)\s*(:)/g,'$1"$2"$3'));

just remove the quotes
data = {
isShowLoginPopup:true,
newFavOfferId:1486882
};
Fiddle: http://jsfiddle.net/QpZ4j/

just remove quotes "" from the
data = "{isShowLoginPopup:true,newFavOfferId:1486882}";
DEMO

Whilst on the surface this looks like JSON data, it's malformed and therefore it does not work directly with JSON.parse(). This is because JSON objects require keys to be wrapped in quotes...
therefore:
"{isShowLoginPopup:true,newFavOfferId:1486882}"
as valid JSON should be:
"{\"isShowLoginPopup\":true,\"newFavOfferId\":1486882}"
So what you have there in fact IS a JavaScript object, not JSON, however the problem you have is that this is a JavaScript object as a string literal. If this is hard coded, then you need to just remove the " from the beginning and end of the string.
var data = {isShowLoginPopup:true,newFavOfferId:1486882};
If this object is serialized and requires transmission from/to a server etc, then realistically, it needs to be transmitted as a JSON formatted string, which can then be de-serialized back into a JavaScript object.
var data = JSON.parse("{\"isShowLoginPopup\":true,\"newFavOfferId\":1486882}");

How to convert a JSON string to a JSON string with a different structure

I am building an application where data is retrieved from a third party system as a JSON string. I need to convert this JSON string to another JSON string with a different structure such that it can be used with pre-existing functions defined in a internal Javascript library.
Ideally I want to be able to perform this conversion on the client machine using Javascript.
I have looked at JSONT as a means of achieving this but that project does not appear to be actively maintained:
http://goessner.net/articles/jsont/
Is there a de facto way of achieving this? Or do I have to roll my own mapping code?

You shouldn't be passing JSON into an internal JavaScript library. You should parse the JSON into a JS object, then iterate over it, transforming it into the new format
Example
var json = '[{"a": 1:, "b": 2}, {"a": 4:, "b": 5}]';
var jsObj = JSON.parse(json);
// Transform property a into aa and property b into bb
var transformed = jsObj.map(function(obj){
return {
aa: obj.a,
bb: obj.b
}
});
// transformed = [{aa:1, bb:2},{aa:4, bb:5}]
If you really want JSON you'd just call JSON.stringify(transformed)
https://developer.mozilla.org/en-US/docs/JavaScript/Reference/Global_Objects/Array/map
Here's another answer with an even more complicated transformation How to make a jquery Datatable array out of standard json?

From what I can tell from the home page, the JSONT project is about transforming JSON into entirely different formats anyway (i.e. JSON => HTML).
It's going to be a lot simpler to write your own mapping code, possibly just as a from_json() method on the object you're creating (so YourSpecialObject.from_json(input); returns an instance of that object generated from the JSON data).
From your question, I'm not sure if this fits your use case, but hopefully someone else will have a better answer soon.

Another option is using XSLT. As there are SAX readers and writers for JSON, you can write happily use XSLT with JSON. There's no horrific JSON to XML and back conversion needs to go on. See: http://www.gerixsoft.com/blog/json/xslt4json
I can definitely see the irony in using a XML based language to tranform JSON - but it seems like a good option.
Otherwise you're probably best of writing your own mapping code.

Saving javascript objects as strings

This is for a gaming application.
In my game I want to save special effects on a player in a single field of my database. I know I could just put a refrence Id and do another table and I haven't taken that option off the table.
Edit: (added information) This is for my server in node not the browser.
The way I was thinking about storing the data is as a javascript object as follows:
effects={
shieldSpell:0,
breatheWater:0,
featherFall:0,
nightVision:0,
poisonResistance:0,
stunResistance:0,
deathResistance:0,
fearResistance:0,
blindResistance:0,
lightningResistance:0,
fireResistance:0,
iceResistance:0,
windResistance:0}
It seems easy to store it as a string and use it using effects=eval(effectsString)
Is there an easy way to make it a string or do I have to do it like:
effectsString=..."nightVision:"+effects.nightVision.toString+",poisonResistance:"+...

Serialize like that:
JSON.stringify(effects);
Deserialize like that:
JSON.parse(effects);

Use JSON.stringify
That converts a JS object into JSON. You can then easily deserialize it with JSON.parse. Do not use the eval method since that is inviting cross-site scripting
//Make a JSON string out of a JS object
var serializedEffects = JSON.stringify(effects);
//Make a JS object from a JSON string
var deserialized = JSON.parse(serializedEffects);

JSON parse and stringify is what I use for this type of storatge
var storeValue = JSON.stringify(effects); //stringify your value for storage
// "{"shieldSpell":0,"breatheWater":0,"featherFall":0,"nightVision":0,"poisonResistance":0,"stunResistance":0,"deathResistance":0,"fearResistance":0,"blindResistance":0,"lightningResistance":0,"fireResistance":0,"iceResistance":0,"windResistance":0}"
var effects = JSON.parse(storeValue); //parse back from your string

Here was what I've come up with so far just wonering what the downside of this solution is.
effectsString="effects={"
for (i in effects){
effectsString=effectsString+i+":"+effects[i]+","
}
effectsString=effectsString.slice(0,effectsString.length-1);
effectsString=effectsString+"}"
Then to make the object just
eval(effectsString)

Can't pass JSON array with AJAX

I'm trying to get a PHP array with AJAX and turn it into a JSON array.
Right now in the external file I'm echoing the php array with JSON encoding:
echo json_encode($palabras);
Then on the main file I get the response and assign it to variable "jsarray ";
success:function(data_response){
jsarray = data_response;
}});
However I can't access jsarray as an array. How can I turn it into a proper array I can access?

You can use eval:
eval(data_response)
You can use jquery.parseJSON too:
var obj = jQuery.parseJSON(data_response);

If you are 100% certain you can trust the contents of data_response, you can simply evaluate it.
I don't know what library you are using, but that should be something like jsarray = eval(data_response.responseText);
WARNING: This does pose a security risk if a third party can inject data into data_response. Since it's a simple eval(), any javascript code can be executed.
I believe that most modern webbrowsers provide a JSON library to work around this issue. You should be able to use JSON.parse(data_response.responseText) in that case. However, that will not work on all browsers.