I have been struggling with a site I am scrapping using scrappy.
This site, returns a series of Javascript variables (array) with the products data.
Example:
datos[0] = ["12345","3M YELLOW CAT5E CABLE","6.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1,"",\'\'];
datos[1] = ["12346","3M GREEN CAT5E CABLE","7.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1,"",\'\'];
...
So on...
Fetching the array into a string with scrapy was easy, since the site response prints the variables.
The problem is I want to transform it into Json so I can process it and store it in a database table.
Normally I would use Javascript's function Json.stringify to convert it to Json and post it in PHP.
However when using Python's json.loads and even StringIO I am unable to load the array into json.
Probably is a format error, but I am unable to identify it, since I am not expert in Json nor Python.
EDIT:
I just realize since scrapy is unable to execute Javascript probably the main issue is that the data is just a string. I should format it into a Json format.
Any help is more than welcome.
Thank you.
If you wanted to take an array and create a json object, you could do something like this.
values = ["12345","3M YELLOW CAT5E CABLE","6.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1]
keys = [x for x in range(len(values))]
d = dict(zip(keys, values))
x = json.dumps(d)
There is a section in the scrapy doc to find various ways to parse the JavaScript code. For your case, if you just need to have it in an array, you can use the regex to get the data.
Since the website you are scraping is not present in the question, I am assuming this would be a more straightforward way to get it, but you could use whichever way seems suitable.
How can I get the distance value and how to assign it to another variable.
I get that data from bluemix (Watson IoT Platform) to node-red
{distance:"45.9"};
I tried like
var data=msg.distance;
Use Json.parse to convert the string to an array.Now you can access the elements of the array.
If you include the JSON node it will convert your JS Object to JSON. But maybe you have messed with the quotes and actually have json; by default you would.
I would add a Debug node to your IoT-In node. Check what exactly you receive. And then it is usually easy to parse like (it depends on what you send):
var distance = msg.payload.d.distance
var distance = msg.payload.distance
You might want to edit your Question to include exactly what you receive in the Debug node and need to parse.
Also be aware that your value for distance is a string, you'll probably want to convert it to a number at somepoint. If it is in your control, it would be better practice to send it as a number to start with.
In Node.js you can assign values to keys off of the global object. This gives you the ability to "remember" something between requests. Assuming the node.js process doesn't die/hang and restart by a process like iisnode.
Does Vert.x have an equivalent? Essentially, I'm looking for the simplest possible cache for a piece of data so I do not have to fetch it on every request. I assume the solution on Vert.x may be able to work across threads?
the code:
{{id:1,name:"Yahoo"},{id:2,name:"Google"}}
break cause it's not valid json,you can use
{companies : [{id:1,name:"Yahoo"},{id:2,name:"Google"}]} //notice than they are inside an array
now..the doc says
To prevent issues due to mutable data, vert.x only allows simple immutable types such as number, boolean and string or Buffer to be used in shared data
that means, maybe you will need use
var map = vertx.getMap('demo.mymap');
map.put('data', JSON.stringify({companies : [{id:1,name:"Yahoo"},{id:2,name:"Google"}]}))
and then in other verticle
var map = vertx.getMap('demo.mymap');
var yourJSON = JSON.parse(map.get('data');
now..maybe a good option which would be use redis like a cache system, although the vertex map seems solve your needs so far...
In an app, made with TideSDK; i assign a global variable (shocking I know) to a the JSON parse of a string stored in Titanium.App.Properties:
var workbookArray = JSON.parse(Titanium.App.Properties.getString('workbookArray'));
workbookArray is an array of objects.
And then on the unloading of a page, I assign Titanium.App.Properties string the value of workbookArray, which may have been changed by whoever has used the app:
Titanium.App.Properties.setString('workbookArray', JSON.stringify(workbookArray));
Each time I open the app, however, I'm told that JSON was unable to parse the first code snippet (initializing workbookArray).
Aside from this issue, I don't expect to use the app Properties API for my storage needs in the longterm, I wish i could use indexedDB with titanium. SQL is an option, but is a little messy when it comes to objects. Any other suggestions for a database solution?
Try getList and setList
http://docs.appcelerator.com/titanium/latest/#!/api/Titanium.App.Properties
What is stored in the list?
This is for a gaming application.
In my game I want to save special effects on a player in a single field of my database. I know I could just put a refrence Id and do another table and I haven't taken that option off the table.
Edit: (added information) This is for my server in node not the browser.
The way I was thinking about storing the data is as a javascript object as follows:
effects={
shieldSpell:0,
breatheWater:0,
featherFall:0,
nightVision:0,
poisonResistance:0,
stunResistance:0,
deathResistance:0,
fearResistance:0,
blindResistance:0,
lightningResistance:0,
fireResistance:0,
iceResistance:0,
windResistance:0}
It seems easy to store it as a string and use it using effects=eval(effectsString)
Is there an easy way to make it a string or do I have to do it like:
effectsString=..."nightVision:"+effects.nightVision.toString+",poisonResistance:"+...
Serialize like that:
JSON.stringify(effects);
Deserialize like that:
JSON.parse(effects);
Use JSON.stringify
That converts a JS object into JSON. You can then easily deserialize it with JSON.parse. Do not use the eval method since that is inviting cross-site scripting
//Make a JSON string out of a JS object
var serializedEffects = JSON.stringify(effects);
//Make a JS object from a JSON string
var deserialized = JSON.parse(serializedEffects);
JSON parse and stringify is what I use for this type of storatge
var storeValue = JSON.stringify(effects); //stringify your value for storage
// "{"shieldSpell":0,"breatheWater":0,"featherFall":0,"nightVision":0,"poisonResistance":0,"stunResistance":0,"deathResistance":0,"fearResistance":0,"blindResistance":0,"lightningResistance":0,"fireResistance":0,"iceResistance":0,"windResistance":0}"
var effects = JSON.parse(storeValue); //parse back from your string
Here was what I've come up with so far just wonering what the downside of this solution is.
effectsString="effects={"
for (i in effects){
effectsString=effectsString+i+":"+effects[i]+","
}
effectsString=effectsString.slice(0,effectsString.length-1);
effectsString=effectsString+"}"
Then to make the object just
eval(effectsString)