I have been struggling with a site I am scrapping using scrappy.
This site, returns a series of Javascript variables (array) with the products data.
Example:
datos[0] = ["12345","3M YELLOW CAT5E CABLE","6.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1,"",\'\'];
datos[1] = ["12346","3M GREEN CAT5E CABLE","7.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1,"",\'\'];
...
So on...
Fetching the array into a string with scrapy was easy, since the site response prints the variables.
The problem is I want to transform it into Json so I can process it and store it in a database table.
Normally I would use Javascript's function Json.stringify to convert it to Json and post it in PHP.
However when using Python's json.loads and even StringIO I am unable to load the array into json.
Probably is a format error, but I am unable to identify it, since I am not expert in Json nor Python.
EDIT:
I just realize since scrapy is unable to execute Javascript probably the main issue is that the data is just a string. I should format it into a Json format.
Any help is more than welcome.
Thank you.
If you wanted to take an array and create a json object, you could do something like this.
values = ["12345","3M YELLOW CAT5E CABLE","6.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1]
keys = [x for x in range(len(values))]
d = dict(zip(keys, values))
x = json.dumps(d)
There is a section in the scrapy doc to find various ways to parse the JavaScript code. For your case, if you just need to have it in an array, you can use the regex to get the data.
Since the website you are scraping is not present in the question, I am assuming this would be a more straightforward way to get it, but you could use whichever way seems suitable.
I want to get some values from a sql database for a graph via php. I want to give the javascript graph the values via JSON but the weird thing is, the valuesneed to look like this:
{
Player: 3,
Game: 'Diablo 3'
}, {
Player: 1,
Game: 'Overwatch'
}
I am not sure...but is this some sort of JSON? because when I echo a json_encoded array with php it looks like this:
["3","1"]
when the first is no JSON is it something own and how could I format the values in the best way that they fit for the graphs?
This might be hjson.
Hjson is a syntax extension to JSON. It's NOT a proposal to replace JSON or to incorporate it into the JSON spec itself. It's intended to be used like a user interface for humans, to read and edit before passing the JSON data to the machine.
Also, it may be a valid JavaScript object, except the parent array ‘[]’ notation is missing.
I am trying to create my own sort of "state" routing and am struggling with the manipulation of the URL. I have all the working parts for saving the state as an object and such, but I need to turn that object into a URL. So for reference, here is what my object looks like :
urlObject =
[
{"module":"module1",
"customUrl":[{"mod1":["1","2"]},{"mod2":["1","2"]}]
},
{"module":"module2",
"customUrl":[{"mod3":["true","false"]},{"mod4":["5","6"]}]
}
]
So right now I am just doing a simple
$location.search(JSON.stringify(urlObject));
To toss it in the URL. It would be neat if there were some way to format that, parse the URL formatting in my own way, so like It would change to like
/module1="module1sobject/module2="module2object"
When I say module2sobject, I mean the customUrl inside that object--so in that case it would be [{"mod1":["1","2"]},{"mod2":["1","2"]}]. I'm wondering if I could get some guidance on how to begin this process of setting and getting the object out of a url like this (specifically for use in my angular controllers).
I have a little bit of an issue with a JavaScript function that needs to read data from a TextFile (something JS is already limited with) and then process tha TextFile data into a MultiDimensional Array (another thing that JS doesn't nativelly suport).
With that in mind, I have a text file in this format:
1, Name, Data, Serial
2, Name, Data, Serial
3, Name, Data, Serial
And so on.
So, the objective is to get that same data and put it, like that, into an array.
I suppose that, from what I've been reading, I need an Array of an Array, segmenting the first one by lines [/n] and the second one by commas [,]. However, given the "by-default" limitations, I'm very confused at this point. I do suppose I need jQuery, however.
I tried this:
var fs = require('fs');
var array = fs.readFileSync('file.txt').toString().split("\n");
for(i in array) {
var array = fs.readFileSync('file.txt').toString().split(",");
for(f in array) {
}
}
With little success, because then I don't really know how to store it, the objective being a Multidimensional Array that Replicates the Format of the text file, so latter it could be used to search by index or instance following an user input to get results.
I really appreciate any help.
At first glance it seems like you are trying to read in a CSV file. If that is indeed the case I recommend node-csv:
http://www.adaltas.com/projects/node-csv/
https://github.com/wdavidw/node-csv
This helped me reading file to JavaScript, however this example converts retrieved data to JSON. Just looking at the format of your text file, I would assume a JSON string or Javascript object would work with your data.
Example convert to JSON
With JSON and JS objects, instead of referencing a array indexes eg. array[i][x]. you would replace [x] with .propertyName
data = {
"id": 1,
"name": "Fred"
};
//access data like this
data[i].name //will return "Fred" as i =0
to create JS object, just initialize array properties without the ""(quotation marks). accessing JS and JSON properties are done in the same way, main advantage over a multidimensional array is that you can reference an actual property name, as opposed to indexes.
In an app, made with TideSDK; i assign a global variable (shocking I know) to a the JSON parse of a string stored in Titanium.App.Properties:
var workbookArray = JSON.parse(Titanium.App.Properties.getString('workbookArray'));
workbookArray is an array of objects.
And then on the unloading of a page, I assign Titanium.App.Properties string the value of workbookArray, which may have been changed by whoever has used the app:
Titanium.App.Properties.setString('workbookArray', JSON.stringify(workbookArray));
Each time I open the app, however, I'm told that JSON was unable to parse the first code snippet (initializing workbookArray).
Aside from this issue, I don't expect to use the app Properties API for my storage needs in the longterm, I wish i could use indexedDB with titanium. SQL is an option, but is a little messy when it comes to objects. Any other suggestions for a database solution?
Try getList and setList
http://docs.appcelerator.com/titanium/latest/#!/api/Titanium.App.Properties
What is stored in the list?