I have been struggling with a site I am scrapping using scrappy.
This site, returns a series of Javascript variables (array) with the products data.
Example:
datos[0] = ["12345","3M YELLOW CAT5E CABLE","6.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1,"",\'\'];
datos[1] = ["12346","3M GREEN CAT5E CABLE","7.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1,"",\'\'];
...
So on...
Fetching the array into a string with scrapy was easy, since the site response prints the variables.
The problem is I want to transform it into Json so I can process it and store it in a database table.
Normally I would use Javascript's function Json.stringify to convert it to Json and post it in PHP.
However when using Python's json.loads and even StringIO I am unable to load the array into json.
Probably is a format error, but I am unable to identify it, since I am not expert in Json nor Python.
EDIT:
I just realize since scrapy is unable to execute Javascript probably the main issue is that the data is just a string. I should format it into a Json format.
Any help is more than welcome.
Thank you.
If you wanted to take an array and create a json object, you could do something like this.
values = ["12345","3M YELLOW CAT5E CABLE","6.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1]
keys = [x for x in range(len(values))]
d = dict(zip(keys, values))
x = json.dumps(d)
There is a section in the scrapy doc to find various ways to parse the JavaScript code. For your case, if you just need to have it in an array, you can use the regex to get the data.
Since the website you are scraping is not present in the question, I am assuming this would be a more straightforward way to get it, but you could use whichever way seems suitable.
I would like to come straight to the point and show you my sample data, which is around the average of 180.000 lines from a .csv file, so a lot of lines. I am reading in the .csv with papaparse. Then I am saving the data as array of objects, which looks like this:
I just used this picture as you can also see all the properties my objects have or should have. The data is from Media Transperency Data, which is open source and shows the payments between institiutions.
The array of objects is saved by using the localforage technology, which is basically an IndexedDB or WebSQL with localstorage like API. So I save the data never on a sever! Only in the client!
The Question:
So my question is now, the user can add the sourceHash and/or targetHash attributes in a client interface. So for example assume the user loaded the "Energie Steiermark Kunden GmbH" object and now adds the sourceHash -- "company" to it. So basically a tag. This is already reflected in the client and shown, however I need to get this also in the localforage and therefore rewrite the initial array of objects. So I would need to search for every object in my huge 180.000 lines array that has the name "Energie Steiermark Kunden GmbH", as there can be multiple and set the property sourceHash to "company". Then save it again in the localforage.
The first question would be how to do this most efficient? I can get the data out of localforage by using the following method and set it respectively.
Get:
localforage.getItem('data').then((value) => {
...
});
Set:
localforage.setItem('data', dataObject);
However, the question is how do I do this most efficiently? I mean if the sourceNode only starts with "E" for example we don't need to search all sourceNode's. The same goes of course for the targetNode.
Thank you in advance!
UPDATE:
Thanks for the answeres already! And how would you do it the most efficient way in Javascript? I mean is it possible to do it in few lines. If we assume I have for example the current sourceHash "company" and want to assign it to every node starting with "Energie Steiermark Kunden GmbH" that appear across all timeNode's. It could be 20151, 20152, 20153, 20154 and so on...
Localforage is only a localStorage/sessionStorage-like wrapper over the actual storage engine, and so it only offers you the key-value capabilities of localStorage. In short, there's no more efficient way to do this for arbitrary queries.
This sounds more like a case for IndexedDB, as you can define search indexes over the data, for instance for sourceNodes, and do more efficient queries that way.
So, I used data declared by myself, but now i switch the code to LocalStorage, and I'd like to know, how to get the data from one element of LocalStorage, and insert it to a block of data from my program.
Here's a part of code which shows the procedure which i use for inserting data
let l = this.lists;
localStorage.setItem('lists', JSON.stringify(l));
l is of type string, and lists is an array with data block.
I wanted to use this command
this.lists = localStorage.getItem('lists');
but unfortunately,it wants a string element, and doesn't want to work with my lists element...
Info time:
LocalStorage is implementation of Storage interface and it accepts and
returns plain strings so every time you want to store there something
a little bit more complex you have to serialize when inserting
(JSON.stringify) and deserialize when retrieving (JSON.parse)
You can use JSON.parse()
The JSON.parse() method parses a JSON string, constructing the JavaScript value or object described by the string.
this.lists = JSON.parse(localStorage.getItem('lists'));
localStorage is implementation of Storage interface, It works on plain strings. When you want to work with complex object serialize it using JSON.stringify() and deserialize using JSON.parse()
How can I get the distance value and how to assign it to another variable.
I get that data from bluemix (Watson IoT Platform) to node-red
{distance:"45.9"};
I tried like
var data=msg.distance;
Use Json.parse to convert the string to an array.Now you can access the elements of the array.
If you include the JSON node it will convert your JS Object to JSON. But maybe you have messed with the quotes and actually have json; by default you would.
I would add a Debug node to your IoT-In node. Check what exactly you receive. And then it is usually easy to parse like (it depends on what you send):
var distance = msg.payload.d.distance
var distance = msg.payload.distance
You might want to edit your Question to include exactly what you receive in the Debug node and need to parse.
Also be aware that your value for distance is a string, you'll probably want to convert it to a number at somepoint. If it is in your control, it would be better practice to send it as a number to start with.
I have a web page that queries a database to get a list of products with their related data like price, size,weight e.t.c and displays it to the user, I want to add one of those sort dropdown list so that a user can sort the products by price or any other key I specify.
I plan to do it using Ajax on dropdown change query the database again and use the selected value as the sort key in the query.
My question is, is there a way that this can be done client side without running another query? Can php send the query result to the browser and store it there and then use jquery to sort it over and over again?
Thanks everyone for the help!
There are many ways this can be achieved. A simple example can be done with HTML5 LocalStorage.
Here's a general overview of how that can be done:
You make an initial call to your DB for products via AJAX, returns a JSON object.
You serialize that JSON object via javascript e.g. var stringData = JSON.stringify(data)
Store said variable into localstorage: window.localStorage.setItem("products", stringData);
You can then later access it via var products = window.localStorage.getItem("products) and finally deserialize it with var productsObj = JSON.parse(products) or do it all in one c-c-c-c-c-combo breaker: productsObj = JSON.parse(window.localStorage.getItem("products))
Do as you please by using filtering functions!
Another way is to store the initial JSON object using some form of store e.g. Redux, but let's save that for another day ;)
And of course, you may optionally have a globally accessible object that you can assign the initial JSON data to as a property and later access as you would any other property.
Take a look at this jquery plugin
http://tablesorter.com/docs/example-ajax.html
Maybe it's what you are looking for? No need to requery the database.