Develop Reference

Posting long text from scripted browser (phantomjs) to a php script wont go over 2kb /2400 char

im doing some data scraping using phantomjs (basically its a browser you can code in and run it from command line)
i'm collection information about football matches (teams / countries / leagues , .... ) each as an object , put them all in an array , encode array to jason format and post the result to a php script
result could be a very long text based on amount of games on each day and when that happens i wont get the full text on the php script
in the php script i've stored the posted data in a text file to see whats going on , each time its about 2.3kb max and 2397~ characters
while the original posted text which i can see on the terminal is about 40kb and 40000 chars
so something must be limiting the characters here is my php.ini info which is more than enough :
post_max_size -> 20M
max_input_vars -> 100000
memory_limit -> 256M
here the simplified version of my code :
var res = page.evaluate(function(sport) {
var matches = new Array();
$('div#table-matches').find('.table-main').find('tr').each(function(index, element) {
var obj = {
teams : $(this).find('td').text() ,
link : $(this).find('td').find('a:last').attr('href') ,
};
matches.push( obj );
});
return matches ;
});
var postBody = 'sport='+sport+'&data='+JSON.stringify(res);
console.log(postBody);
page.open('http://xxxxx/result/save', 'POST', postBody, function(status) {
phantom.exit();
});
here is what i got in the php script :
[{"country":"Japan","league":"Emperors Cup","link":"/soccer/japan/emperors-cup/kobe-urawa-65k5LIMh/","match_date":"2015/12/26 04:00","teams":"Kobe - Urawa"},
{"country":"England","league":"Ryman League","link":"/soccer/england/ryman-league/wingate-finchley-metropolitan-police-rwuqgSz9/","match_date":"2015/12/26 12:00","teams":"Wingate
the json code has been cut off in the middle , so its not a valid json
is there anything else i should do ?

Usually it's much simpler to look for the answer when HTML that your code operates on is provided in the question (a link to the site is as good).
Luckily you'fe left some clues in a portion of json file, namely the link to one of the pages from the scraped site: /soccer/england/ryman-league/wingate-finchley-metropolitan-police-rwuqgSz9/
If we find it with Google and look at the source it will be clear that data in php script is truncated right at the title of the team «Wingate & Finchley - Metropolitan», that contains an ampersand, which serves as a delimiter for variables and values sent via an URI request and breaks your data variable into several others.
So, to amend your script you just have to encode the data string properly:
var postBody = 'sport='+sport+'&data='+encodeURIComponent(JSON.stringify(res));
Other way to find the cause of this issue would be to check $_SERVER and $_POST arrays at the server side, just dumping it to a file and looking if the whole data was really present, since you've already done great job checking PHP config that it should be present.
A way to go around the issue, had it not been solved (if, for example, PhantomJS had a weird bug with POST equests), would be to create a temporary file with data and send to php the path to that file (presuming the parsing is done on one and the same server):
var fs = require('fs');
var filename = '/tmp/scraped_' + (new Date()).getTime() + ".json";
fs.write(filename, JSON.stringify(res), 'w');
page.open('http://xxxxx/result/save', {"filename" : filename}, function(status) {
phantom.exit();
});

JavaScript JSON Combination and Data Anylish

I am trying to add values from multiple JSON responses that are saved in a .txt file. The .txt files has about 4000 entries. They are each the same format as follows:
{"id":"8f546dcf-b66a-4c53-b3d7-7290429483b8","price":"247.96000000","size":"0.03121005","product_id":"BTC-USD","side":"sell","stp":"dc"}
{"id":"0ec4b63a-b736-42af-a0aa-b4581bf12955","price":"247.90000000","size":"0.03910014","product_id":"BTC-USD","side":"sell","stp":"dc"}
{"id":"be403848-74dc-4494-8095-bd468777c958","price":"247.89000000","size":"0.04280854","product_id":"BTC-USD","side":"sell","stp":"dc"}
{"id":"ae2ae129-e850-4d8f-b945-55e65eb68a88","price":"247.83000000","size":"0.07941840","product_id":"BTC-USD","side":"sell","stp":"dc"}
{"id":"96194be4-40d8-446d-9f7e-ce72bc84af48","price":"247.63000000","size":"0.06225897","product_id":"BTC-USD","side":"sell","stp":"dc"}
I believe I need to combine the different JSON data sets before I can loop through them with a for loop and do the summing/analysis part (size:JSONSET1 + size:JSONSET2 + .....) but I'm not sure how I should call the .txt file in javascript and have it combine the multiple json parts. Suggestions??

Do you have any control over the file with the data set? If you do, you can make the input file one big JSON string.
Run this command in a terminal to add a comma to the end of every line:
sed -i 's/$/,/' yourFile.txt
Then edit the file with a text editor and put a [ at the beginning of the first line, and replace the last line's ending comma with a ].
Then after you read the file into a string, you can parse it like so:
var dataArray = JSON.parse(dataString);
And you could access the data like this:
console.log(dataArray[0].id);
This will print "8f546dcf-b66a-4c53-b3d7-7290429483b8" to the console

I would recommend using a .json file since you are working with JSON{Objects}.
I've made a demo of the file here and I've made a demo of the data analysis here.
Note: Demo file is hosted in a personal server, so it may not work later on.
Now the file isn't proper JSON Syntax, so it needs some parsing.
entries.json
{"id":"8f546dcf-b66a-4c53-b3d7-7290429483b8","price":"247.96000000","size":"0.03121005","product_id":"BTC-USD","side":"sell","stp":"dc"}
{"id":"0ec4b63a-b736-42af-a0aa-b4581bf12955","price":"247.90000000","size":"0.03910014","product_id":"BTC-USD","side":"sell","stp":"dc"}
(..)
JavaScript
Req = new XMLHttpRequest();
Req.onload = process_entries;
Req.open("get", "http://butler.tk/entries.json", true);
Req.send();
function process_entries() {
var response = Req.responseText;
var arrayed = "[" + response
.split("}")
.join("},")
.slice(0,-1)
+ "]";
var entries = JSON.parse(arrayed);
for (var i = 0, l = entries.length; i < l; i++) {
var entry = entries[i];
//entry.id,entry.size,etc.
}
}
We fetch the file with a XMLHttpRequest().
We parse the file so that it's valid JSON Syntax, the file will now look like this.
entries.json (parsed)
[
{"id":"8f546dcf-b66a-4c53-b3d7-7290429483b8","price":"247.96000000","size":"0.03121005","product_id":"BTC-USD","side":"sell","stp":"dc"},
{"id":"0ec4b63a-b736-42af-a0aa-b4581bf12955","price":"247.90000000","size":"0.03910014","product_id":"BTC-USD","side":"sell","stp":"dc"},
(..)
]
We parse the file into a JSON{Object}
We iterate through the array of objects and access their properties.
Note: If you have control over how the data is saved, you can save the data in the array format instead of parsing it.
Hope it helps! :)

How can I read a .json file without jQuery in JavaScript?

I am only asking the question because I've spent the last 2 days probably reading through countless other questions that are similar and tutorials and still can't get this to work.
I have a local .json file that I want to load up and parse with JavaScript. The file is called 'fakeData.json'. Its format is as such:
{"UIGroup": {"Parent": null, "Type": "public"}}
I'm using this to try to load the file:
<script src="fakeData.json"></script>
I am using this to try to parse the file:
var jsonData = JSON.parse('fakeData.json');
I am getting these errors:
Uncaught SyntaxError: Unexpected token : fakeData.json:1
Uncaught SyntaxError: Unexpected token ILLEGAL : planetPage.html:11
May someone please help me, thanks.

If you want it as simple as it gets, then I would prefix your json content with
var jsonData = {"UIGroup": {"Parent": null, "Type": "public"}}....
which turns your file into a valid js file and then load it with
<script src="fakeData.json.js"></script>
after that, jsonData will have the required content because of literal object notation.
There is no way that you can load a json file into your page otherwise without ajax/httprequest.

You need to get the JSON text into a string variable before you can parse it.
This is generally achieved using the XMLHttpRequest object.
<script src="fakeData.json"></script> will attempt to parse the JSON as JavaScript. This will either throw an error (as it does in your case) or create a data structure and then immediately discard it as it isn't assigned there.

var jsonData;
function reqListener () {
jsonData = JSON.parse(this.responseText);
console.log(jsonData);
};
var oReq = new XMLHttpRequest();
oReq.onload = reqListener;
oReq.open("get", "fakeData.json", true);
oReq.send();

If you want to keep it all inline, with a JSON string, and access the object directly you could do something like this:
// copy all the JSON text from the file an keep store it in a local string variable.
var jsonString = '{ "UIGroup": { "Parent": null, "Type": "public" }}';
// parse the string into a JavaScript object
var jsonObj = JSON.parse(jsonString);
// access the object as you usually would
alert(jsonObj.UIGroup.Type);
Fiddle

Django/JS: json.dumps and parse.json

I'm no programmer, so I can't go to source code of Django or Jquery and figure out how and why these function don't return what I want from them, because I simply wouldn't understand the source code.
I do one little project for myself and here's my confusion about json part:
here's my django/python function:
def searchPatients(request):
patients = Patients.objects.filter(KeyName__icontains=request.POST.get('KeyName'))
response = []
for patient in patients:
tmpvar = {}
tmpvar = { 'Name1':patient.Name1, 'Name2':patient.Name2 }
response.append(tmpvar)
return HttpResponse(json.dumps(response), content_type="application/json")
I checked in shell, json.dumps(response) gave me this:
'[{"Name2": "TestName2", "Name1": "TestName1"}, {"Name2": "TempName2", "Name1": "TempName1"}]'
Looks ok form me. And then I don't understand part starts. This is my JS/JQuery function:
input_newRecord_Search.keyup(function() {
$.post('/edit/ajax_search_patients', { KeyName: $(this).val() }, function(data) {
var patients = jQuery.parseJSON(data);
for (var patient in patients) {
$('#searchResults ul').append('<li>'+patients[patient].Name1+'</li><li>+'patients[patient].Name2+'</li>');
};
}, "json");
});
I get an error: "SyntaxError: JSON.parse: unexpected character".
I checked what data jquery gets from server: console.log(data):
[{Name2: "TestName2", Name1: "TestName1"}, {Name2: "TempName2", Name1: "TempName1"}]
So, as far as I know JSON syntax looks like - {"key":"value"} and I'm missing quotes on key field. And I don't understand why I'm missing them. I can put them manually through regex, for instance, but I don't think it's the right way. And using regex I can parse my entire data without need of jQuery.parseJSON(), but again I want to use jQuery function - after all it was made exactly for this purpose.
Can anyone help me with this one?

The trick is that when you tell jQuery.post that the server is returning JSON it parses it for you.
// This line can be safely removed;
// jQuery is doing it for you behind the scenes
var patients = jQuery.parseJSON(data);
When you use parseJSON on the already parsed data you wind up trying to parse the string representation of a JavaScript object. Simply use the already parsed data and everything should work correctly.

jQuery is automatically converting the json to js objects for you. You don't need to call parse yourself.

Json object in jquery can't be read?

I am trying to read the finance info from the google page into a json object.
Code is below:
try {
$.getJSON("http://finance.google.com/finance/info?client=ig&q=NSE:GOLDBEES&jsoncallback=?",function(data){
alert(data);//var jsondata = data;
//jsonobj = $.parseJSON(jsondata);
//alert(jsonobj[0].id);
});
} catch(e) {
alert(e.toString());
}
However I keep getting this error all the time on firebug
invalid label
"id": "4052464"
Is there any way this info can be read. My ultimate goal is to create a windows 7 gadget that doesnt use server side scripting and can be used from any Windows 7 system.
Appreciate all the help.
John

Response isn't valid JSON (response is prefixed with //), so jQuery won't be able to parse it correctly anyway.
To solve change &jsoncallback=? to &callback=?
so
$.getJSON("http://finance.google.com/finance/info?client=ig&q=NSE:GOLDBEES&callback=?", function(data) {
alert(data)
});

The response from Google has two leading /'s, making the response invalid JSON... for some reason.
Because of this, you cannot use jQuery.getJSON, as it expects a JSON response. Instead, you should use jQuery.get, and parse the JSON yourself after removing the two leading slashes.
jQuery.get('http://finance.google.com/finance/info?client=ig&q=NSE:GOLDBEES&jsoncallback=?', function (string) {
var validJson = string.slice(2);
var obj = jQuery.parseJSON(validJSON);
// use obj
});
Two additional points:
No JSONP is being used, so you don't need the jsoncallback=? in your request URL
The Windows Sidebar has been retired, so you cannot publish you finished gadget to the official gallery.