How to scrape a JavaScript value in PHP - javascript

I’m new to PHP and coding in general and I can’t figure this out. I’m trying to get the number of kills from this profile page.
At the moment, the string I am trying to get is:
29362
When I view the page source, this number is nowhere to be seen.
When I use inspect element, however, I find:
<td class="num">29362</td>
How can I get the content shown in inspect element instead of the content shown by viewing the page source?

In using a tool like Firebug for Firefox, or the inspector for Safari and Chrome, you can see that at page load a series of AJAX requests are made for data. Though I didn't dig through all of the data returned by those requests, I do see the data you're looking for in at least one of them:
http://uberstrike.com/profile/all_stats/631163
So at page load JavaScript makes a series of AJAX requests back to the server to get all the data, then it manipulates the DOM to insert it all into the view.
If you wanted, your PHP could directly request the URL I pasted above and json_decode the response. This would produce a data structure for you to use which includes that number in the kills_all_time property.
Quick and dirty example:
<?php
$data_url = 'http://uberstrike.com/profile/all_stats/631163';
$serialized_data = file_get_contents($data_url);
$data = json_decode($serialized_data, true);
var_dump($data['kills_all_time']);

I looked and it looks like there is no API currently, so your best method will be to do an inter-web-server http request. Get the page you want and then it is a lot of string math from there.
I would recommend using string search to find <td class="name">Kills</td> and the kills row will appear right after it. From there its simply extracting the number using string math.

To add to what JAAulde has explained, it seems like there is a method to these AJAX requests. And they are all based on the profile ID that can be found at the end of the URL:
http://uberstrike.com/public_profile/631163
Then in the Safari debugger (which is what I am using) you can see these XHR (XMLHttpRequest) requests which are directly connected to API calls:
Then looking at the data in them shows some really nicely formatted JSON. Great! No scraping! So just go through these URLs to see what you can see:
http://uberstrike.com/profile/items
http://uberstrike.com/profile/user_info/631163
http://uberstrike.com/profile/user_loadout/631163
http://uberstrike.com/profile/all_stats/631163
And looking at the all_stats endpoint shows:
"kills_all_time":29362,
Nice!
So now let’s use some PHP json_decodeing like this:
// Set the URL to the data.
$url = 'http://uberstrike.com/profile/all_stats/631163';
// Get the contenst of the URL via file_get_contents.
$all_stats_json = file_get_contents($url);
// Decode the JSON string with the 'true' optionso we get any array.
$all_stats_json_decoded = json_decode($all_stats_json, true);
// Dump the results for testing.
echo '<pre>';
print_r($all_stats_json_decoded);
echo '</pre>';
Which will dump an array like this:
Array
(
[headshots_record] => 24
[nutshots_record] => 33
[damage_dealt_record] => 6710
[damage_received_record] => 31073
[kills_record] => 50
[smackdowns_record] => 45
[headshots_all_time] => 4299
[nutshots_all_time] => 1925
[kills_all_time] => 29362
[deaths_all_time] => 16491
…
Now to get kills_all_time just do this:
// Return the 'kills_all_time'.
echo $all_stats_json_decoded['kills_all_time'];
Which gives us:
29362

Related

How to change : Fullcalendar generates GET request

I am implementing fullcalendar on my website.
I have created a div with calendar as id.
I have put data from a SQL query in a php variable and used json encode to get the right format.
This is how I create the calendar :
$html .= "<script src='/wp-content/plugins/biobelt/moment.min.js'></script>
<script src='/wp-content/plugins/biobelt/fullcalendar.min.js'></script>
<link rel= 'stylesheet' href='/wp-content/plugins/biobelt/fullcalendar.css' type='text/css'>
<script>
jQuery(document).ready(function() {
var bevents = '".$buildingevents."'
console.log(bevents)
jQuery('#calendar').fullCalendar({
defaultDate: '" . $_SESSION['statDateFrom'] ."',
editable: true,
events: bevents,
});
});
</script>";
The console log gives me an output of the array that I am passing to events, and it is the correct format :
[{"id":"1","titre":"1","start":"2018-04-09 07:00:01","stop":"2018-04-09 11:00:00"},{"id":"2","titre":"1","start":"2018-04-09 07:00:01","stop":"2018-04-09 11:00:00"},{"id":"3","titre":"2","start":"2018-04-09 16:00:01","stop":"2018-04-09 21:00:00"},{"id":"4","titre":"2","start":"2018-04-09 16:00:01","stop":"2018-04-09 21:00:00"}, etc...
What I get from this is :
GET 403 Forbidden Error
I checked in apache logs, this is because the URL is too long since every field of the array is put into the url.
For some reason I don't want to change the limit request line in apache conf file.
I want to generate a POST instead of GET request.
And I would like to know how it generates a GET request since I didn't put GET anywhere in my files.
EDIT :
according to : https://fullcalendar.io/docs/events-json-feed
Fullcalendar create the get request and the URL. The problem persists since the URL is too long and I want to create a POST request instead. How to do that?
You seem to have misunderstood the documentation somewhat.
You said
The console log gives me an output of the array that I am passing to events, and it is the correct format
And indeed what you've showed does look like a Javascript array. So...it's a static array and not a URL. You do not have any kind of server endpoint to which you can make a separate ajax call to get your events. Therefore the article you linked to (https://fullcalendar.io/docs/events-json-feed) is not relevant. Instead you are providing a static list of events as per the method described at https://fullcalendar.io/docs/events-array).
Except that...you're not. Due to the way you've written your code, you're providing a string instead of an array. That is causing fullCalendar to assume you're providing a URL, and then trying to call that URL, and it's no surprise that it errors.
If you simply remove the single quotes from
var bevents = '".$buildingevents."'
so that it becomes
var bevents = ".$buildingevents.";
then your code should work ok, because this will inject a hard-coded array into the JavaScript instead of a string.

Laravel - load new content from DB using ajax dynamically

I am still learning about how Ajax works and I encountered this problem:
#getDashboard Controller:
$data = Post::latest()->get();
return view('view', ['data'] => $data);
view.blade.php Page:
<div>
#foreach($data as $post)
<p>{{ $post->ad_title }}</p>
#endforeach
</div>
This loads all posts from the database, but new ones appear only on refresh. How can i load new posts from database without refreshing, i suppose using AJAX? Thank you.
Yes, you can do so using ajax.
If you want to return JSON from Laravel controller, you can do so very easily.
Just return the object or array instead of view. Laravel will automatically return JSON as response.
For example:
getDashboardJson(){
return Post::latest()->get();
}
Form Ajax if you call the corresponding URL of this method, you will get json of all latest posts.
What you would want to do is create a function that executes an XMLHTTP Request and then populates your page with the results, then set an interval on that function.
In your PHP, you will want to populate an array with all the data you want in your page. You will then json_encode it and echo it out. Ensure to set your content-type header to application/json.
In your Javascript, you will want to:
XMLHTTP Request the information
If there was an error, then handle it
JSON.Parse it into an array
Validate it (you may return in your array that no information was found)
Update your HTML
There are 2 options I would recommend for your ajax. The first is Javascript's built in XMLHTTPRequest method, which you can read up on here. The alternative is jQuery.ajax, which you can read up on here. Both pages should conrtain enough information for you to hit the ground running but if not, I would be happy to prepare some examples for you.
Good luck.

PHP $_GET and underlines

I have a very short piece of PHP that I use to make HTTP requests from JavaScript.
<?php echo file_get_contents($_GET['url']); ?>
I have used it successfully in a few projects, but am running into a problem with making requests in my current project. Based on my searching, I believe it may be caused by the underscore in the request, though through my searching and not knowing PHP, I have not been able to confirm that.
Below is an example of what I am doing from JavaScript:
$.get("grabber.php?url=" + "http://tidesandcurrents.noaa.gov/api/datagetter?station=8573364&begin_date=20160202&end_date=20160203&product=predictions&units=english&time_zone=gmt&format=json&application=poseidonweathercom+&datum=MLLW", function(forecast) {
console.log(forecast);
});
If I copy the url and put in it in a browser, I get back the JSON that I requested. When I use the code above, I end up getting an error message from NOAA:
Wrong Product : Product cannot be null or empty Wrong Time zone: Time zone cannot be null or empty Wrong Unit:Unit cannot be null or empty Wrong Format: Format cannot be null or empty Wrong Date: The beginDate cannot be null or empty
Do I need to use a regex for the underscore in PHP? Is there some other issue that I do not understand?
Thanks.
You need to send it encoded, which will convert all the underscores/spaces/ampersands etc. with their encoded equivalents:
var url = "http://tidesandcurrents.noaa.gov/api/datagetter?station=8573364&begin_date=20160202&end_date=20160203&product=predictions&units=english&time_zone=gmt&format=json&application=poseidonweathercom+&datum=MLLW";
$.get("grabber.php?url=" + encodeURIComponent(url), function(forecast){
console.log(forecast);
}
Using encodeURIComponent() on that URL shows:
http%3A%2F%2Ftidesandcurrents.noaa.gov%2Fapi%2Fdatagetter%3Fstation%3D8573364%26begin_date%3D20160202%26end_date%3D20160203%26product%3Dpredictions%26units%3Denglish%26time_zone%3Dgmt%26format%3Djson%26application%3Dposeidonweathercom%2B%26datum%3DMLLW
Alternatively, if you just want to access the JSON data and handle it within the JavaScript function, you can retrieve the data via the URL directly, without having to encode the URL:
$.get("http://tidesandcurrents.noaa.gov/api/datagetter?station=8573364&begin_date=20160202&end_date=20160203&product=predictions&units=english&time_zone=gmt&format=json&application=poseidonweathercom+&datum=MLLW", function(forecast) {
console.log(forecast);
});
Um why do you even need your php code ... the code below will work just fine and eliminate your server overhead.
$.get("http://tidesandcurrents.noaa.gov/api/datagetter?station=8573364&begin_date=20160202&end_date=20160203&product=predictions&units=english&time_zone=gmt&format=json&application=poseidonweathercom+&datum=MLLW", function(forecast) {
console.log(forecast);
});

Working with JSON via GET requests in Javascript

I've built a php API that provides data in json output, I need to get the values via a get request to then plot as a graph on the page.
The front end web component in hosted on the same server as in the api in this basic structure:
index.php
graph.php
/api/
/api/src
/api/src/api.php
My current code in graph.php is as follows:
<script>
var myJson;
$.getJson('api/src/api.php/poll/results/current/13/', function(jd){
myJson = jd.AnswerCount.1;
});
document.getElementById('jsonhere').innerHTML = myJson; //just to test
</script>
The endpoint outputs data like the following:
{"AnswerCount":{"1":5,"3":1,"2":2,"4":1,"5":5,"6":3,"7":2}}
Which I need loaded into a key-value pair array,
1:5
3:1
2:2
4:1
...
to then be put into the graphing library.
How do I fix my code/write new code to do this? I'm pretty stuck here.
EDIT:
On a hunch I logged all the get requests via wireshark, and no request is ever sent to the url in question. Even with an empty function { } ? http://grab.kfouwels.com/pmgW
You can't use a number as an identifier, to access the 1 property you have to say [1] not .1
You have to use the variable containing your data, not x which hasn't been mentioned until you try to assign it somewhere
The A in Ajax stands for Asynchronous. You have to work with your data inside your callback since the function you pass to getJson won't be called until the HTTP response arrived but the line starting document.get will run as soon as the HTTP request has been sent.

how to get http request without using XHR object in javascript

Is it possible to get information from URL without using the XmlHttpRequest .
I am trying to get document from my noSqlDatabase from URL
by the XHR object and It did not work . but If I type the URL in the browser I get the data.
If by 'get information' you mean load data from a 3rd party url without using XHR requests, the answer is yes, but with caveats.
One commonly used method to load json data cross-domain is called jsonp. Essentially, you define a function on your page:
var x = function(data){ //do something with data }
Then you create a script tag and append it to you page where the src attribute points to some other url that returns a js file like this:
x({ param: 'some data' });
The x function then executes on your page and has access to the 'param' data in the object. You can also pass a string or number using this method.
There is another method involving iframes and the window.name property of the iframe. This technique is a bit older but still works, but I won't go into a lot of detail about it here. You can read more about it here, http://skysanders.net/subtext/archive/2010/10/11/leveraging-window.name-transport-for-secure-and-efficient-cross-domain-communications.aspx, and other places.

Categories

Resources