Scraping AJAX requests with random strings appended to URL - javascript

I am trying to monitor cricket scores on scorespro/cricket by making browser AJAX requests. Analysing the network traffic in Google Chrome, I can see my browser making requests of the form:
http://www.scorespro.com/cricket/ajax.php?g_sort=league&date=2014-10-02&mut=1412265716&sut=0&(some_random_number)
When I click on the response IN Google Chrome, I can see the data that has been received. However when I try to request the request URL myself, no data is received. Why is that happening (is it to do with the random string) and how can I get around it?

Is doing this from javascript a requirement? Have you considered abstracting the requests by calling a script on a server you control?
For example on your server you could have a PHP script called, for example, "grabber.php"
<?php
$r = '0.' . rand(1000000000000000, 9000000000000000);
$url = 'http://www.scorespro.com/cricket/ajax.php?g_sort=league&date=2014-10-03&mut=1412328280&sut=0&' . $r;
$useragent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:32.0) Gecko/20100101 Firefox/32.0';
$referer = 'http://www.scorespro.com/cricket/';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_REFERER, $referer);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, '/tmp/cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, '/tmp/cookie.txt');
$response = curl_exec($ch);
curl_close($ch);
$data = array('payload' => $response);
echo json_encode($data);
exit();
?>
You could then call that page via a simple ajax request :
<script src="//code.jquery.com/jquery-1.11.0.min.js"></script>
<script type="text/javascript">
$.ajax({
url: 'http://yourserver.com/grabber.php',
dataType: 'json',
type: 'GET',
success: function(data, textStatus, jqXHR){
if (data['payload']){
alert(data['payload']);
} else {
alert ('oops');
}
}
});
Of course if you went with this approach you'd have to decide how to get the URL's you need to request from the cricket site to the grabber script (i.e. pass them from javascript or get them directly from within the PHP script depending on your requirements)

Related

Soundcloud PUT API returning 401

I am trying to update soundcloud track details using HTTP API with CURL. I am getting 401 Unauthorized error as response eventhough I have passed my Client ID.
PUT https://api.soundcloud.com/tracks/11111111?client_id=12345666666
The response is
{
"errors": [
{
"error_message": "401 - Unauthorized"
}
]
}
Also wondering if I can pass access_token with the request.
If you want to pass your token, simply append "&oauth_token=" + TOKEN_VALUE
https://api.soundcloud.com/tracks/11111111?client_id=12345666666&oauth_token=YOUR_TOKEN
Edited to add example code
Here is an example of PUT using Curl & with PHP for soundcloud auth token. This code is from a working soundcloud project.
$data = array(
'code' => $token,
'client_id' => $this->getClientId(false), // your client ID
'client_secret' => $this->getClientSecret(), // your client secret
'redirect_uri' => $this->getCallback(), // callback URL
'grant_type' => 'authorization_code'
);
$url = "https://api.soundcloud.com/oauth2/token";
try {
$ch = curl_init();
// set options
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
$response = curl_exec($ch);
$header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$header = substr($response, 0, $header_size);
$body = substr($response, $header_size);
// read / process result
curl_close($ch);
} catch(Exception $e) {
// error handling...
}
This may work. Try turning off various CURL options. In my case I had PUT working under PHP5.5 but after migrating to new server with PHP7 the POST operation would fail until I set CURLOPT_SAFE_UPLOAD to false for soundcloud POST operations (file uploads); however after making this change to all my curl inits, my PUT operations failed because they also had safeupload set to false, so I removed this setting from my PUT operations and the PUT operations started working.
In short, try turning off SAFE UPLOAD option for your PUT operations.
More info about this curl option on this post.

Get https website html code by PHP or JAVASCRIPT

I have tried file_get_content and curl but both don't seem to work on the website. I have used both on previous projects.
Website: https://colruyt.collectandgo.be/cogo/nl/zoeken?z=5030
Anyone has a working solution. Been looking and testing for hours now :).
Curl also does not seem to work.
HTTP/1.1 200 OK Content-Length: 5395 Pragma: no-cache Cache-Control: no-cache Content-Type: text/html
Redirects to my own main domain name
I used this code:
<?php
function geturl($url){
(function_exists('curl_init')) ? '' : die('cURL Must be installed for geturl function to work. Ask your host to enable it or uncomment extension=php_curl.dll in php.ini');
$cookie = tempnam ("/tmp", "CURLCOOKIE");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (compatible; CrawlBot/1.0.0)');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT , 5);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
//curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_ENCODING, "");
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); # required for https urls
curl_setopt($ch, CURLOPT_MAXREDIRS, 15);
$html = curl_exec($ch);
$status = curl_getinfo($ch);
curl_close($ch);
if($status['http_code']!=200){
if($status['http_code'] == 301 || $status['http_code'] == 302) {
list($header) = explode("\r\n\r\n", $html, 2);
$matches = array();
preg_match("/(Location:|URI:)[^(\n)]*/", $header, $matches);
$url = trim(str_replace($matches[1],"",$matches[0]));
$url_parsed = parse_url($url);
return (isset($url_parsed))? geturl($url):'';
}
}
return $html;
}
echo geturl("https://colruyt.collectandgo.be/cogo/nl/zoeken?z=5030");
?>
Take a look at request
var request = require('request');
request('https://colruyt.collectandgo.be/cogo/nl/zoeken?z=5030', function (error, response, body) {
console.log(body)
})
This prints the body of the response.

400 bad request in google translate free api using curl

I am trying to use free Google Translate API which is extracted from Firefox's S3 Google Translator addon, ie.
https://translate.google.com/translate_a/single?client=t&sl=auto&
tl=en&hl=en&dt=bd&dt=ex&dt=ld&dt=md&dt=qca&dt=rw&dt=rm&dt=ss&dt=t
&dt=at&ie=UTF-8&oe=UTF-8&otf=2&srcrom=1&ssel=0&tsel=0&q=Hello
in PHP cURL ie.
$isPOST=isset($_POST) && !empty($_POST);
$q=$isPOST ? $_POST['q'] : $_GET['q'];
$url='https://translate.google.com/translate_a/single';
$data='client=t&sl=auto&tl=en&hl=en&dt=bd&dt=ex&dt=ld&dt=md&dt=qca&dt=rw&dt=rm&dt=ss&dt=t&dt=at&ie=UTF-8&oe=UTF-8&otf=2&srcrom=1&ssel=0&tsel=0&q='.$q;
$ch=curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_NOBODY, 0);
curl_setopt($ch, CURLOPT_URL, !$isPOST ? $url.'?'.$data : $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
if($isPOST){
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
}
$return=curl_exec($ch);
curl_close($ch);
I am calling this page using ajax..
$.ajax({
type: text.length>750 ? 'post' : 'get',
url: 'translate.php',
data: 'q='+text,
success: function(d){ alert(d); }
});
but doing this all, I get this response from Google Translate, ie.
Error: 400. That’s an error.
Your client has issued a malformed or illegal request. That’s all we know.
Please, help me solve this error and get the translated text..
I checked your URL in your browser it shows 400 Error. it means illegal request.
try http://www.sitepoint.com/using-google-translate-api-php/ this URL.
<?php
$apiKey = '<paste your API key here>';
$text = 'Hello world!';
$url = 'https://www.googleapis.com/language/translate/v2?key=' . $apiKey . '&q=' . rawurlencode($text) . '&source=en&target=fr';
$handle = curl_init($url);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($handle);
$responseDecoded = json_decode($response, true);
curl_close($handle);
echo 'Source: ' . $text . '<br>';
echo 'Translation: ' . $responseDecoded['data']['translations'][0]['translatedText'];
?>
Sorry, I tried to POST with same code and it worked..
Thank all.
I have the same issue in a Visual Basic 6 Project and Thamaraiselvam's comment guided me to the correct direction..
I was building the URL correctly and if I try it in the browser it works, but in the http component of vb6 it wasn't working. the solution was to simply url_encode the data sent. (putting it in the browser was automatically doing it)
hope this helps someone else.

Curl return html code when ajax request

I using curl to send request to another page and return value to ajax, i have prevented curl redirect to another and just return value to ajax but final result i receive from ajax have contain html code, my code:
php:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://abc.com');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_MAXREDIRS, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($params));
$data = curl_exec($ch);
curl_close($ch);
echo 'my value';
ajax:
jQuery.ajax({
type: "POST",
url: "http://pagephp.com",
data: data,
success: function(result) {
alert(result);
}
});
alert will show: <html>...</html> my value
help me
You need to enable CURLOPT_RETURNTRANSFER to true/1 then your return value from curl_exec will be the actual result from your successful operation. In other words it will not return TRUE on success. Although it will return FALSE on failure.
http://php.net/manual/en/function.curl-exec.php
You should enable the CURLOPT_FOLLOWLOCATION option for redirects but this would be a problem if your server in in safe_mode and/or open_basedir in effect which can cause issues with curl as well.
header('Content-type: application/json');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://abc.com');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_MAXREDIRS, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($params));
$data = curl_exec($ch);
curl_close($ch);
echo json_encode('my value');
JQUERY
jQuery.ajax({
type: "POST",
url: "http://pagephp.com",
data: data,
success: function(result) {
alert(result);
}
});

Javascript to make Post or Get call to a URL and return the text that's on the page

I've been trying to do this all morning I need to make either a POST or a GET call to this
URL http://cabbagetexter.com/send.php
I need it to return the text that's on the page, I know it can't be that difficult but I'm completely code blocked on this one,
I've tried using JQuerys .post and .get functions but I cant seem to return just the text on the page
any help would be appriciated
EDIT:
Ah ok so there's a technical reason for not being able to do it. balls, Thanks for the help guys
(function ($) {
$.ajax({
url: 'http://cabbagetexter.com/send.php',
type: 'text',
success: function (response) {
//do something with the text from the site
}
});
}(jQuery));
Obviously you need to host this script on the URL you are loading because of the same origin policy
You are running into the cross domain limitation. You can only to a request to a page in the same domain.
There is another possibility if you need to post calls to a page on another domain. Let's say your Javascript is being run from index.php. You might create a file called ctexter.php.
ctexter.php would use curl to make the post request to http://cabbagetexter.com/send.php, and would then output the response (the output from) send.php. So, if index.php makes an ajax call to ctexter.php, and ctexter.php is outputting the response from send.php, you have in effect achieved your goal.
You could make the curl post requests with this function:
function post_request($url, $data) {
$output = array();
foreach ($data as $key => $value) {
if(is_object($value) || is_array($value)){
$data[$key] = serialize($value);
}
}
$output = array();
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
$result = curl_exec($ch);
if ($result) {
$output['status'] = "ok";
$output['content'] = $result;
} else {
$output['status'] = "failure";
$output['error'] = curl_error($ch);
}
curl_close($ch);
return $output;
}
where $url is (obviously) the url to post to, and $data is an associative array containing the data you want to submit.
Then on ctexter.php you could do something like:
// Since we already built the post array in the
// ajax call, we'll just pass it right through
$response = post_request("http://cabbagetexter.com/send.php", $_POST);
if($response['status'] == "ok"){
echo $response['content'];
}
else{
echo "It didn't work";
}
Finally, hit ctexter.php using JQuery .post():
$.post("ctexter.php",
{
firstparamname: "firstparamvalue",
somethingelse: "llama"
},
function(data) {
alert("It worked! Data Loaded: " + data);
});
If you're on the same domain, you'd use some code like this:
var ajax = new XMLHttpRequest();
ajax.onreadystatechange=function()
{
if (ajax.readyState==4 && ajax.status==200)
{
document.getElementById("targetElementID").textContent = ajax.responseText;
}
}
ajax.open("GET","http://cabbagetexter.com/send.php",true);
ajax.send();
Learn how to use AJAX
If not, then, sorry, you're out of luck because you'll run into the same origin policy error.
There is a way to make a request to that URL and get around the same origin policy. Put something like a PHP script on your own domain that makes a request to http://cabbagetexter.com/send.php and then call your own script from the javascript.
If your host supports PHP and cURL a script like this would do the job:
<?php
$url="http://cabbagetexter.com/send.php";
$post="";
if(strstr($url,"?")){
$split=explode("?",$url);
$url=$split[0];
$post=$split[1];
}
$ch = curl_init ($url);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_HEADER, 0);
if($post!=""){
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
}else{
curl_setopt($ch, CURLOPT_POST, 0);
}
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$result = curl_exec($ch);
curl_close($ch);
print $result;
?>

Categories

Resources