Develop Reference

Parse from javascript var with SimpleHTMLDom

I have this code that outputs me source page of source URL with curl!
$url = 'http://source-page.com';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // add this one, it seems to spawn redirect 301 header
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13'); // spoof
$output = curl_exec($ch);
curl_close($ch);
$html = str_get_html($output);
In $output i have this:
var flashvars = {
"image_url":"http://path-to-image.com",
"video_title":"This is video title",
"videoUrl":"http://this-is-path-to-mp4.com"
}
I want to echo videoUrl and I have tried with this:
$videoUrl = $html->find('flashvars[0].videoUrl');
echo $videoUrl
And is giving me empty results. What is a good code for doing that?

Someone else suggessted regex + json_decode and then deleted it.
Here's what I would do:
$output = <<<EOF
var flashvars = {
"image_url":"http://path-to-image.com",
"video_title":"This is video title",
"videoUrl":"http://this-is-path-to-mp4.com"
}
EOF;
$str = preg_match('/var flashvars = (\{.*?\})/s', $output, $m);
$data = json_decode($m[1], true);
echo $data['videoUrl'];

Scraping AJAX requests with random strings appended to URL

I am trying to monitor cricket scores on scorespro/cricket by making browser AJAX requests. Analysing the network traffic in Google Chrome, I can see my browser making requests of the form:
http://www.scorespro.com/cricket/ajax.php?g_sort=league&date=2014-10-02&mut=1412265716&sut=0&(some_random_number)
When I click on the response IN Google Chrome, I can see the data that has been received. However when I try to request the request URL myself, no data is received. Why is that happening (is it to do with the random string) and how can I get around it?

Is doing this from javascript a requirement? Have you considered abstracting the requests by calling a script on a server you control?
For example on your server you could have a PHP script called, for example, "grabber.php"
<?php
$r = '0.' . rand(1000000000000000, 9000000000000000);
$url = 'http://www.scorespro.com/cricket/ajax.php?g_sort=league&date=2014-10-03&mut=1412328280&sut=0&' . $r;
$useragent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:32.0) Gecko/20100101 Firefox/32.0';
$referer = 'http://www.scorespro.com/cricket/';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_REFERER, $referer);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, '/tmp/cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, '/tmp/cookie.txt');
$response = curl_exec($ch);
curl_close($ch);
$data = array('payload' => $response);
echo json_encode($data);
exit();
?>
You could then call that page via a simple ajax request :
<script src="//code.jquery.com/jquery-1.11.0.min.js"></script>
<script type="text/javascript">
$.ajax({
url: 'http://yourserver.com/grabber.php',
dataType: 'json',
type: 'GET',
success: function(data, textStatus, jqXHR){
if (data['payload']){
alert(data['payload']);
} else {
alert ('oops');
}
}
});
Of course if you went with this approach you'd have to decide how to get the URL's you need to request from the cricket site to the grabber script (i.e. pass them from javascript or get them directly from within the PHP script depending on your requirements)

Scraping data in dynamic sites

I'm trying to scrape data from our local government. What I want is address from kids adoption offices. Here, in Brazil, all adoptions go through the government. So I have the URL of one office, there are 2 or 3 thousands more. But if I can manage to get one, the others will be easy.
I made many attempts, bellow I show three.
The problem could be related to a Javascript (Ajax maybe) that refresh the page.
Note: I am not a PHP developer.
First attempt
echo '<html><head></head><body>';
echo '<h1>Scraper PHP GET 1</h1>';
echo ini_get("allow_url_fopen");
echo ini_get("allow_url_fopen");
// I used this url for test
//$url = 'http://www.portaldaadocao.com.br';
//This is the URL that I really want
$url = 'http://www.cnj.jus.br/cna/Controle/ConsultaPublicaBuscaControle.php?transacao=CONSULTA&vara=2673';
$html = file_get_contents($url);
var_dump($html);
echo '</body></html>';
// Output
// 11
// Warning:
file_get_contents(http://www.cnj.jus.br/cna/Controle/ConsultaPublicaBuscaControle.php?
transacao=CONSULTA&vara=2673) [function.file-get-contents]: failed to open stream: HTTP
request failed! HTTP/1.1 404 Not Found in /home/rsl/www/sc01_get.php on line 14
// bool(false)
Second attempt
echo '<html><head></head><body>';
echo '<h1>Scraper PHP CURL 3</h1>';
// I used this url for test
//$url = 'http://www.portaldaadocao.com.br';
//This is the URL that I really want
$url = 'http://www.cnj.jus.br/cna/Controle/ConsultaPublicaBuscaControle.php?transacao=CONSULTA&vara=2673';
$curl = curl_init($url);
#curl_setopt($curl, CURLOPT_POSTFIELDS, "foo");
#curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
#curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "POST");;
$html=#curl_exec($curl);
if (!$html) {
echo "<br />cURL error number:" .curl_errno($curl);
echo "<br />cURL error:" . curl_error($curl);
exit;
}
else{
echo '<br>begin HTML[';
echo $html;
echo '<br>]end html ';
}
echo '</body></html>';
// Output
// 1
third attempt
function curl($url){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.6 (KHTML, like Gecko) Chrome/16.0.897.0 Safari/535.6');
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_REFERER, "http://www.windowsphone.com");
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
echo '<html><head></head><body>';
echo '<h1>Scraper PHP CURL 5</h1>';
// I used this url for test
//$url = 'http://www.portaldaadocao.com.br';
//This is the URL that I really want
$url = 'http://www.cnj.jus.br/cna/Controle/ConsultaPublicaBuscaControle.php?transacao=CONSULTA&vara=2673';
$curl = curl_init($url);
#curl_setopt($curl, CURLOPT_POSTFIELDS, "foo");
#curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
#curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "POST");;
$html=#curl($curl);
if (!$html) {
echo "<br />cURL error number:" .curl_errno($curl);
echo "<br />cURL error:" . curl_error($curl);
exit;
}
else{
echo '<br>begin HTML[';
echo $html;
echo '<br>]end html ';
}
echo '</body></html>';
// Output
// cURL error number:0
// cURL error:

javascript mandatory data error in screenleap api

i have a json encoded data in a variable named $json, it looks like-
string(1243) "{"screenShareCode":"882919360",
"appletHtml":"",
"presenterParams":"aUsEN5gjxX/3NMrlIEGpk0=",
"viewerUrl":"http://api.screenleap.com/v2/viewer/882919360?accountid=mynet",
"origin":"API"}"
}
i need to pass this json data into javascript function, please see below
script type="text/javascript" src="http://api.screenleap.com/js/screenleap.js">/script>
script type="text/javascript">
window.onload = function() {
var screenShareData = '?php echo $json;?>';
screenleap.startSharing('DEFAULT', screenShareData);
};
/script>
when i am trying to run this code it is giving me an error saying "missing mandatory screen share data".
How to solve this error?
i am following "https://www.screenleap.com/api/presenter"

It looks like $json is a string, you need to pass in a json object. Try the following:
window.onload = function() {
var screenShareData = '?php echo $json;?>';
screenleap.startSharing('DEFAULT', JSON.parse(screenShareData));
};

This is how you implement it based on the documentation
https://www.screenleap.com/api/presenter
<?php
// Config
$authtoken = '';
$accountid = '';
// 1. Make CURL Request
$url = 'https://api.screenleap.com/v2/screen-shares';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('authtoken:<authtoken>'));
curl_setopt($ch, CURLOPT_POSTFIELDS, 'accountid=<accountid>');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($ch);
curl_close($ch);
$json = json_decode($data, true);
?>
<!-- 2. Launch the Presenter App -->
<script type="text/javascript" src="http://api.screenleap.com/js/screenleap.js"></script>
<script type="text/javascript">
window.onload = function() {
screenleap.startSharing('DEFAULT', JSON.parse('<?php echo $json; ?>'));
};
</script>
If this doesn't work, you got to report it to screenleap.

You should only need to actually parse the JSON if you want to access the values. Otherwise, just pass the response data right into the startSharing function, like this:
<?php
$url = 'https://api.screenleap.com/v2/screen-shares';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('authtoken:<your authtoken>'));
curl_setopt($ch, CURLOPT_POSTFIELDS, 'accountid=<your accountid>');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$data = curl_exec($ch);
curl_close($ch);
$json = json_decode($data, true);
?>
<script type="text/javascript" src="http://api.screenleap.com/js/screenleap.js"></script>
<script type="text/javascript">
window.onload = function() {
screenleap.startSharing('DEFAULT', <?php echo $data; ?>);
};
</script>
If you just insert your own accountid and authtoken (without leading spaces), that should work for you.

Curl return html code when ajax request

I using curl to send request to another page and return value to ajax, i have prevented curl redirect to another and just return value to ajax but final result i receive from ajax have contain html code, my code:
php:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://abc.com');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_MAXREDIRS, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($params));
$data = curl_exec($ch);
curl_close($ch);
echo 'my value';
ajax:
jQuery.ajax({
type: "POST",
url: "http://pagephp.com",
data: data,
success: function(result) {
alert(result);
}
});
alert will show: <html>...</html> my value
help me

You need to enable CURLOPT_RETURNTRANSFER to true/1 then your return value from curl_exec will be the actual result from your successful operation. In other words it will not return TRUE on success. Although it will return FALSE on failure.
http://php.net/manual/en/function.curl-exec.php
You should enable the CURLOPT_FOLLOWLOCATION option for redirects but this would be a problem if your server in in safe_mode and/or open_basedir in effect which can cause issues with curl as well.
header('Content-type: application/json');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://abc.com');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_MAXREDIRS, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($params));
$data = curl_exec($ch);
curl_close($ch);
echo json_encode('my value');
JQUERY
jQuery.ajax({
type: "POST",
url: "http://pagephp.com",
data: data,
success: function(result) {
alert(result);
}
});