Get https website html code by PHP or JAVASCRIPT - javascript

I have tried file_get_content and curl but both don't seem to work on the website. I have used both on previous projects.
Website: https://colruyt.collectandgo.be/cogo/nl/zoeken?z=5030
Anyone has a working solution. Been looking and testing for hours now :).
Curl also does not seem to work.
HTTP/1.1 200 OK Content-Length: 5395 Pragma: no-cache Cache-Control: no-cache Content-Type: text/html
Redirects to my own main domain name
I used this code:
<?php
function geturl($url){
(function_exists('curl_init')) ? '' : die('cURL Must be installed for geturl function to work. Ask your host to enable it or uncomment extension=php_curl.dll in php.ini');
$cookie = tempnam ("/tmp", "CURLCOOKIE");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (compatible; CrawlBot/1.0.0)');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT , 5);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
//curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_ENCODING, "");
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); # required for https urls
curl_setopt($ch, CURLOPT_MAXREDIRS, 15);
$html = curl_exec($ch);
$status = curl_getinfo($ch);
curl_close($ch);
if($status['http_code']!=200){
if($status['http_code'] == 301 || $status['http_code'] == 302) {
list($header) = explode("\r\n\r\n", $html, 2);
$matches = array();
preg_match("/(Location:|URI:)[^(\n)]*/", $header, $matches);
$url = trim(str_replace($matches[1],"",$matches[0]));
$url_parsed = parse_url($url);
return (isset($url_parsed))? geturl($url):'';
}
}
return $html;
}
echo geturl("https://colruyt.collectandgo.be/cogo/nl/zoeken?z=5030");
?>

Take a look at request
var request = require('request');
request('https://colruyt.collectandgo.be/cogo/nl/zoeken?z=5030', function (error, response, body) {
console.log(body)
})
This prints the body of the response.

Related

PHP cURL request to JavaScript

I need to convert this PHP cURL request to JavaScript. I'm not exactly sure how to go about doing it.
I basically need to make this request be client sided instead of server sided.
function createSite($template_id) {
//create array with data
$data = array("template_id"=>$template_id);
//turn data into json to pass via cURL
$data = json_encode($data);
print $data;
//Set cURL parameters
$ch = curl_init();
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch,
CURLOPT_URL,'https://api.duda.co/api/sites/multiscreen/create');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERPWD, API_USER.':'.API_PASS);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type:
application/json'));
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
//execute cURL call and get template_idte data
$output = curl_exec($ch);
//check for errors in cURL
if(curl_errno($ch)) {
die('Curl error: ' . curl_error($ch));
}
//decode result
$output = json_decode($output);
//return unique site_name
return $output->site_name;

close php cURL only after Ajax complete

A have cURL login to a network router which is triggered by ajax. But I need to keep the connection alive until an xhr request is finished on success.
Is this somehow possible?
To clarify here is what I want to do, here is my Ajax function for the login:
function logintorouter(){
$.ajax({
type: 'GET',
dataType:"text",
url: 'login.php'
}).done(function(data){
if(data.includes('WiFi')){
setWiFichannel(); /*Wait for this function to finish before closing cURL connection */
}else{
alert('something went wrong, try again later!');
}
}).fail(function(data){
alert('can not connect to router');
});
}
and my login.php looks like this now:
<?php
function openRouter(){
$username = 'admin';
$password = 'pass';
$url="http://10.1.10.1/login/Auth";
$postinfo = "username=".$username."&password=".$password;
$cookie_file_path = "cookie.txt";
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_NOBODY, false);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
curl_setopt($ch, CURLOPT_COOKIE, "cookiename=0");
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:56.0) Gecko/20100101 Firefox/56.0");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_REFERER, $_SERVER['REQUEST_URI']);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postinfo);
curl_exec($ch);
curl_setopt($ch, CURLOPT_URL, "http://10.1.10.1/main.html#WiFi");
$mainPage = curl_exec($ch);
curl_close($ch);
/*As you can see I close the connection here so the setWiFichannel() will fail
but I don't want to keep the connection alive forever*/
return $mainPage;
}
echo openRouter();
?>

Passing a cookie using cURL to a website

Some website I need to make a request to, using cURL in PHP, contains a captcha, which can be deactivated via setting a cookie "downloadcaptcha=1". Now how do I pass that cookie to the website on a cURL request? Before, you mark as duplicate, I've already made research and came across to using cookie jar files. I have never done something like that and couldn't find any newbie-friendly documentation.
The JS-equivalent code of setting the cookie, taken from the website's function that disables captchas:
var exdate=new Date();
var exdays = 1;
var c_name = "downloadcaptcha";
exdate.setDate(exdate.getDate() + exdays);
var c_value=escape(1) + ((exdays==null) ? "" : "; expires="+exdate.toUTCString());
document.cookie=c_name + "=" + c_value+";path=/";
window.location.reload();
I don't want to necessarily make it expire in 1 day.
Help is appreciated.
Here's the output. Not a single mention of the cookie
Request Header is missing the Cookie:
Host: www.emuparadise.me
Accept: */*
Accept-Encoding: deflate, gzip
Referer: https://m.emuparadise.me/
Change the $request to:
$cookie = '[downloadcaptcha=1'
$request = array();
$request[] = "Host: www.emuparadise.me";
$request[] = "Accept: */*";
$request[] = "Accept-Encoding: deflate, gzip";
$request[] = "Referer: https://m.emuparadise.me/";
$request[] = "Cookie: $cookie";
I have tried that already but the website still asks for a captcha.
Take a look at the JS code I posted. It's taken from the website's
function to disable captchas
I could help you much better if I had a URL to test with.
If there is a redirect you must use curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
You may have to make the HTTP Request Header look identical to the JS HTTP header.
You need to look at the cURL HTTP request and response and compare it to the JS HTTP request and response.
To see the JS HTTP request and response:
right click the page
Select Inspect (Chrome) or Insect Element (FireFox)
Select "Network" tab
Go to the page with the JS request or if already there Refresh.
Trouble shooting PHP code:
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_HEADER, true);
$data = curl_exec($ch);
if (curl_errno($ch)){
$data .= 'Retreive Base Page Error: ' . curl_error($ch);
}
else {
$info = rawurldecode(var_export(curl_getinfo($ch),true));
$skip = intval(curl_getinfo($ch, CURLINFO_HEADER_SIZE));
$responseHeader= substr($data,0,$skip);
$response = substr($data,$skip);
var_export($info);
echo "----------------------------------\n$responseHeader\n--------------------------\n$response \n------------------------------------\n";
}
Original Post
curl_setopt($ch, CURLOPT_COOKIE, $cookie );
Single Cookie
$cookie = 'CookieName=CookieValue'
Multiple Cookies separated by semi-colon. There is no semi-colon after the last cookie.
$cookie = 'CookieName=CookieValue; CookieName=CookieValue'
curl_setopt($ch, CURLOPT_COOKIE, $cookie );
Cookies can also be set using CURLOPT_HTTPHEADER
See last line of the following example.
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
$request = array();
$request[] = "Host: www.example.com";
$request[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$request[] = "User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0";
$request[] = "Accept-Language: en-US,en;q=0.5";
$request[] = "Connection: keep-alive";
$request[] = "Cache-Control: no-cache";
$request[] = "Pragma: no-cache";
$request[] = "Cookie: $cookie";

PHP curl post login details

I try to login on axs.com using PHP curl but my script doesn't work.
I want small help to auth this page via PHP.
Is there error in my code?
$username = 'aUsername';
$password = 'aPassword';
$loginUrl = 'https://www.axs.com/login_check';
/* EDIT EMAIL AND PASSWORD */
$EMAIL = $username;
$PASSWORD = $password;
function cURL($url, $header=NULL, $cookie=NULL, $p=NULL)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, $header);
curl_setopt($ch, CURLOPT_NOBODY, $header);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_COOKIE, $cookie);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
if ($p) {
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $p);
}
$result = curl_exec($ch);
if ($result) {
return $result;
} else {
return curl_error($ch);
}
curl_close($ch);
}
$a = cURL($loginUrl,true,null,"_username=$EMAIL&_password=$PASSWORD");
preg_match('%Set-Cookie: ([^;]+);%',$a,$b);
$c = cURL($loginUrl,true,null,"_username=$EMAIL&_password=$PASSWORD");
preg_match_all('%Set-Cookie: ([^;]+);%',$c,$d);
for($i=0;$i<count($d[0]);$i++)
$cookie.=$d[1][$i].";";
/*
NOW TO JUST OPEN ANOTHER URL EDIT THE FIRST ARGUMENT OF THE FOLLOWING FUNCTION.
TO SEND SOME DATA EDIT THE LAST ARGUMENT.
*/
echo cURL("http://www.axs.com/",null,$cookie,null);
?>
Is this possible on this web site?

400 bad request in google translate free api using curl

I am trying to use free Google Translate API which is extracted from Firefox's S3 Google Translator addon, ie.
https://translate.google.com/translate_a/single?client=t&sl=auto&
tl=en&hl=en&dt=bd&dt=ex&dt=ld&dt=md&dt=qca&dt=rw&dt=rm&dt=ss&dt=t
&dt=at&ie=UTF-8&oe=UTF-8&otf=2&srcrom=1&ssel=0&tsel=0&q=Hello
in PHP cURL ie.
$isPOST=isset($_POST) && !empty($_POST);
$q=$isPOST ? $_POST['q'] : $_GET['q'];
$url='https://translate.google.com/translate_a/single';
$data='client=t&sl=auto&tl=en&hl=en&dt=bd&dt=ex&dt=ld&dt=md&dt=qca&dt=rw&dt=rm&dt=ss&dt=t&dt=at&ie=UTF-8&oe=UTF-8&otf=2&srcrom=1&ssel=0&tsel=0&q='.$q;
$ch=curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_NOBODY, 0);
curl_setopt($ch, CURLOPT_URL, !$isPOST ? $url.'?'.$data : $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
if($isPOST){
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
}
$return=curl_exec($ch);
curl_close($ch);
I am calling this page using ajax..
$.ajax({
type: text.length>750 ? 'post' : 'get',
url: 'translate.php',
data: 'q='+text,
success: function(d){ alert(d); }
});
but doing this all, I get this response from Google Translate, ie.
Error: 400. That’s an error.
Your client has issued a malformed or illegal request. That’s all we know.
Please, help me solve this error and get the translated text..
I checked your URL in your browser it shows 400 Error. it means illegal request.
try http://www.sitepoint.com/using-google-translate-api-php/ this URL.
<?php
$apiKey = '<paste your API key here>';
$text = 'Hello world!';
$url = 'https://www.googleapis.com/language/translate/v2?key=' . $apiKey . '&q=' . rawurlencode($text) . '&source=en&target=fr';
$handle = curl_init($url);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($handle);
$responseDecoded = json_decode($response, true);
curl_close($handle);
echo 'Source: ' . $text . '<br>';
echo 'Translation: ' . $responseDecoded['data']['translations'][0]['translatedText'];
?>
Sorry, I tried to POST with same code and it worked..
Thank all.
I have the same issue in a Visual Basic 6 Project and Thamaraiselvam's comment guided me to the correct direction..
I was building the URL correctly and if I try it in the browser it works, but in the http component of vb6 it wasn't working. the solution was to simply url_encode the data sent. (putting it in the browser was automatically doing it)
hope this helps someone else.

Categories

Resources