crawling page does only work for 3% - javascript

I am trying to crawl a full section of a website, but the problem is that the data that I need is not there from the start. Is there anyway to get the data from the website with PHP?
this is the link: https://www.iamsterdam.com/nl/uit-in-amsterdam/uit/agenda and this is the section I need:
After my post was set to duplicate I tried this
https://stackoverflow.com/a/28506533/7007968 but is also doesn't work so I need a other solucion this is what I tried:
get-website.php
$phantom_script= 'get-website.js';
$response = exec ('phantomjs ' . $phantom_script);
echo $response;
get-website.js
var webPage = require('webpage');
var page = webPage.create();
page.open('https://www.iamsterdam.com/nl/uit-in-amsterdam/uit', function(status) {
console.log(page.content);
phantom.exit();
});
this is all I get back (around 3% of the page):
</div><div id="ads"></div><script src="https://analytics.twitter.com/i/adsct?p_id=Twitter&p_user_id=0&txn_id=nvk6a&events=%5B%5B%22pageview%22%2Cnull%5D%5D&tw_sale_amount=0&tw_order_quantity=0&tw_iframe_status=0&tpx_cb=twttr.conversion.loadPixels" type="text/javascript"></script></body></html>
So I have the feeling that i am getting closer this is what I after a lot of searching:
var webPage = require('webpage');
var page = webPage.create();
var settings = {
operation: "POST",
encoding: "utf8",
headers: {
"Content-Type": "application/json"
},
data: JSON.stringify({
DateFilter: 04112016,
LastMinuteTickets: 0,
PageId: "3418a37d-b907-4c80-9d67-9fec68d96568",
Skip: 0,
Take: 12,
ViewMode: 1
})
};
page.open('https://www.iamsterdam.com/api/AgendaApi/', settings, function(status) {
console.log(page.content);
phantom.exit();
});
But what I get back doesn't look good:
Message":"An error has occurred.","ExceptionMessage":"Page could not be found","ExceptionType":"System.ApplicationException","StackTrace":" at Axendo.SC.AM.Iamsterdam.Controllers.Api.AgendaApiController.GetResultsInternal(RequestModel requestModel)\r\n at lambda_method(Closure , Object , Object[] )\r\n
etc.
I hope somewann can help me,

Addressing your main question about 3%.
You use exec incorrectly. When used like this
$response = exec ('phantomjs ' . $phantom_script);
$response will containt the last line of what was printed in terminal during execution of a given command. Because you did console.log(page.contents); the last line of HTML document was placed into $response variable.
The correct use of exec would be
exec ('phantomjs ' . $phantom_script, $response);
This way the result will be placed into $response variable as an array, with each line an element of the array. Then, if you just want to get html, you can do
$html = implode("\n", $response);
But a more simple and correct way is to use the specific function for the task:
passthru ('phantomjs ' . $phantom_script);
passthru executes a function and returns recieved data unmodified, straight to the output.
So if you want to contain it to a variable, do:
ob_start();
passthru ('phantomjs ' . $phantom_script);
$html = ob_get_clean();

Related

Server-sent events not working in Google Chrome

I am trying to update a specific element in my HTML document using server-sent events. I am using the W3Schools example with parameters changed to fit my needs. When this did not work out of the box, I tried this question's solution, which is to include charset=utf-8 in my Content-Type header. Still no updates to the HTML. I then realized I was getting the following error:
EventSource's response has a MIME type ("text/html") that is not "text/event-stream". Aborting the connection.
Here is the relevant JavaScript in my .js file:
var source = new EventSource("warehouse.php");
source.onmessage = function(event) {
document.getElementById("processing").innerHTML += event.data + "<br>";
};
In my PHP file, I have the following:
if (isset($args->products)) {
orderProducts($args->products);
}
function orderProducts($products) {
$bigResponse = outgoingData(constant('bigURL'), $products[0]);
$littleResponse = outgoingData(constant('lilURL'), $products[3]);
header('Content-Type: text/event-stream; charset=utf-8');
header('Cache-Control: no-cache');
echo "data: Order has been placed.";
flush();
}
I am completely stumped. What am I missing? How do I fix this error?
Try sending a json encoded data part/element back to the browser.
Lets see if I can remember correctly... It's out of my mind, so it isn't tested!
PHP Side:
// Set SSE headers which are send along first output.
header('Content-Type: text/event-stream; charset=UTF-8');
header('Cache-Control: no-cache');
$sseId = 0; //Increment this value on each next message
$event = 'eventName'; //Name of the event which triggers same named event-handler in js.
//$data needs to be json encoded. It can be a value of any type.
$data = array ('status' => 'Progress', 'message' => 'My message or html');
//Send SSE Message to browser
echo 'id: ' . $sseId++ . PHP_EOL; //Id of message
if (!is_null($event)) {
echo 'event: ' . $event . PHP_EOL; //Event Name to trigger eventhandler
}
retry: 10000 . PHP_EOL; //Define custom reconnection time. (Default to 3s when not specified)
echo 'data: ' . json_encode($data) . PHP_EOL; //Data to send to eventhandler
//Note: When sending html, you might need to encode with flags: JSON_HEX_QUOT | JSON_HEX_TAG
echo PHP_EOL;
ob_flush();
flush();
JavaScript side:
var es = new EventSource('sse.php');
es.addEventListener('eventName', function(event) {
var returnData = JSON.parse(event.data);
var myElement = document.getElementById("processing");
switch (returnData.status) {
case 'Error':
es.close();
myElement.innerHTML = 'An error occured!';
break;
case 'Done':
es.close();
myElement.innerHTML = 'Finished!';
break;
case 'Progress':
myElement.innerHTML += returnData.message + "<br>";
break;
}
});
es.onerror = function() {
console.log("EventSource failed.");
};
In this case I make use of 1 (named) event only where the data element contains a status on which js reacts.
You can also define multiple event-names to be used as a status and define corresponding event-handlers or define a handler which is triggered on each message received.
es.onmessage = function(event) {
//This handler fires on each message received.
console.log("Incoming message.");
};
If you have server-side script running longer then 3s, specify a longer retry time in the message sent (See code at PHP side). Otherwise the browser will reconnect after 3 seconds which might run your script from the beginning and abandon the previous run.

can't insert values in sql via ajax php

i'm trying to understand this error but i dont how to solve it
this is my JS code:
$.ajax({
url: 'C:\\inetpub\\wwwroot\\VisionwareHelp\\Php/CriaUserEempresa.php',
type: "POST",
data: ({Pname: Pname, Uname: Uname, email: email, Ename: Ename, Sigla: Sigla}),
complete:function(data)
{
resposta = data;
console.log(resposta);
}
});
this is my php code:
$serverName = $server;
$uid = $uid;
$pwd = $pass;
$connectionInfo = array( "UID" => $uid, "PWD" => $pwd,"Database"=>"Portal");
//$connectionInfo = array("Database"=>"Portal");
$conn = sqlsrv_connect( $serverName, $connectionInfo);
$Pname = $_POST["Pname"];
$Uname = $_POST["Uname"];
$email = $_POST["email"];
$Ename = $_POST["Ename"];
$Sigla = $_POST["Sigla"];
if( $conn )
{
$sqlCliente = "INSERT INTO Portal.dbo.Empresa VALUES ($Ename, $Sigla)";
if (mysqli_query($conn, $sqlCliente)) {
echo "New record created successfully";
} else {
echo "Error: " . $sqlCliente . "<br>" . mysqli_error($conn);
}
}
when i do console.log of data, give me this:
what is wrong with my code?
Try to change into $sqlCliente = "INSERT INTO Portal.dbo.Empresa VALUES ('$Ename', '$Sigla')";
I think problem with your sql insert statement.
$sqlCliente = "INSERT INTO Portal.dbo.Empresa(Ename,Sigla) VALUES ($Ename, $Sigla)";
OR
$sqlCliente = "INSERT INTO Portal.dbo.Empresa VALUES ($Ename, $Sigla,..... Add all parameters to here)";
url: 'C:\\inetpub\\wwwroot\\VisionwareHelp\\Php/CriaUserEempresa.php',
This line is incorrect.
PHP files need a server to get executed. By the looks of your path, it looks like you have hosted in IIS. IIS doesn't suppport PHP by default - so assuming you have installed PHP and configured properly, read below.
You don't post to the physical path of the PHP file, rather URL of the PHP.
Therefore, it should look something similar to:
url: localhost/VisionwareHelp/php/CriaUserEempresa.php',
Until you fix the URL to the correct one, none of your posting code will work correctly.
If you have not installed PHP on IIS, then you need to download and install a PHP-enabled server like 'WampServer' and host your PHP in that.
Also, read below articles - it will help to improve your knowledge.
Installing and Testing Wampserver
Beginner’s Guide to Ajax Development with PHP
In Ajax, The correct syntax of sending multiple data is: http://api.jquery.com/jQuery.ajax/
So, Replace
data: ({Pname: Pname, Uname: Uname, email: email, Ename: Ename, Sigla: Sigla}),
with
data: {'Pname':Pname, 'Uname':'Uname', 'email':email, 'Ename':Ename, 'Sigla':Sigla },
And this line url: 'C:\\inetpub\\wwwroot\\VisionwareHelp\\Php/CriaUserEempresa.php', is also incorrect.
replace
url: 'C:\\inetpub\\wwwroot\\VisionwareHelp\\Php/CriaUserEempresa.php',
with
url: 'Php/CriaUserEempresa.php',
Hope it will help you..

Display data from php script in a table using ajax

I am looking to display data by running some query via php script and then using ajax to show it on an html page.
I have a php script that echos the data from a sql query in json format. The output looks like this:
{"Username":"Server","RICS":12739,"Exclusive_RICS":0}{"Username":"eikon1","RICS":4,"Exclusive_RICS":0}{"Username":"balbla","RICS":552,"Exclusive_RICS":0}{"Username":"somename","RICS":221,"Exclusive_RICS":201}
I would like to display this data using an $.ajax call.
I did some research and came up with this:
$(document).ready(function(){
$.ajax({
url : 'query.php',
type : 'POST',
data : {
//'numberOfWords' : 10
},
dataType:'json',
success : function(data) {
window.alert(data.Username)
},
error : function(request,error)
{
alert("Request: "+JSON.stringify(request));
}
});
});
However, it's not working properly. I just get this alert:
I am new to js/ajax/php so excuse me if I missed something simple.
Any help is appreciated. Thanks!
EDIT:
php code:
$sql = 'select * from table';
$retval = sqlsrv_query($conn,$sql);
if(! $retval )
{
die('Could not get data: ' . mysql_error());
}
while( $row = sqlsrv_fetch_array( $retval, SQLSRV_FETCH_ASSOC) ) {
echo json_encode($row);
}
sqlsrv_free_stmt($retval);
//echo "Fetched data successfully\n";
sqlsrv_close($conn);
EDIT 2:
Managed to get the output of php in correct JSON format through this. Now just need to display this using ajax.
while( $row = sqlsrv_fetch_array( $retval, SQLSRV_FETCH_ASSOC) ) {
$rows[] = $row;
}
echo json_encode($rows);
Looping through your SQL result set and echoing each row separately produces an invalid JSON format. Instead you should json_decode the entire SQL result set.
Here's how you can update your PHP code so that it outputs the correct format:
php code:
$sql = 'select * from table';
$retval = sqlsrv_query($conn,$sql);
if(! $retval ) {
die('Could not get data: ' . mysql_error());
}
echo json_encode( sqlsrv_fetch_array( $retval, SQLSRV_FETCH_ASSOC) );
sqlsrv_free_stmt($retval);
//echo "Fetched data successfully\n";
sqlsrv_close($conn);
There may be one more step necessary in your AJAX success callback. You'll have to JSON.stringify the result because PHP will send it back as plain text:
success : function(data) {
var result = JSON.stringify( data );
window.alert(result.Username)
},
Thanks for the help everyone. I was eventually able to figure out the issue.
First of all, as pointed by users, my json format was not right. I
fixed that with the solution in my edit.
Secondly, I had to reference my php directly with the exact address. I think it has to do with running queries from same server. Not sure perfectly.
Thirdly, i tried a plain ajax call and even that was failing. It turns out my browser (chrome) needed a clean up. I cleared my cookies and it started to work fine. Why it was acting weird? I have no idea!
Finally, now I needed to display the data in a table format and
update the table frequently. I am still working with that but this is
what I got going for me right now:
$(document).ready(function() {
(function poll() {
$.ajax({
url : 'http://localhost/scripts/query.php',
type : 'POST',
data : {},
dataType:'json',
success : function(response) {
var json_obj = $.parseJSON(JSON.stringify(response));
var output="";
for (var i in json_obj)
{
output+="<tr>";
output+="<td>" + json_obj[i].time.date + "</td>" + "<td>" + json_obj[i].username + "</td>" + "<td>" + json_obj[i].rics + "</td>" + "<td>" + json_obj[i].exclusive_rics +"</td>";
output+="</tr>";
}
$('#table_content').html(output);
},
error : function(request,error)
{
alert("Request: "+JSON.stringify(request));
} ,
dataType: "json",
complete: setTimeout(function() {poll()}, 5000),
timeout: 2000
})
})();
});
I really don't understand why this was so hard to do. I am sure this is a common scenario and I really wish there was a more straightforward way of doing this. Hopefully, my final code will avoid others from wasting so much of their time. Good luck!

JavaScript $.get("curl.php",{ Array Value

I'm trying to send two values from a javascript to a PHP file that in it's turn sends them to my arduino webserver. The values are strings and my wish is to send the pin value from my "pinArray" like this:
$.get("curl.php",{pin: pinArray[i],state:"1"});
But it is not working.
$.get("curl.php",{pin: "23",state:"1"});
is working fine though.
My "pinArray" looks like this:
var pinArray = ["23","25","27","29"];
The PHP file "curl.php" looks like this:
<?php
$pin = $_GET['pin'];
$state = $_GET['state'];
// Create cURL call
$service_url = 'http://arduino.local/digital/' . $pin . '/' . $state;
echo 'alert("Pin: "+ $pin +" state: "+ $state )';
$curl = curl_init($service_url);
....
Can anyone please help me with this?
It looks like either
pinArray is not defined at the time of the request
i is not defined at the time of the request
pinArray[i] is not defined at the time of the request
try to debug the values with chrome devotees or firebug:
console.log('pinArray', pinArray);
console.log('i', i);
console.log('pinArray[i]', pinArray[i]);
var data = {pin: pinArray[i],state:"1"};
console.log('data ', data);
$.get("curl.php", data);

Dynamically delete MySQL data using AJAX and JS

Hi I'm trying dynamically remove database entries using JS and AJAX without refreshing whole page.
Here is my data.php file:
<?php
require_once('db.php');
if (isset($_GET['list'])) {
$query = "SELECT * FROM message";
mysql_query("SET NAMES 'UTF8'");
$qq=mysql_query($query);
$i = 1;
echo '<div id="rezult">';
while($ff = mysql_fetch_array($qq)){
echo '<div id="id'.$ff['id'].'">'.$i++.'. Name: '.$ff['name'].' Message:'.$ff['message'].'</div>';
}
echo '</div>';
}
?>
With this code I'm retrieving a data from mysql table:
index.php
function list() {
$.get('data.php?list=1', function(o) {
$('#list').html(o);
});
}
How to dynamically delete desired entry without refreshing a page?
tried to add this code below as a link to the entry, but it getting cut javascript:$.post( like that.
<a href="javascript:$.post('delete_post.php', { id: '$ff[id]' } );" class='delete_post' title='delete post'>delete post</a>
Thanks for advices
Be careful!, if someone make a call to your php file data.php?list=ANYNUMBER he will be able to delete any row, be sure you are using security tools to avoid that.If you use ajax and Jquery I think it will be easier.
try something like this:
$.ajax({
type: "POST",
url: "list.php",
data: { list: "1", session_id: session_ID }
}).done(function( msg ) {
alert( "Data deleted: " + msg );
});
where session_id is the value of the field (in your example is 1), session_id is when someone go to your page you assign him a SESSION_ID, after he click the delete button, you compare if the session ID that you assign is equal to the the session_id from the server (you avoid people from another site to call your list.php). If session_id from the ajax is equal to session session_id on the server allow to delete take a look to this: PHP Session Security

Categories

Resources