I have a PhantomJS script that works when I run it locally (Mac), but when I run it on my Linux server, it returns the following error:
ReferenceError: Can't find variable: $
https://fantasy.premierleague.com/a/statistics/value_form:5712 in global code
The code is:
var page = require('webpage').create();
var fs = require('fs');
var args = require('system').args;
page.settings.userAgent = 'SpecialAgent';
page.open('https://fantasy.premierleague.com/a/statistics/value_form', function (status) {
if (status !== 'success') {
console.log('Unable to access network');
} else {
var ua = page.evaluate(function () {
var result ="";
// ...
return result;
});
}
phantom.exit();
});
There may be a race condition between your code and jQuery being loaded on the page. Wrap the statements in your page.evaluate callback with a $(document).ready(function() { /* your statements here */ }); to ensure scripts on the page have loaded fully.
For anyone who is still using PhantomJS and encounters this problem, I solved it with
phantomjs --ignore-ssl-errors=yes
I don't intend for upvote.
I'm providing solution to solve some situations without simulating browser behavior with phantomjs just to retrieve data that can be handled directly by requesting to url.
You need the data from the page, so why not just do request to this url: https://fantasy.premierleague.com/drf/bootstrap-static
var request = require('request'); // install: npm i request
var fs = require('fs');
var args = require('system').args;
request.get({url: 'https://fantasy.premierleague.com/drf/bootstrap-static'}, function(err, response, body) {
console.log(body);
});
How I found this url?
Simple:
Related
I have a php file that calls phantomjs via the exec command. The phantom.js file calls a url that needs to include variables. I need to send these variables from php along with the exec command and have the variables show up in the phantom.js url.
Here is my current hardcoded php:
$response = [];
exec('/usr/bin/phantomjs phantomjstest.js', $response);
$data = $response[19];
and phantom.js
var page = require('webpage').create();
console.log('The default user agent is ' + page.settings.userAgent);
page.settings.userAgent = 'SpecialAgent111';
page.open('https://www.aa.com/travelInformation/flights/status/detail?search=AA|1698|2019,1,23&ref=search', function(status) {
if (status !== 'success') {
console.log('Unable to access network');
} else {
var ua = page.evaluate(function() {
return document.getElementById('aa-content-frame').innerHTML;
});
console.log(ua);
}
phantom.exit();
});
What I would like to do is change the php so that it passes 2 variables (from a form submit) to the javascript. Something like:
PHP:
exec('/usr/bin/phantomjs phantomjstest.js?variable1=$forminput1&variable2=$forminput2', $response);
JS
page.open('https://www.aa.com/travelInformation/flights/status/detail?search=AA|variable1|variable2&ref=search', function(status) {
Or I can construct the entire URL in the php and send it along with the exec command.
Any ideas on either method, or some other way to get from here to there, are most appreciated. Thanks.
Based on suggestions in comments, I have updated my PHP to contain:
exec("/usr/bin/phantomjs phantomjstest.js https://www.aa.com/travelInformation/flights/status/detail?search=AA|1698|2019,1,23&ref=search", $response);
and my JS to:
var page = require('webpage').create();
console.log('The default user agent is ' + page.settings.userAgent);
page.settings.userAgent = 'SpecialAgent111';
var system = require('system');
var args = require('system').args;
var address = system.args[0];
page.open(address, function (status) {
if (status !== 'success') {
console.log('Unable to access network');
} else {
var ua = page.evaluate(function() {
return document.getElementById('aa-content-frame').innerHTML;
});
console.log(ua);
}
phantom.exit();
});
But all I get back is a null response. Any idea why the address is either not getting passed with PHP exec or not getting picked up and run by JS?
Thanks.
****** SOLUTION ********
There were a couple of things that needed fixing, and I want to thank those that commented below, as well as others on SO that offered solutions to similar issues.
First, the JS did not like the & symbol in the URL. I had to use # in the passed argument instead and then replace it on the JS side.
Second, the JS did not like the pipe symbol in the URL so I had to escape them.
The finished PHP looks like:
exec("/usr/bin/phantomjs phantomjstest.js https://www.aa.com/travelInformation/flights/status/detail?search=AA\|$flight\|$date#ref=search", $response);
and the JS like:
var page = require('webpage').create();
console.log('The default user agent is ' + page.settings.userAgent);
page.settings.userAgent = 'SpecialAgent111';
var system = require('system');
var args = require('system').args;
var address = system.args[1];
var address = address.replace(/#/gi,"&");
page.open(address, function (status) {
if (status !== 'success') {
console.log('Unable to access network');
} else {
var ua = page.evaluate(function() {
return document.getElementById('aa-content-frame').innerHTML;
});
console.log(ua);
}
phantom.exit();
});
Again, my thanks to all!!!!
****** SOLUTION ********
There were a couple of things that needed fixing, and I want to thank those that commented below, as well as others on SO that offered solutions to similar issues.
First, the JS did not like the & symbol in the URL. I had to use # in the passed argument instead and then replace it on the JS side.
Second, the JS did not like the pipe symbol in the URL so I had to escape them.
The finished PHP looks like:
exec("/usr/bin/phantomjs phantomjstest.js
https://www.aa.com/travelInformation/flights/status/detail?
search=AA\|$flight\|$date#ref=search", $response);
and the JS like:
var page = require('webpage').create();
console.log('The default user agent is ' + page.settings.userAgent);
page.settings.userAgent = 'SpecialAgent111';
var system = require('system');
var args = require('system').args;
var address = system.args[1];
var address = address.replace(/#/gi,"&");
page.open(address, function (status) {
if (status !== 'success') {
console.log('Unable to access network');
} else {
var ua = page.evaluate(function() {
return document.getElementById('aa-content-frame').innerHTML;
});
console.log(ua);
}
phantom.exit();
});
I'm fetching an website's content via phantomjs by including jquery with the page. Now i have to write them to a file via program. For that i used the following code
page.onLoadFinished = (function(status) {
if (status === 'success') {
page.includeJs('http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js', function() {
page.evaluate(function() {
var mkdirp = require('mkdirp');
mkdirp(counter+'_folder', function(err) {
var html = $('pre[data-language="html"]').html();
var js = $('pre[data-language="js"]').html();
var css = $('pre[data-language="css"]').html();
var fs = require('fs');
fs.writeFile(counter+"_folder/"+"fiddle.html", html, function(err) {});
fs.writeFile(counter+"_folder/"+"fiddle.css", css, function(err) {});
fs.writeFile(counter+"_folder/"+"fiddle.js", js, function(err) {});
console.log("******* "+counter+" *************");
});
});
});
}
});
page.open(url[counter]);
Now what happening is inside evalute method when I'm using require the program is getting stopped there showing error cannot find variable require. Any idea why this is appering?
page.evaluate() is the sandboxed page context. It has no access to require, page, phantom ...
Furthermore, mkdirp is a node module which will not work with PhantomJS. If you want to use PhantomJS from node, you will have to use a bridge like phantom. See also: Use a node module from casperjs
Using that bridge, you have to pass the variables to the outside and save it from there:
page.open(url, function(){
var mkdirp = require('mkdirp');
mkdirp(counter+'_folder', function(err) {
page.evaluate(function() {
var html = $('pre[data-language="html"]').html();
var js = $('pre[data-language="js"]').html();
var css = $('pre[data-language="css"]').html();
return [html, js, css];
}, function(result){
var fs = require('fs');
fs.writeFile(counter+"_folder/"+"fiddle.html", stuff[0], function(err) {});
fs.writeFile(counter+"_folder/"+"fiddle.css", stuff[1], function(err) {});
fs.writeFile(counter+"_folder/"+"fiddle.js", stuff[2], function(err) {});
console.log("******* "+counter+" *************");
});
});
});
Note: PhantomJS' fs module doesn't have a writeFile function. Node and PhantomJS have different execution environments.
I've been trying this for a while, but I keep getting the error:
Error: Command failed: Invalid Parameter - /images
I installed ImageMagick and the gm package, so that's definitely not the problem.
gm(imageLocation)
.resize(100) // use your own width and height
.write('here.jpg', function (err) {
if (!err) console.log(' hooray! ');
else console.log(err);
});
imageLocation being ./images/3.jpg. Why does this error keep happening? I looked at the documentation
I'm on a Windows 32 bit machine. My server is supposed to get an image from a folder, resize it, and then display it. It seems like I have to write the resized photo and then display that, but the writing process always errors out and the image ends up being empty.
If there's a way to skip the writing part and just displaying the photo directly, that would be awesome too.
Thanks!
URL Query I used: http://localhost:8123/images/3.jpg
Complete code:
var querystring = require('querystring'); //used for parsing parts of urls
url = require('url');
http = require('http');
fs = require('fs');
gm = require('gm').subClass({ imageMagick: true });;
var server = http.createServer();
server.on('request', function(request, response){
var parsed_url = url.parse(request.url, true); //true gets the query as well
imageLocation = '.' + parsed_url.pathname;
gm(imageLocation)
.resize(100) // use your own width and height
.write('here.jpg', function (err) {
if (!err) console.log(' hooray! ');
else console.log(err);
});
if (getImage('here.jpg', response)){
//image is displayed
}
else{
respond404(parsed_url.pathname, response);
}
})
function respond404(path, response){
respond(404, "The requested path " + path + " was not found", response)
}
function getImage(location, response)
{
try{
var img = fs.readFileSync(location);
response.writeHead(200, {'Content-Type':'image/jpg'}); //parse this end
response.end(img, 'binary');
return true;
}catch(e){
return false;
}
}
server.listen(8123);
The answer Svbaker put can be used in Linux (maybe Mac as well?)
For Windows I got it to work by opening the command line in administrator mode and starting my server there.
I was able to get your code to work by changing how you required gm as follows:
var gm = require('gm');
I also had to remember to execute node with the correct permissions in my case:
sudo node server.js
I am new to nodejs, what I'm trying to do is to scan all the url of my site (with javascript and jquery enabled) and check that the url contains a given string.
To do this I'm using jsdom, but when I launch the script extracts only some url and then crashes giving this error:
timers.js:110
first._onTimeout();
^
TypeError: Property '_onTimeout' of object [object Object] is not a function
at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
Surely there is something wrong but I don't understand where..
This is my script:
var request = require('request');
var jsdom = require('jsdom');
request({ uri: 'http://www.example.com' }, function (error, response, html) {
if (!error && response.statusCode == 200) {
var doc = jsdom.jsdom(html, null, {
features: {
FetchExternalResources : ['script'],
ProcessExternalResources : ['script'],
MutationEvents : '2.0',
}
});
var window = doc.createWindow();
jsdom.jQueryify(window, "http://code.jquery.com/jquery-1.5.min.js", function() {
var $ = window.jQuery;
$('a').each(function(i, element){
var a = $(this).attr('href');
console.log(a);
if (a.indexOf('string') != -1) {
console.log('The winner: '+a);
//return a;
}
});
window.close();
});
}
});
This is because of somewhere in your page they are calling setTimeout/setInterval with a string that is not supported in node and it results in that error.
To find out where is it coming from, I suggest just require longjohn module(require('longjohn')) and you get long stack traces, which they will help you to find the error. For example I got something like this from doing this in the repl:
at listOnTimeout (timers.js:110:15)
---------------------------------------------
at startTimer (/home/alfred/repos/node_modules/jsdom/lib/jsdom/browser/index.js:75:15)
at DOMWindow.setTimeout (/home/alfred/repos/node_modules/jsdom/lib/jsdom/browser/index.js:124:50)
at file:///home/alfred/repos/repl:undefined:undefined<script>:1:1
at Contextify.sandbox.run (/home/alfred/repos/node_modules/jsdom/node_modules/contextify/lib/contextify.js:12:24)
at exports.javascript (/home/alfred/repos/node_modules/jsdom/lib/jsdom/level2/languages/javascript.js:5:14)
at define.proto._eval (/home/alfred/repos/node_modules/jsdom/lib/jsdom/level2/html.js:1523:47)
at /home/alfred/repos/node_modules/jsdom/lib/jsdom/level2/html.js:76:20
at item.check (/home/alfred/repos/node_modules/jsdom/lib/jsdom/level2/html.js:345:11)
If by any chance that didn't work for you or you didn't like it, then I suggest you to modify this jsdom file: node_modules/jsdom/lib/jsdom/browser/index.js, function startTimer. Throw an error there if the callback wasn't a function. This will throw whenever offending code was run.
In case if you are running code that you can't change(like from websites you don't own, which I don't suggest it because foreign javascript like that could be used to attack your app), you could override DOMWindow.setTimeout/.setInterval to support string arguments. You could also make open an issue for jsdom to have this opt-in.
Right now I'm using this script in PHP. I pass it the image and size (large/medium/small) and if it's on my server it returns the link, otherwise it copies it from a remote server then returns the local link.
function getImage ($img, $size) {
if (#filesize("./images/".$size."/".$img.".jpg")) {
return './images/'.$size.'/'.$img.'.jpg';
} else {
copy('http://www.othersite.com/images/'.$size.'/'.$img.'.jpg', './images/'.$size.'/'.$img.'.jpg');
return './images/'.$size.'/'.$img.'.jpg';
}
}
It works fine, but I'm trying to do the same thing in Node.js and I can't seem to figure it out. The filesystem seems to be unable to interact with any remote servers so I'm wondering if I'm just messing something up, or if it can't be done natively and a module will be required.
Anyone know of a way in Node.js?
You should check out http.Client and http.ClientResponse. Using those you can make a request to the remote server and write out the response to a local file using fs.WriteStream.
Something like this:
var http = require('http');
var fs = require('fs');
var google = http.createClient(80, 'www.google.com');
var request = google.request('GET', '/',
{'host': 'www.google.com'});
request.end();
out = fs.createWriteStream('out');
request.on('response', function (response) {
response.setEncoding('utf8');
response.on('data', function (chunk) {
out.write(chunk);
});
});
I haven't tested that, and I'm not sure it'll work out of the box. But I hope it'll guide you to what you need.
To give a more updated version (as the most recent answer is 4 years old, and http.createClient is now deprecated), here is a solution using the request method:
var fs = require('fs');
var request = require('request');
function getImage (img, size, filesize) {
var imgPath = size + '/' + img + '.jpg';
if (filesize) {
return './images/' + imgPath;
} else {
request('http://www.othersite.com/images/' + imgPath).pipe(fs.createWriteStream('./images/' + imgPath))
return './images/' + imgPath;
}
}
If you can't use remote user's password for some reasons and need to use the identity key (RSA) for authentication, then programmatically executing the scp with child_process is good to go
const { exec } = require('child_process');
exec(`scp -i /path/to/key username#example.com:/remote/path/to/file /local/path`,
(error, stdout, stderr) => {
if (error) {
console.log(`There was an error ${error}`);
}
console.log(`The stdout is ${stdout}`);
console.log(`The stderr is ${stderr}`);
});