I've got an app based on phantomjs that works so:
1. I'm running a php script that gets data from my databse (postgres) as an array,
2. Then via shell_exec I'm running phantomjs script and as argument I'm passing array with data (1),
3. In phantom I'm processing the data - checking domains WHOIS - and collecting for each domain expiration date. As result I'm getting an array that I'm storing in a file,
4. In the end phantom runs php script that gets the data from stored file and saves it in my database.
I'm wondering if there is a better option? Maybe doing everything in the phantomjs script? Maybe there is a js client for postgres?
I'd change workflow starting from step 3 and start saving data right away (PhantomJS is no stranger to crashing so it may not always get to step 4).
You could send data via an AJAX or POST request to an endpoint of your own. It could be another PHP script available via HTTP, even if on localhost. So you'd do another page.open to there and send data.
An even more reliable approach: after processing data execute a local PHP script feeding it data via CLI (or save data to a file like before and feed the script path to it).
Related
I need to monitor xhr requests on a website (which I don't control) and get realtime data. For example: tickdata.
My plan: log in with selenium and python after build websocket server. finally, acsess that.
Problem: target xhr data is like crash file. So, after debugging some js files I found out xhr data is crashed for not getting. But, the data I want exists as an array variable in a 3rd js. adding not global object.
Is it possible to get the variable in a running js script?
just like this:
Capture AJAX response with selenium and python
I'm getting data from a form filled in by users. I'm using Javascript and XMLHttpRequest to fill in the data on another form. My issue is that when the user enters their data, the JSON file on my server updates accordingly but it takes time to show on the main page, how do I force javascript to always get the most updated version of my file quickly ?
Switch your application to WebSocket if it's possiable - https://github.com/websockets/ws
I am writing my first web application with Javascript and WebGL. For now I am running the app on localhost from Apache. The app needs to work with data that is provided instantly. Until now I worked with AJAX calls that happen during runtime which doesn't work out for my purposes anymore. So instead of serving individual files from Server to Client when asked, I want the application to load all files from the Server to Client side at initialization time (I want this to happen automatically at the start so I don't have to add every new file as a url in the html index). I understand I should do this with Server Side scripting; probably with PHP since I have a Apache localhost? I have different folders which hold my necessary resources in a uniform dataformat (.txt, .png and .json). So what I want to do is, before the Javascript app starts, look through the folder and send one object per folder that holds filenames as keys bound to filedata. Is my intuition right that I need to do that with PHP? If yes, where do I start to tell my application what to do when (first start serving files with php, then start the javascript app)? How do I do this on localhost? Should I already think about extending my toolset (e.g. using nodeJS on ServerSide(locally for now))? If so what lightweight tools do you propose for this kind of work? I feel I am missing some design principles here.
EDIT:
Keep in mind that I don't want to specifically call a single file... I am already doing that. What I need is a script that automatically serves all the files of a certain folder on the server to the client side at init time of the app before the program logic of the actual application starts.
Your question is kind of broad so I'll try my best. Why does AJAX not work for real-time data but loading all the files once does? If you're working with real time data, why not look into a websocket or at the bare minimum, AJAX queries?
If you want to pass data from the server to the client, you will need to use a HTTP request no matter what. A GET request or POST request is necessary for the client to request data from the server and receive it as a response.
You could theoretically just pass the data from PHP straight to the view of the application (which is technically done through a GET request whenever a user requests data such as .php files from the server) but this isn't as flexible as if Javascript had access to the data. You can do some hacks and 'transfer' the data from the view to Javascript with some .value methods, but this isn't ideal and can be prone to some security holes. This also means data is only being passed once.
So what would need to happen is that the data would need to be processed upon initialization and then immediately transferred to the client by use of Javascript and HTTP requests.
So if you want Javascript to have access to the data and use it in variables or manipulate it further, then you'd need to use an HTTP request such as GET or POST which is called by Javascript. Otherwise, you need to immediately pass the data to the view upon initialization (through PHP), but this means you can't work with real-time data because the data is only being passed once when there is a page refresh.
Example of scandir():
<?php
//scandir() returns filenames, not data from files
$fileArray = scandir('datafolder/') //this is a relative path reference to the folder 'datafolder'
$finalArray = [];
foreach($fileArray as $filename){
tempArray = [];
$file = fopen('datafolder/' . $filename, 'r'); //im pretty sure scandir only retrieves the filenames and not the path, so you might need to append the filepath so your script knows where to look
$tempArray = fgetcsv($file, 1024); //temp array to hold contents of each iteration of foreach loop
array_push($finalArray, $tempArray); //this will store the data for later use
}
Or the data can be used however, depending on what it is. Say, if you need to combine the data from multiple .csv files, you can read each file and append it to a single array. If you want to read multiple distinct files and preserve the independence of each file, you can create multiple arrays and then pass back a single JSON encoded object that contains each file's data as a separate attribute of the object such as:
{
'dataOne': [0,1,2,3,4,5,6...],
'dataTwo': ['new', 'burger', 'milkshake'],
'dataThree': ['Mary', 'Joe', 'Pam', 'Eric']
}
Which can be created with a PHP associative array using one of the following methods:
//assuming $arrayOne is already assigned from reading a file and storing its contents within $arrayOne
$data['dataOne'] = $arrayOne;
// or
array_push($data['dataTwo'], $arrayTwo);
// or
array_push($data, [
'dataThree' => ['Mary', 'Joe', 'Pam', 'Eric']
]);
Then $data can simply be passed back which is a single array containing all the different sets of data, if each set needs to be distinct.
I am trying to write my first chrome extension. The workflow goes something like this -When the extension is installed and active if a user hovers over a specific product/ID displayed on the page, the extension retrieves related vendor data about the product with the ID.
This is how I thought about this:
Use jQuery attr to access the ID on mouse over.
Post this ID to a retrieve.php file with .post() method
The retrieve.php file retrieves the data from database
Display the data in a tool tip on the web page.
I have some queries for the above process:
I am able to get this working on a local XAMPP server but how will it work online as the chrome extension will not have access to server. What is the way around to retrieve data without using PHP?
I am able to get the logic working but am unable to place these in respective files - Will all my logic reside in background.js ?
Any suggestions on getting this started will be much appreciated.
You could build a very simple API on your server that responds with JSON to any request it receives after processing it. Like this:
{"firstVar":"foo","secondVar":"bar" }
Your chrome extension can then make an xmlhttp request to this server and and process the returned data.(You could also use JSONP and wrap the response in a callback function which will execute as soon as you have the reponse)
The JS extension will be able to deal with the JSON nicely as it can understand that format so you can then choose to display the data in whatever way you want.
Essentially, what you want is a server that can take an ID posted to it and return the corresponding date in a nice and readable format. And a chrome extension that can make an request to a server and then process the response. Build and test them separately (keep positing an ID to the server and see the response and for your JS side at first instead of making requests to your unfinished API just set a static response to begin with which will be the same as an expected response.
I'm a consultant working on a web app that's basically a single page app. All it does is constantly retrieve new json data behind the scenes (like once a minute), and then display it on screen.
Our clients load this app, and leave it running 24/7, for weeks on end. If errors happen when retrieving new json data, the app ignores it and keeps running.
We're rolling out an update, and want the existing clients to either become invalidated, or reload themselves without any user interaction. This feature wasn't "built in" by anyone, and we're trying to do this after the fact.
Is there some way to make the existing clients reload without telling our end users to just reload the page?
The following conditions define the app a bit more:
The app uses jQuery 1.9.0
Runs exclusively in Chrome
Retrieves new json data frequently using jquery
Throws away any errors it finds in json responses and uses old data.
EDIT:
I've had it suggested that we could try the following:
send invalid data through the JSON responses to crash chrome (like 500 megs of data, for example)
send window.location.reload through the JSON response (which supposedly won't work due jquery protecting against this type of thing)
send "script" data in the JSON response and if it gets $.html(....) at some point, then it may run the script as well.
and am open to any suggestions on getting this to reload or kill chrome, so the client is forced to reload the page.
If you're using $.ajax to request your data, and not specifically setting your content type, then you may be able to do the following on the server:
set the content type header to "text/javascript"
respond with javascript, e.g. window.location = "http://www.yoursite.com"
jQuery may eval that, and simply run your javascript.
No it is not possible. As far as I can tell you do not execute code from the JSON response (which is a very good thing). Thus you have no way of altering your current client's behaviour. According to your own statement:
"Throws away any errors it finds in JSON responses and uses old data"
You will not be able to crash the user's browser by sending invalid JSON data as the errors will be suppressed.
You can build in automatic deployment in to future versions by sending an application version number and testing for changes or by using WebSockets (which the application seems better suited to anyway as you can ensure your clients only poll the server when the JSON has actually changed).
If I get it correctly, create a version referance page, and make the client check this page very couple seconds, when you update the file, client will reload itself with this script.
var buildNo = "1.2.0.1";//
var cV = setInterval(checkVersion,(5*1000))//Every 5 sec.
function checkVersion(){
$.ajax({
url:"checkVersion.php?v="+buildNo,
dataType:"JSON",
success:function(d){
if(d.version != buildNo){//if version is different
window.location.reload();
//chrome.runtime.reload(); //for chrome extensions
}
}
})
}
if you cant add extra page, you may just add extra variable to end of your JSON data.