I'm new to JS and am trying to execute a function on a site to pull all the data in table in JSON format.
I am using Parse Cloud Code to send my http requests, and the requests themselves are working, but I can't seem to get just the data itself.
It seems I am only able to get it in HTML and even then the objects do not display the same way that they do in the webpage's elements.
Any help/advice would be greatly appreciated!
Thank you in advance.
This is the link:
http://www.capetown.gov.za/Media-and-news#k=thinkwater
Here is the code:
Parse.Cloud.define('hello', function(req, res) {
res.success('Hi');
});
Parse.Cloud.define('htmlTest', function(req, res) {
Parse.Cloud.httpRequest({
method: 'POST',
url: 'http://www.capetown.gov.za/Media-and-news#k=thinkwater',
params: {
action: '/Media-and-news',
id: 'aspnetForm',
onsubmit: 'javascript:return WebForm_OnSubmit();'
},
headers: {
'Content-Type': 'application/json;charset=utf-8'
}
}).then(function(httpResponse) {
// success
res.success(httpResponse.text);
}, function(httpResponse) {
// error
res.error('Request failed with response code ' + httpResponse.status);
});
});
You can't execute client-side JavaScript function with an HTTP request.
Here's what happens when you load that page:
Server (the site you're trying to fetch) receives an HTTP request (from you)
Server generates initial HTML and responds to whoever made the above request, be it a browser, or your NodeJS Code. This "initial" HTML is what you get with a simple HTTP request. (which in your case doesn't contain the results you need)
If the HTML was served inside a browser, additional client-side JavaScript code is executed (i.e. the "javascript function" which you're trying to want to execute). This can only happen in a browser (or browser-like) environment. This JavaScript code (or function) modifies HTML (when loaded in a browser-environment, using DOM) and thus the final HTML is rendered. You can't get to these results with a simple HTTP request*, as that will only get you till #2.
*You can find out which URL the client javascript uses to fetch those results itself. Network tab in console tools might help with this. When you click on the button that triggers it to fetch the results keep an eye on which requests are made.
.
In your case it seems to be fetching JSON with a POST request from http://www.capetown.gov.za/_vti_bin/client.svc/ProcessQuery although it doesn't seem straightforward, it makes a series of requests each depending on the previous one, at least that's what it seems on the first glance. Feel free to explore this route yourself.
So in order to get the final HTML you will either
Need the direct URL that serves those results. This is usually the quickest but requires understanding the site's API and how it fetches results if it does so via AJAX (fetching via client-side JavaScript)
Use a fetcher with a browser or browser-like environment. E.g. PhantomJS (deprecated), Puppeteer, selenium, zombie
Related
I have a client-side web-app which is all static html and javascript, and I use JQuery. I am more of a back-end/middle-ware guy with Spring, and so I am not really good withthe front-end stuff.
I have a RESTful API from my client UI to my middle-ware web app written in Spring as a Controller. I can call this successfully, and there is middle-ware business logic where I can a third-party API and it returns HTML. I return this HTML to my client and I was trying to load it in an iframe or within a div tag. Of course, I got all kinds of javascript errors and a lot of CORS issues. So, if this isn't a feasible option to get HTML from another web-site and deploy it on my page, then I am willing to let that go.
I created another RESTful API on my back-end that now just returns a URL this third-party RESTful API. Specifically, I am calling the authorization page for Coinbase, but it could easily be any other third-party authorization page. I have the full URL that they recommend, and I am ready to do something with it.
So, I get a URL (dataType is 'text') like this, and I can see it with an alert:
https://www.coinbase.com/oauth/authorize?
response_type=code&client_id=MY_CLIENT_ID
&redirect_uri=MY_REDIRECT_URL
&state=SECURE_RANDOM
&scope=wallet:accounts:read
I figure I would do a redirect as follows:
window.location.replace(response);
but, what happens is I get an HTTP 404 because my system is pre-pending my UI locahost to that url. So, it looks like:
http://localhost:8080/my_web_ui_app/https://www.coinbase.com/oauth/authorize?
response_type=code&client_id=MY_CLIENT_ID
&redirect_uri=MY_REDIRECT_URL
&state=SECURE_RANDOM
&scope=wallet:accounts:read
And obviously, that is not going to work as I get the HTTP 404 error.
Anything else I should try? Like I said, I am not a good UI guy, but I just need something that works ok.
Thanks!
It was the Ajax call itself.
$.ajax({
url : 'http://localhost:8080/myapp-be/api/myrestapi/authorize',
headers : {'token' : token},
dataType: 'text',
cache: false,
success: function(response) {
console.log(response)
$(location).attr('href',response);
}
});
The dataType: 'text' was the culprit.
The URL was the response, and as 'text', it added another layer of quotes around the response.
Once I removed the dataType from the Ajax call, then the default dataType worked and I no longer got a URL with extra quotes. On success, I was able to do a simple redirect and that worked. Was a simple fix.
Thank goodness for Chrome and the debugging tools so I could see that the response coming back had that extra quotes around it.
I got a URL from a ecommerce website and when i access it i get all the last 5 products that i've visited in their site. I don't know how it works, i guess it's because of the cookie that this ecommerce website have left in my browser.
I would like to use this URL to show in my website something like this: "The Last 5 Products You Have Seen at X Ecommerce Website".
But to do that this URL must be executed in somehow in the client side and i will still need to get the JSON content returned by this URL.
Is there exist anyway to do that by using PHP or any other web technology?
Thank you!
It might be cookies, localStorage (there are other APIs to save data on local computer, imo they are unused or deprecated e.g. openDatabase) or last views could be connected with account and saved on internal database.
You should use AJAX, but by default in browser mechanism called CORS blocks all requests coming from other domain than resource.
In PHP you can download external page using file_get_contents function or cURL library, but without localStorage/cookies (which can be accessed from JS executed on domain, where that cookies are saved).
AJAX is your option for client side requests. Here's the jQuery guide for it.
https://api.jquery.com/jquery.ajax/
Here's a quick example:
$.ajax({
url: "http://ecommerce.com/your/url/here",
method: 'get',
dataType: 'json', //if you're sure its returning json you can set this
success: function(data) {
//handle success json here
//be sure that you're going to receive json though, possibly could receive some other data type and you should handle appropriately
},
error: function(error) {
//handle error json here
}
});
I am trying to record constantly updating data on a webpage. In the Google Chrome developer tools, I can see that my incoming data is obtained by an AJAX request.
When I click on the 'got' text file, I can see the data that I want in Google Chrome. I would like to use PhantomJS to receive the AJAX responses and then save these responses to files.
So far I have a program that opens the URL of the webpage I'm interested in and can print out an overview of the network traffic that is being received, but I do not know how I can save the actual files as they come in. How would I do this?
Code so far:
var page = require('webpage').create();
var url = "www.site_of_interest.com";
page.onResourceRequested = function(request) {
console.log('Request ' + JSON.stringify(request, undefined, 4));
};
page.onResourceReceived = function(response) {
console.log('Receive ' + JSON.stringify(response, undefined, 4));
};
page.open(url);
Currently, this is not possible with PhantomJS. It does not expose the request/response content in those callbacks. Possible workarounds would be:
If the AJAX requests can be replayed (multiple requests to the same URL yield the same response every time), then you can make your own AJAX request in the onResourceReceived handler and save the response into a file using the fs module.
AJAX responses for the same URL would mean that some content changes in the page. You could write custom code to check the DOM for those changes and infer what the AJAX request might have been. It doesn't necessarily have to be DOM. Maybe the data is accessible in some JavaScript variable from the page context or it is saved in localStorage.
It is also possible to write a custom XMLHttpRequest implementation as a proxy which saves the responses so that they can be grabbed. It must be injected before any page JavaScript runs. So the page.onInitialized handler works best.
I have written a post about those workarounds for CasperJS, but they can be easily converted to be used with plain PhantomJS: A: How can I catch and process the data from the XHR responses using casperjs?.
Its known that interactions between Javascript and SQL-Databases are not very secure. But most Websites use it cause the Webside doesent reload to show matches in a search.
With PHP it isn't possible to change Page-Contents without a completely Page-Refreshing.
Witch is the right way to get Data from SQL with Javascript without security-neglects.
Aspeccialy for a Searching function with directly matches in a list.
You can use 2 way to get data from db by using js;
1. Ajax:
function refresh() {
$.ajax({
url:"your url",
method: "GET",
data: your_params,
success: function(response) {
$("#specific_div_id").html(response);
}
});
}
You can do this within an interval like;
setInterval(refresh, 5000);
in order to get content in every 5 sec.
2. Websockets
In AJAX, you are requesting in every 5 secs to get updated content from server. Think that, you are not getting content server pushes updated content to you. In other words, server notifies you on any updated data. You can have a look at Socket.io for an example implementation of websockets. When server notifies you, you can take data and put it related html area
As mention in the commentaries, the best way is to use AJAX, which is an acronym that stands for Asynchronous Javascript and XML.
The last part, XML, is a bit misleading. It kept that name because that's what is was first use for. But AJAX can now be use to make HTTP request and interface with any language, including PHP.
Depending on the technology you are built on, there are several implementation available. Chances are you have jQuery installed already. In that case, jQuery Ajax, and particularly jQuery.get() would address your concerns.
If you are using a router on the backend, you can simply call a route, specifying it as the url, first argument of the function. Otherwise, you can directly call a file by using the relative path from the html page the javascript is embedded in.
jQuery.get will return anything you echo within you server script. In other words, anything that is directly rendered on the page. You can use a callback catch the data returned and process it.
Example :
$.get('/path/to/file.php', function (data) {
console.log('Here is the data received from the server!', data)
// Process data here
});
I'm trying my hand at unobtrusive JS, using JQuery in my Ruby On Rails app.
After the user fills out a form, the client-side JQuery code calls:
$.post("/premises", ui.form)
I can see the POST hit the server, and I can see the server emit a redirect notice to http://localhost:3000/users/42 complete with the data to be displayed.
But the browser page doesn't change. This doesn't really surprise me -- the whole point of client-side javascript is to control what gets updated -- I get that. But in this case, I'd like to honor whatever the server replies with.
I tried extending the call to post() based on How to manage a redirect request after a jQuery Ajax call:
$.post("/premises",
ui.item,
function(data, textStatus) {
if (data.redirect) {
// data.redirect contains the string URL to redirect to
window.location.href = data.redirect;
} else {
// data.form contains the HTML for the replacement form
$("#myform").replaceWith(data.form);
}
});
... but (among other problems) data.redirect is undefined. I suspect the real answer is simple, right? Looking forward to it!
The post you refer to uses JSON as return value and it is constructing that json on server side. it means if there is redirect your data object would look like
{redirect:'redirecturl.html'}
and if it is not redirect then data object would be like
{form:html-string-for-form}
now job is to construct json object accordingly on server side
The server is saying that the data you want to process with JavaScript is available at a different URL, not that the browser should load a new document into the top level frame. Sending the browser to the URL where it was told the data it was requesting with JS is wouldn't be honouring the redirect.
If you want to do that, then the server should respond with data (in the body of the response) that the JavaScript interprets as a reason to assign a new value to location.
data.redirect is probably undefined because you're not specifying it on the server side. In the answer you linked to the point was to have the server always respond with 200 regardless of the outcome, and then the JSON body it sends back determines how the client reacts. So, on the server side you'd want to respond with {"redirect" : "/where/to/go"}