Strictly using Javascript, is it possible to call a remote URL and store it in a variable? To specify, this would not be on the same domain. Basically, what i am trying to figure out the best solution for scraping/parsing data in a chrome extension.
If you're trying to load the page from an external domain, you'll run into the same-origin policy. You can get around this limitation by setting up a simple proxy on your own site. This would be a bit of server-side code whose function is to make the http request to the external page on your behalf, retrieve the response, then serve that content back to the on-page asynchronous JavaScript code making the original request.
So if you set up your proxy at /your_internal_proxy.foo, and you're using jQuery, you could build a JS function something like this:
var foreignContent = {};
var loadPage = function(pageURL,varName) {
$.ajax({
type: 'get',
url: '/your_internal_proxy.foo',
data: {
url_to_get: pageURL
},
success: function(result) {
foreignContent[varName] = do_something_to( result );
}
});
};
Then, to load content into JS variables:
loadPage('http://www.google.com/','goog');
loadPage('http://stackoverflow.com','stack');
While Ajax may be the way to go it depends on what you mean by "remote URL". Ajax will only work for url's with the same domain as the domain where the javascript originated from, due to Javascript cross-domain restrictions. There are various work arounds but most require the co-operation of the remote domain. I've have heard of using YQL as an intermediary but have never tried it. Easiest is to host a proxy on your own server.
If you are allowed to use jQuery you can have something like:
$.ajax({
url: url,
data: data,
success: success,
dataType: dataType
});
Also the $.get() can be used to achieve this
http://api.jquery.com/jQuery.get/
$.get("test.html",
function(data){
// do something with response data
});
Related
I had a problem about callback function which was located in $(document).ready . Callback function didn't used to work. When I put it outside of $(document).ready the code has started to work perfectly. I don't understand why. Is location is important?
This works:
$(document).ready(function() {
$("#button1").click(function() {
$.ajax({
url: "http://www.example.com/data.php",
type: "get",
dataType: "jsonp",
jsonpCallback: "read",
});
});
});
var read = function(data) {
console.log(data);
}
This does not work.
$(document).ready(function() {
$("#button1").click(function() {
$.ajax({
url: "http://www.example.com/data.php",
type: "get",
dataType: "jsonp",
jsonpCallback: "read",
});
});
var read = function(data) {
console.log(data);
}
});
UPDATE1: Sorry guys, links werent't different. I forgot to change 2nd one. There is just one difference that location of read function.
The reason you pass in the JsonP callback name as a string like that is because JQuery needs to add it to your URL like this ?callback=read. A JsonP request is just a <script> tag created by JQuery in the background and added to the page <script src="http://www.csstr.com/data.json?callback=read"></script>. Once JQuery adds that script tag to your page the browser will treat it like it's loading a normal JavaScript document to be exectued. Because of the ?callback=read part of the request, the remote server knows to respond with executable JavaScript and not just the raw data. That executable JavaScript is just a function call to a function with the name you provided, in this case the read function. Because the returned script is being executed in the global scope, the read function also needs to exist in the global scope. The global scope in the browser is the window object, so basically the read function needs to be present on the window object in order for the executed script to find the function.
$(document).ready(function() {
$("#ara-button").click(function() {
$.ajax({
url: "http://www.csstr.com/data.json",
type: "get",
dataType: "jsonp",
jsonpCallback: "read",
});
});
window.read = function(data) {
console.log(data);
}
});
It works outside of the ready function in your first example because anything defined at the root level like that is globally scoped.
Codpen Demo: http://codepen.io/anon/pen/qNbRQw
If you want to know more about how JsonP works, read on.
If you're still confused it's probably because you're not 100% familiar with how JsonP actually works. JsonP is a hack to get around the Same-origin Policy. The short version of the Same-origin Policy is that the browser won't let you read the response returned from requests to domains other than the one the request originated from unless the server on that other domain says it's okay.
The Same-origin Policy was implemented by most browsers to help protect users from malicious scripts. For example, if you authenticated with your bank website in one browser tab and then went to a nefarious website in another tab, without same-origin restrictions in the browser that nefarious site could make an ajax request to your bank website. That request would carry any cookies your browser has stored for that domain and your cookies would show you as authenticated, thus giving the attacking script access to authenticated data from your bank account. The Same-origin Policy prevents the nefarious site from being able to see the response data from that request.
In the beginning there was no official way for the client and server to opt-in to cross-domain sharing. It was just straight up blocked by the browsers at the time. To get around this JsonP was invented. The Same-origin Policy only hides the response from ajax requests, but as you may have noticed the browser is totally fine with loading scripts from other websites via <script> tags. The script tag just does a plain old GET request for a javascript document, then starts executing that script on your page. JsonP takes advantage of the fact that same-origin restrictions don't apply to <script> tags.
Notice if you go directly to http://www.csstr.com/data.json in your browser you'll see the data you're after. However, try going there with the following query string added.
http://www.csstr.com/data.json?callback=read
Notice anything different? Instead of just returning the data you want the response comes back with your data being passed into a function named read. This is how you know the server understands the JsonP hack. It knows to wrap the data you want in a function call so that JQuery can perform the JsonP hack on the client, which it does by creating a <script> tag and embedding it in your web page. That script tag points to the URL but also adds that querystring to the end of it. The result is a script tag that loads a script from that URL and executes it. That script is being loaded into your page's global scope, so when the read function is called it expects that function to also exist in the global scope.
Today, the official way around the Same-origin Policy is via the Cross-origin Resource Sharing policy (aka CORS). JsonP essentially accomplishes the same thing that a proper CORS request does. The server has to opt-in by knowing how to format the response as executable JavaScript, and the client has to know not to do a normal ajax request but instead dynamically create a script tag and embed it in the page body. However, JsonP is still a hack and as hacks do it comes with its own set of downsides. JsonP is really hard to debug because handling errors is almost impossible. There's no easy way to catch errors if the request fails because the request is being made via a <script> tag. The <script> tag also lacks control over the format of the request being made; it can only make plain GET requests. The biggest downside to JsonP is the need to create a global function to be used as a callback for the data. Nobody likes polluting the global namespace but for JsonP to work it's required.
Proper CORS requests don't require extra effort on the client's part. The browser knows how to ask the server if it is allowed to read the data. The server just has to respond with the right CORS headers saying its okay and then the browser will lift the same-origin restrictions and allow you to use ajax as normal with that domain. But sometimes the resource you're trying to hit only knows JsonP and does not return proper CORS headers so you have to fallback on it.
For more info about CORS I wrote a pretty detailed blog post a little while ago that should really help you understand it. If you control the server that is returning the data you're after then you should consider just having it return the proper CORS headers and forget about JsonP. But it's totally understandable when you don't control the server and can't configure it to return the proper headers. Sometimes JsonP is the only way to get what you need, even if it does make you throw up in your mouth a little bit as you write the code.
Hope that helps clear some things up for you :)
$(document).ready(function() { means : Do it when the page finished to load.
In the second example, you are defining the read function only after the page finished to load.
In the working example, you are defining the read function first and say, "Once the page will be loaded, do the ajax call and then, call read function"
Edit : Also, you can read #IGeoorge answer for a more detailed explaination.
Add read method before ajax
$(document).ready(function() {
var read = function(data) {
console.log(data);
}
$("#ara-button").click(function() {
$.ajax({
url: "http://www.csstr.com/data.json",
type: "get",
dataType: "jsonp",
jsonpCallback: "read",
});
});
});
Your function is defined into Jquery Scope, so at the moment the ajax is executed, cannot find definition of "read" function.
That's why the first example works.
Hope this helps.
I am running ajax without a server on my system, I have created one index.html with this.
JavaScript function:
function do_the_click(url)
{
alert('inside this method do_the_click');
$.ajax({
async: false,
url : url,
contentType: "text/html",
type : "GET",
dataType : "text/html",
success: function(data){
alert('data');
}});
}
My HTML body content is:
<a href="Java Collections.html" class="button" id="javacollections" onclick="do_the_click('this.href');"/>Java Collections</a>
I'm able to get the this message in alert window inside this method do_the_click but am not able to get the data in alert window, and so not able to get the page Java Collections.html in the index.html,
I have searched very much on Google re. how to load the data locally without using server in jquery.ajax but I didn't find any good solution, so I think that its good for others as well if it resides on Stack overflow.
You cannot do such thing. if you dont have a server you cannot send ajax it will be a cross browser issue. as the others says ajax will not work with file:// protocol you need a server to make an http:// call that ajax supports.
When you are making AJAX requests using stand alone html files, the HTTP request to the url is made by the browser. All you need to make sure is that you have included JQuery. Here is what you have to do.
Include Jquery:
<script src="js/jquery-1.10.2.min.js"></script>
The JQuery here resides in the js folder. This is included in the stand alone html file. Now to make the AJAX request use the following code.
$.ajax({
type: 'GET',
url: url,
success: function(data){
console.log(data);
},
async:false
});
AJAX creates an HTTP request that must be responded to by a server. If you don't have a server you can't use an AJAX request. You can run a server on your computer, but this is not the same as getting a file from the local filesystem.
I m able to run the ajax without webserver the problem was with my code the code written below that i had run on my filesystem u can use that as well.
function do_the_click(brl)
{
alert('inside this method do_the_click');
alert(brl);
var request = $.ajax({
async: false,
url: brl,
contentType: "text/html",
type : "GET",
dataType : "html",
});
request.done(function( msg ) {
alert(msg);
$( "#contentarea" ).load( msg, function() {
alert( "Load was performed." );
});
});
request.fail(function( jqXHR, textStatus ) {
alert( "Request failed: " + textStatus );
});
}
Java Collections
You won't be able to do this. If you don't have a server, you will violate the browser's cross-domain policy, as you won't have a domain!
AJAX won't work via the filesystem, you need a server.
This is an older thread but a browser doesn't need to run on a common OS like OSX/Windows/etc .. many people today are making their own browsers from WPE/Wayland or QT5Webkit where same origin policies don't apply.
Now in current scenario it depends upon where your server at. If the server is some place else, it will require CORS (cross origin request). So using file:// with CORS works perfectly.
I came to this question because I was searching for the reason, why I am able to make ajax request without using a web server. Got the answer Thanks!
This returns a json object
http://developer.echonest.com/api/v4/artist/search?api_key=youAPIKey&name=radiohead
i tried this way but didn't work.
function cc() {
jQuery.ajax({
url: "http://developer.echonest.com/api/v4/artist/search?api_key=APIKey&name=radiohead",
type: "POST",
dataType: "json",
async: false,
success: function (data) {
console.log(data);
}
});
}
how can i call this url and access the object values using java script or jquery.
Due to Same Origin Policy, you can't access a resource on a different domain.
The solution is to proxy the request through a server side script on the same domain as your Javascript. JSONP is an option but only if the third party supports it.
As suggested by MrCode, you will need to use JSONP to access the data as your script is running in a different domain.
Echonest supports JSONP, using the format and callback options. You need to amend your url to include the format option.
http://developer.echonest.com/api/v4/artist/search?api_key=FILDTEOIK2HBORODV&name=radiohead&format=jsonp
If you amend the dataType option in your ajax options to jsonp then jQuery will append a callback parameter, which Echonest will use when sending the data back to you. You can then access the data in your success handler as required.
Unfortunately, you have posted your API key in your question, which means that lots of Stack Overflow users may be looking at your link, and screwing your access limit. You may need to wait for this question to become unpopular before your rate limit drops off again.
I'm horrible at Javascript, so sorry in advance for what I'm going to go ahead and assume is an amazingly stupid question.
I'm simply trying to perform a GET request to GitHub's public repo API for a given user, and return the value as JSON.
Here's the function I'm trying to use:
function get_github_public_repos(username) {
var the_url = "http://github.com/api/v2/json/repos/show/" + username
$.ajax({
url: the_url,
dataType: 'json',
type: 'get',
success: function(data) {
alert('raw data: ' + data)
var json_response = $.parseJSON(data);
alert(json_response);
}
});
}
This is returning Null for data. And in the console, I see Failed to load resource: cancelled. I know the URL is correct, because if I run curl on the url, it returns the expected data.
jQuery's ajax function supports JSONP which allows cross-domain requests (which you need because you're trying to request data from github.com from another domain). Just change the dataType from 'json' to 'jsonp';
function get_github_public_repos(username) {
var the_url = "http://github.com/api/v2/json/repos/show/" + username
$.ajax({
url: the_url,
dataType: 'jsonp',
type: 'get',
success: function(data) {
var json_response = data;
alert(data);
}
});
}
UPDATE: It's import to note that the end pint (in this case github.com's API) has to support JSONP for this to work. It's not a guarnateed solution for ANY cross-domain request as pointed out in the comments.
JavaScript is subject to cross-domain restrictions when making requests on a different server.
Well, unless you run your code in the github.com domain, that won't work.
You can use simle ajax only in your domain.
One solution is to create a proxy for it. Make a page on your server that does one thing, gets your requested (out of domain) content with curl, and prints it. Then you call this proxy with ajax.
The XmlHttpRequest object (which $ajax uses) cannot download content from a different domain due to the same origin policy. You would need to use something such as JSONP to be able to do this from a browser.
As the others have said, you cannot execute ajax on a remote domain.
You will need to write a server sided script on your domain (such as php), that will do the dirty work retrieving the information needed from the github domain.
Then, use your ajax to query your server side script for the information.
I know the URL is correct, because if
I run curl on the url, it returns the
expected data.
Use that idea to get your ajax working. Create a proxy page on your site that uses curl to retrieve the data you want, then have your "the_url" variable point to that page on your site. Cross-domain restrictions prevent you from being able to use ajax in the manner you attempted.
I have a rails app and i wanna to parse the xml response from API using Javascript (ajax).
I tried:
$(document).ready(function()
{
$.ajax({
type: "GET",
url: "http://localhost:3000/users/mike.xml",
dataType: "xml",
success: parseXml
});
});
function parseXml(xml)
{
...
}
but don't work. When i am changing the 'url' with a local xml file e.x url: 'data.xml',
works fine!
How can i parse this response?
any help will be highly appreciated :-)
As Max suggested, it would be very helpful to install and use Firebug in a Firefox browser so you can watch the GET request and response. There isn't a lot to go on from your question, but it sounds like a problem on the "server" end.
When you say it doesn't work, do you mean parseXml(xml) isn't called? In your AJAX request, you define a success handler - if your GET request is failing then that handler is never called. I've found it's more useful to define the general callback ('complete' for JQuery, which it kind of looks like you're using) since it will get called no matter what the response. Then you just check to for success or failure yourself and take appropriate action.
Try removing the ".xml" from your URL. I believe AJAX calls accept xml response by default, so having xml in your request header as well as your URL might be confusing the controller. That's just a guess though.
If the URL is not on the same domain as your page with the JavaScript, your browser will prevent it from working because of the cross domain security policy. Check that you are rendering the page via a URL that looks exactly the same as the one you are requesting in your ajax call (so in the localhost example make sure you aren't running your server on a 127.0.0.1 URL, or something like that)