Develop Reference

Python requests handle url contains callback

https://www.goodreads.com/api/index contains an example how to call json api by js
<script type="text/javascript">
function myCallback(result) {
alert('nb of reviews for book: ' + result.reviews.length);
}
var scriptTag = document.createElement('script');
scriptTag.src = "https://www.goodreads.com/book/isbn/0441172717?callback=myCallback&format=json&user_id=123456789";
document.getElementsByTagName('head')[0].appendChild(scriptTag);
</script>
To be frank, it is so strange to me that url could contains a callback function name. What's the secret here? Any relevant js document?
Anyway, it is a javascript example. If I want to use python requests to do the same job. How and what should I do?
I am completely stuck here.
Thanks for your advice.

This type of call is really only used from the browser, so it wouldn't apply to a Python request.
This is an example of a JSONP request, which is a way to make a cross-origin request, EG, your app served at foobar.com wants to make a POST request to example.com. In many (but not all) circumstances such requests will be blocked by your browser for security reasons.
In this case, the result of the request is passed to the callback, which is then executed by the browser.
Related questions have been asked before, there's a nice explanation of JSONP-- how it works, and why you would want use it here: https://stackoverflow.com/a/2067584/3084820
If you are using requests you won't have the cross-origin concern, since you'll be making that request from the server side. I am not familiar with the goodreads API but I suspect that they have a version of the endpoint that can be called from a server.
NOTE: The Goodreads API is fairly weird and not well-documented. I played around with this and was able to get it to work using requests, but EG you need to send your API key as key=... not user_id=...

Requesting remote XML data with javascript

Ok here's my problem. I'm working on this little site called 10winstreak and I'm trying to detect if a stream is live or not with javascript because our server that we run the site off of cant handle processing every single request with PHP. The basis of detecting if a stream is live or not is you go to their XML file and in one of their tags (if it's live) it will say something along the lines of true and often time the XML file on their site will be empty if a particular stream isn't live. for example if you have a twitch.tv stream for gamespot you go to http://api.justin.tv/api/stream/list.xml?channel=gamespot and if it's got stuff in it then it's live if not then it's not.
so basically my code looks like this:
function check (URL, term){
$.get(URL , function(data){
console.log(data);
//data is whatever the server returns from the request, do whatever is needed with it to show who is live.
var number = data.search(term);
if (number > -1)
{
document.write("Live");
}
else
{
document.write("Offline");
}
});
}
and URL is a url that gets passed in and term is the term to search for in the xml file (usually "true" or "True"). but before anything happens I end up with "XMLHttpRequest cannot load http://api.own3d.tv/liveCheck.php?live_id=6815. Origin (my server's URL) is not allowed by Access-Control-Allow-Origin."
I've looked into it all over the net and i dont seem to be able to find anything that I can use. there's alot of theory stuff but not enough actual code and i dont understand the theory stuff to be able to start typing code out. from what i've seen you have 2 ways to go, use JSONP or add a line somewhere in your sever to allow for cross-domain accessing. neither of which i understand fully nor know how or what to do. It would be alot of help for someone to show me what needs to be done to get rid of this error. of course if you can explain it to a non-coder like me it would be even more awesome but at my current point, as long as the code works for all I care it might as well be magic lol.

You can solve it :)
Take a look at xReader
<script src="http://kincrew.github.com/xReader/xReader.full.js"></script>
<script type="text/javascript">
xReader("http://api.own3d.tv/liveCheck.php?live_id=6815", function(data) {
alert(data.content);
})
</script>
I think you need cacheburst option. but you can be banned from YQL.

I think its because the path is not relative. You may be calling this from a different domain/sub-domain. You can potentially allow other origins to access, which may open up a security hole or you can create a proxy locally.
In PHP creating a proxy is easy: http://blog.proxybonanza.com/programming/php-curl-with-proxy/
Now, instead of directing your request straight to that URL send the request from jQuery to your own local url and have it access it on the server side.
Another option would be to use YQL: http://www.parrisstudios.com/?p=333 (I wrote an article about this a while ago)... In that way you can turn the response into JSON, which can be accessed cross-domain (as can javascript).
You could ask for the API responses to all be returned using a JSONP server and in JSON.

You aren't going to be able to do this via client-side javascript unless they've enabled some way to retrieve their data cross-domain (CORS, JSONP, some flash widgety thing getting read permissions from crossdomain.xml file(s) located on their server...)

Short answer: unless 10winstreak offers a JSONP service, you'll have to do things on the server-side.
Slightly longer answer:
For security reasons browsers won't let you make AJAX requests from www.example.com to www.example2.com (or any other domain except www.example.com). There isn't much you can do about this except use JSONP (and you can only do that if the remote webservice offers it).
Therefore, what you end up needing to do is ask your server "hey what's on that other server?" and (since it's not limited the way a browser is) it can go get the XML from that other server. There are various ways of doing this, either with code or Apache config; not sure what's right for you, but hopefully now you understand the general principle.
P.S. See this question: Wouldn't have been simpler to just discard cookies for cross-domain XHR? if you are curious why browsers do this.
* EDIT *
I just checked out JustinTV's site, and it appears that they already have a PHP library for you to use:
https://github.com/jtvapi/jtv_php_api
This is very likely your best bet (if you want to keep using PHP that is; if not they have libraries for other languages: http://www.justin.tv/p/api).

how to use JavaScript to sniff url header

the url is input by end users as string on my page, so may point to any domains.
JavaScript in current page needs to sniff the url, verify whether it's still valid, and return the types as image, or video, or audio, even considering html5 video audio tag and existent flash embed. And No need to wait for the complete file transfer.
Can someone help, from concept? thanks very much.
i'm aware the cross domain problem on ajax. So no idea on basic how-to.

If what you're asking, is:
Given any URL -> lookup given URL using a javascript ajax request, and determine if it is a video/audio/image - then, once detected, use the URL accordingly, then you can do something like this:
jQuery and AJAX response header
However, you'll not be able to make a request using client-side JavaScript to another domain, as it will require a cross-domain request (where your alternatives are JsonP, or weird headers in the response).
You're better off passing the URL to your own server, and performing the logic there (Via some kind of server-side web request) and passing a payload back to the client, with the required information in JSON or something - e.g.
{payload: 'video'}

Old question, but I recently wrote a utility that might help you out. It's a CORS-enabled MIME-type checker. See the API doc at lecoq.herokuapp.com
Use it like so: example

Perl - Submit Javascript action to host

I am building a Spider in Perl and have a problem:
The Site I want to spider uses a JavaScript for Age-Verification and I don't know how to get past this in Perl...?
The Script looks like this:
<script type = "text/javascript">
function set_age_verified(){
new Request({
method: "post",
url: "/user/set_age_verified"
}).send();
$('age_verification').setStyles({visibility: 'hidden', display: 'none'});
$('page_after_verification').setStyles({visibility: 'visible', display: 'block'});
return false;
}
</script>
And here the OnClick Event :
<img src="http://example.com/age-verification-enter.gif" alt="ENTER">

The function has two effects. One is to POST a request to the URL "/user/set_age_verified" and the other is to alter the display visibility of some HTML.
Your spider can easily ignore the second effect, but presumably the first effect, by going to the server, sets some cookie or server variable which the server will require.
You do not have to actually run the javascript, so long as the server sees the same POST data.
The answer is for your Perl script to detect pages which have this javascript, and to call a Perl function to POST the data to the age verification URL.
Any cookie or similar which is returned will have to be recorded by you - your HTTP library may take care of this for you though.

What Perl modules are you using? WWW::Mechanize has an AJAX plugin, although it hasn't been updated in a while. I guess you could also look at something like WWW::Selenium.
But I bet that AJAX request is going to inject some HTML that requires the user to input some data, then submit a form. Pretty tricky to cover all bases for that general case...

Take a look at the WWW::Mechanize::Firefox module. It allows you handle some JavaScript.

Also, in Firefox HTTPHeaders is your best friend.
Turn it on, manually click what ever you need to in order for the Javascript to run and submit to the server, then go back to the HTTPHeaders window. It will show you exactly what that Javascript event sent to the server (GET or POST + the data, even if it is HTTPS) - as well as the server response.

Cross domain Ajax request from within js file

Here's the problem:
1.) We have page here... www.blah.com/mypage.html
2.) That page requests a js file www.foo.com like this...
<script type="text/javascript" src="http://www.foo.com/jsfile.js" />
3.) "jsfile.js" uses Prototype to make an Ajax request back to www.foo.com.
4.) The ajax request calls www.foo.com/blah.html. The callback function gets the html response and throws it into a div.
This doesn't seem to work though, I guess it is XSS. Is that correct?
If so, how can I solve this problem? Is there any other way to get my html from www.foo.com to www.blah.com on the client without using an iframe?

It is XSS and it is forbidden. You should really not do things that way.
If you really need to, make your AJAX code call the local code (PHP, ASP, whatever) on blah.com and make it behave like client and fetch whatever you need from foo.com and return that back to the client. If you use PHP, you can do this with fopen('www.foo.com/blah.html', 'r') and then reading the contents as if it was a regular file.
Of course, allow_remote_url_fopen (or whatever it is called exactly) needs to be enabled in your php.ini.

There is a w3c proposal for allowing sites to specify other sites which are allowed to make cross site queries to them. (Wikipedia might want to allow all request for articles, say, but google mail wouldn't want to allow requests - since this might allow any website open when you are logged into google mail to read your mail).
This might be available at some point in the future.

As mentioned above JSONP is a way around this. However, the site that you are requesting the data from needs to support JSONP in order for you to use on the client. (JSONP essentially injects a script tag into the page, and provides a callback function that should be called with the results)
If the site you are making a request to does not support JSONP you will have to proxy the request on your server. As mentioned above you can do this on your own server or what I have done in the past is use a http://www.jsonpit.com, which will proxy the request for you.

One option is to implement a proxy page which takes the needed url as a parameter. e.g. http://blah.com/proxy?uri=http://foo.com/actualRequest

JSONP was partially designed to get around the problem you are having
http://ajaxian.com/archives/jsonp-json-with-padding
JQuery has it in their $.getJSON method
http://docs.jquery.com/Ajax/jQuery.getJSON

The method shown above could become a large security hole.
Suggest you verify the site name against a white list and build the actual URI being proxied on the server side.

For cross domain hits this is a good working example and now is considered as some what "standard" http://www.xml.com/pub/a/2005/12/21/json-dynamic-script-tag.html.
there are other ways as well, for eg injecting iframes with document.domain altered
http://fettig.net/weblog/2005/11/28/how-to-make-xmlhttprequest-connections-to-another-server-in-your-domain/
I still agre that the easy way is calling a proxy in same domain but then it's not truly client side WS call.