Develop Reference

How to use requests in Python 3 to fetch data from website that utilizes JavaScript and jQuery

I have been playing around with the requests library in Python 3 for quite some time now, and have decided to create a test program. For this program, I'm using the website https://ytmp3.cc/ as an example. But it turns out that a lot is going on, on the client-side it seems.
Some keys and other stuff are being generated, and I have been using Firefox's built-in network monitor, to figure out in which requests this is being made, but without luck.
As far as I know, the requests-library can't keep a "page" open and modify the DOM and content, by making more requests.
Anyone whom could take a look, and give a qualified guess on how the special keys are generated, and how I could possibly get these for my own requests.
Fx when loading the webpage, the first request made is for the root, and the response contains the webpage HTML. What I noticed is that at the bottom, there's an url containing some key and number.
<script id="cs" src="js/converter-1.0.js?o=7_1a-a&=_1519520467"></script>
id 7_1a-a
number _1519520467`
This is used for making the next request, but then a lot of following requests are being made, and some other keys are made as well. But I can't find where these come from since they are not returned by a request.
I know that when inserting a Youtube link, a request will be made to an url, as seen below.
https://d.ymcdn.cc/check.php?callback=jQuery33107639361236859977_1519520481166&v=eVD9j36Ke94&f=mp3&k=7_1a-a&_=1519520481168
This returns the following:
jQuery33107639361236859977_1519520481166({"sid":"21","hash":"2a6b2475b059101480f7f16f2dde67ac","title":"M\u00d8 - Kamikaze (Official Video)","ce":1,"error":""})
From this I can construct the download url, using the hash from above:
https://yyd.ymcdn.cc/ + 2a6b2475b059101480f7f16f2dde67ac (hash) + /eVD9j36Ke94 (youtube video id)
But how do I get
jQuery33107639361236859977_1519520481166&v=eVD9j36Ke94 and 1519520481168
Which I need to create the request?

You can probably save yourself and the operator of that website a lot of headache by just using youtube-dl, specifically with the --extract-audio --audio-format mp3 options. It's probably what that website itself uses.
youtube-dl is written in Python and can easily be used programatically.
If you insist on sending requests to that website for whatever reason, here's how I'd do it:
callback=jQuery33107639361236859977_1519520481166 specifies the name of the callback for the JSONP request. Any name you provide will be printed back out. For example, passing callback=foo will result in the following response:
foo({...})
You can omit it entirely and the server will serve just a JSON response in this case, which is nice.
_=1519520481168 is just to prevent the response being cached. It's randomly generated, just like the above parameter. The website checks for existence, however, so you have to at least pass something in.
The website, like many, checks for a valid Referer header.
Here's a minimal cURL command line to make a request to that website:
curl 'https://d.ymcdn.cc/check.php?v=eVD9j36Ke94&f=mp3&k=aZa4__&_=1' -H 'Referer: https://ytmp3.cc/'

How to properly cache AJAX

I'm doing some research on how to properly cache AJAX responses, since that speeds up a page with lots of AJAX requests. I found this piece on the Yahoo website:
Let's look at an example. A Web 2.0 email client might use Ajax to download the user's address book for autocompletion. If the user hasn't modified her address book since the last time she used the email web app, the previous address book response could be read from cache if that Ajax response was made cacheable with a future Expires or Cache-Control header. The browser must be informed when to use a previously cached address book response versus requesting a new one. This could be done by adding a timestamp to the address book Ajax URL indicating the last time the user modified her address book, for example, &t=1190241612. If the address book hasn't been modified since the last download, the timestamp will be the same and the address book will be read from the browser's cache eliminating an extra HTTP roundtrip. If the user has modified her address book, the timestamp ensures the new URL doesn't match the cached response, and the browser will request the updated address book entries.
This makes it only less clear. The reason I want to know all this, is that I'm building a simple webpage where users can add shortcuts to websites. They see a grid of icons and can click on or search for the website they need. This is only meant as a project to get to know PHP and most importantly AJAX a lot better; nothing that actual users will ever see.
As you can imagine, the search function slows the website down a lot. Especially since it's performing an AJAX request after every typed letter. Therefore, I think it would greatly improve the website if some parts of this would be cached.

You have to make up your mind first:
Do you want a cached response used, or do you want the server to create a fresh one each time?
It isn't clear to me what you are asking exactly. What do you want to be cached, and how long?
If you want a cached response, simply use an URL you used before. (In combination with an expires-header)
For example:
http://www.example.com/myAjaxHelper.php?q=ab
If server responds with an expires header that says "one week from now", you will get the cached response for a week. Your browser will make that so.
(Remember YOU can set up the expires value in the response headers)
There are problems with browsers NOT respecting the expires header, and sometimes they fetch the cached version instead of the new one, even if it expired in the past. (Notably older IE needed a few of extra headers to fix that, and it is highly confusing)
If you want a fresh response (and this can lead to slowness because on each key-up you make a roundtrip to the server), make a NEW url.
Eg:
http://www.example.com/myAjaxHelper.php?q=ab?time=< ?php echo time(); ?>
This will lead to a new URL each second. And that one will never be cached, because the URL differs (only for the time=.... part, but that is enough to force a new request).

Requesting remote XML data with javascript

Ok here's my problem. I'm working on this little site called 10winstreak and I'm trying to detect if a stream is live or not with javascript because our server that we run the site off of cant handle processing every single request with PHP. The basis of detecting if a stream is live or not is you go to their XML file and in one of their tags (if it's live) it will say something along the lines of true and often time the XML file on their site will be empty if a particular stream isn't live. for example if you have a twitch.tv stream for gamespot you go to http://api.justin.tv/api/stream/list.xml?channel=gamespot and if it's got stuff in it then it's live if not then it's not.
so basically my code looks like this:
function check (URL, term){
$.get(URL , function(data){
console.log(data);
//data is whatever the server returns from the request, do whatever is needed with it to show who is live.
var number = data.search(term);
if (number > -1)
{
document.write("Live");
}
else
{
document.write("Offline");
}
});
}
and URL is a url that gets passed in and term is the term to search for in the xml file (usually "true" or "True"). but before anything happens I end up with "XMLHttpRequest cannot load http://api.own3d.tv/liveCheck.php?live_id=6815. Origin (my server's URL) is not allowed by Access-Control-Allow-Origin."
I've looked into it all over the net and i dont seem to be able to find anything that I can use. there's alot of theory stuff but not enough actual code and i dont understand the theory stuff to be able to start typing code out. from what i've seen you have 2 ways to go, use JSONP or add a line somewhere in your sever to allow for cross-domain accessing. neither of which i understand fully nor know how or what to do. It would be alot of help for someone to show me what needs to be done to get rid of this error. of course if you can explain it to a non-coder like me it would be even more awesome but at my current point, as long as the code works for all I care it might as well be magic lol.

You can solve it :)
Take a look at xReader
<script src="http://kincrew.github.com/xReader/xReader.full.js"></script>
<script type="text/javascript">
xReader("http://api.own3d.tv/liveCheck.php?live_id=6815", function(data) {
alert(data.content);
})
</script>
I think you need cacheburst option. but you can be banned from YQL.

I think its because the path is not relative. You may be calling this from a different domain/sub-domain. You can potentially allow other origins to access, which may open up a security hole or you can create a proxy locally.
In PHP creating a proxy is easy: http://blog.proxybonanza.com/programming/php-curl-with-proxy/
Now, instead of directing your request straight to that URL send the request from jQuery to your own local url and have it access it on the server side.
Another option would be to use YQL: http://www.parrisstudios.com/?p=333 (I wrote an article about this a while ago)... In that way you can turn the response into JSON, which can be accessed cross-domain (as can javascript).
You could ask for the API responses to all be returned using a JSONP server and in JSON.

You aren't going to be able to do this via client-side javascript unless they've enabled some way to retrieve their data cross-domain (CORS, JSONP, some flash widgety thing getting read permissions from crossdomain.xml file(s) located on their server...)

Short answer: unless 10winstreak offers a JSONP service, you'll have to do things on the server-side.
Slightly longer answer:
For security reasons browsers won't let you make AJAX requests from www.example.com to www.example2.com (or any other domain except www.example.com). There isn't much you can do about this except use JSONP (and you can only do that if the remote webservice offers it).
Therefore, what you end up needing to do is ask your server "hey what's on that other server?" and (since it's not limited the way a browser is) it can go get the XML from that other server. There are various ways of doing this, either with code or Apache config; not sure what's right for you, but hopefully now you understand the general principle.
P.S. See this question: Wouldn't have been simpler to just discard cookies for cross-domain XHR? if you are curious why browsers do this.
* EDIT *
I just checked out JustinTV's site, and it appears that they already have a PHP library for you to use:
https://github.com/jtvapi/jtv_php_api
This is very likely your best bet (if you want to keep using PHP that is; if not they have libraries for other languages: http://www.justin.tv/p/api).

request parameters ordering undefined [in multipart/form-data or in general] - what to do?

I am writing a web application that submits a form (one of its fields is mulitpart/form-data, so obviously POST must be used and not GET, since the files might be really big). One of the fields is kinda transaction/upload_id and the other is obviously the file contents. While uploading, a progress bar must be displayed.
The known fact says that the order of parameters is undefined in general, meaning that any of (file content / upload_id) might come first.
Is there any acceptable / recommended way to cause the browser to send the upload_id before sending the file content?
Is it a considered a correct implementation - to expect the upload_id to come first OR there is a better / most common / more correct way to handle the problem? In that case - it would be fantastic to hear some details.
Update: my server-side language is Java/Servlets 3.0

Well, the better answer (without utilizing filters) would be to publish the upload_id(s) as a part of the URL (after '?'), even when issuing a POST request. In that case, they will be always processed ahead of files' contents.

Using servlets as well, and in my case I wanted to run my CSRF filter in my servlet before I started streaming the file: if the filter failed, I can kill the request before I've uploaded my 20gb video file, as opposed to the default PHP implementation where the server only hits your script AFTER its parsed the entire request.
Its been a bit of a hack on my part, but in the couple of cases I've had to do this I've cheated and put the non-file request parameters into the URL and in every case (using pretty much every browser I've tested with) an Iterator over the request parameters on the server (I'm using commons fileupload in streaming mode) has received the non-file request parameters first before the file data was received. Somewhat fragile, but not unworkable.
I'm assuming that if you order your request parameters with the file <input> as the last item you'll get the same behavior.

You shouldn't have to worry about the order in which the parameters are sent. If so, then your server-side code is very brittle.
A multi-part request will contain the field name of every form field that is passed in. Use the name to reference that field regardless of the order it was sent in.
If you are parsing the post body by hand, I suggest you look at existing projects like Apache FileUpload which abstract that away.

Disable browser cache

I implemented a REST service and i'm using a web page as client.
My page has some javascript functions that performs several times the same http get request to REST server and process the replies.
My problem is that the browser caches the first reply and not actualy sends the following requests..
Is there some way to force the browser execute all the requests without caching?
I'm using internet explorer 8.0
Thanks

Not sure if it can help you, but sometimes, I add a random parameter in the URL of my request in order to avoid being cached.
So instead of having:
http://my-server:8080/myApp/foo?bar=baz
I will use:
http://my-server:8080/myApp/foo?bar=baz&random=123456789
of course, the value of the random is different for every request. You can use the current time in milliseconds for that.

Not really. This is a known issue with IE, the classic solution is to append a random parameter at the end of the query string for every request. Most JS libraries do this natively if you ask them to (jQuery's cache:false AJAX option, for instance)

Well, of course you don't actually want to disable the browser cache entirely; correct caching is a key part of REST and the fact that it can (if properly followed by both client and server) allow for a high degree of caching while also giving fine control over the cache expiry and revalidation is one of the key advantages.
There is though an issue, as you have spotted, with subsequent GETs to the same URI from the same document (as in DOM document lifetime, reload the page and you'll get another go at that XMLHttpRequest request). Pretty much IE seems to treat it as it would a request for more than one copy of the same image or other related resource in a web page; it uses the cached version even if the entity isn't cacheable.
Firefox has the opposite problem, and will send a subsequent request even when caching information says that it shouldn't!
We could add a random or time-stamped bogus parameter at the end of a query string for each request. However, this is a bit like screaming "THIS IS SPARTA!" and kicking our hard-won download into a deep pit that no Health & Safety inspector considered putting a safety rail around. We obviously don't want to repeat a full unconditional request when we don't need to.
However, this behaviour has a time component. If we delay the subsequent request by a second, then IE will re-request when appropriate while Firefox will honour the max-age and expires headers and not re-request when needless.
Hence, if two requests could be within a second of each other (either we know they are called from the same function, or there's the chance of two events triggering it in close succession) using setTimeout to delay the second request by a second after the first has completed will make it use the cache correctly, rather than in the two different sorts of incorrect behaviour.
Of course, a second's delay is a second's delay. This could be a big deal or not, depending primarily on the size of the downloaded entity.
Another possibility is that something that changes so rapidly shouldn't be modelled as GETting the state of a resource at all, but as POSTing a request for a current status to a resource. This does smell heavily of abusing REST and POSTing what should really be a GET though.
Which can mean that on balance the THIS IS SPARTA approach of appending random stuff to query strings is the way to go. It depends, really.