Develop Reference

How to use requests in Python 3 to fetch data from website that utilizes JavaScript and jQuery

I have been playing around with the requests library in Python 3 for quite some time now, and have decided to create a test program. For this program, I'm using the website https://ytmp3.cc/ as an example. But it turns out that a lot is going on, on the client-side it seems.
Some keys and other stuff are being generated, and I have been using Firefox's built-in network monitor, to figure out in which requests this is being made, but without luck.
As far as I know, the requests-library can't keep a "page" open and modify the DOM and content, by making more requests.
Anyone whom could take a look, and give a qualified guess on how the special keys are generated, and how I could possibly get these for my own requests.
Fx when loading the webpage, the first request made is for the root, and the response contains the webpage HTML. What I noticed is that at the bottom, there's an url containing some key and number.
<script id="cs" src="js/converter-1.0.js?o=7_1a-a&=_1519520467"></script>
id 7_1a-a
number _1519520467`
This is used for making the next request, but then a lot of following requests are being made, and some other keys are made as well. But I can't find where these come from since they are not returned by a request.
I know that when inserting a Youtube link, a request will be made to an url, as seen below.
https://d.ymcdn.cc/check.php?callback=jQuery33107639361236859977_1519520481166&v=eVD9j36Ke94&f=mp3&k=7_1a-a&_=1519520481168
This returns the following:
jQuery33107639361236859977_1519520481166({"sid":"21","hash":"2a6b2475b059101480f7f16f2dde67ac","title":"M\u00d8 - Kamikaze (Official Video)","ce":1,"error":""})
From this I can construct the download url, using the hash from above:
https://yyd.ymcdn.cc/ + 2a6b2475b059101480f7f16f2dde67ac (hash) + /eVD9j36Ke94 (youtube video id)
But how do I get
jQuery33107639361236859977_1519520481166&v=eVD9j36Ke94 and 1519520481168
Which I need to create the request?

You can probably save yourself and the operator of that website a lot of headache by just using youtube-dl, specifically with the --extract-audio --audio-format mp3 options. It's probably what that website itself uses.
youtube-dl is written in Python and can easily be used programatically.
If you insist on sending requests to that website for whatever reason, here's how I'd do it:
callback=jQuery33107639361236859977_1519520481166 specifies the name of the callback for the JSONP request. Any name you provide will be printed back out. For example, passing callback=foo will result in the following response:
foo({...})
You can omit it entirely and the server will serve just a JSON response in this case, which is nice.
_=1519520481168 is just to prevent the response being cached. It's randomly generated, just like the above parameter. The website checks for existence, however, so you have to at least pass something in.
The website, like many, checks for a valid Referer header.
Here's a minimal cURL command line to make a request to that website:
curl 'https://d.ymcdn.cc/check.php?v=eVD9j36Ke94&f=mp3&k=aZa4__&_=1' -H 'Referer: https://ytmp3.cc/'

IE : window.open url which is more than 2,083 characters

I am getting a url from server with lot of data as query string
(E.g. http://www.test.com/?n=1,2,3,4,5,6,7,8.....100000) and I want to open it using window.open().
But the moment I pass the URL to window.open the url which gets gets truncated. After searching for sometime I could figure out that the maximum limit for URL is 2,083 characters(IE) so it passes PART of query string and truncates the rest..
How can I overcome this?
Please let me know if I need to provide more details.

I think the only solution is to use POST instead of GET. Just use a form, instead of window.open. Please see this older answer:
https://stackoverflow.com/a/17089124/907420

You could try URL shorteners, like goo.gl or bit.ly:
https://goo.gl/
Speaking of programming, you could try to shorten your URL-s yourself, for the given example:
Exact URL for your example, shorter by ~2000 characters.
Where x..y is translated on server side as range(x, y) -> 1,2,3,4,5...100000 for x and y being 1 and 100000

If you want to stick with GET (and I would recommend that you use POST) you can try to compress the parameters. Instead of giving a huge number of parameters you create a javascript object holding the parameters, jsonfy it and you end up with a string that can be compressed and uncompressed again at the other end. Afterwards you can deserialize the JSON string and you have your parameters. Depending on the number of parameters it might still not be sufficient for a GET request.
But at the end a POST request is the best solution I think.

How do I send a long JSON string via a GET/POST request in IE8?

I have a long JSON string that I want to send from the front-end to PHP on the back-end, but it doesn't seem to work in IE8 because the string exceeds the 2,048 character limit.
The request gets cut off regardless of whether I use a GET request or a POST request.
Are there any ways to get around this limitation?
Thank you.
Edit: To save time reading, the solution is to use POST instead of GET. POST was in fact working, but I had another error in my code, which led me to believe that it wasn't working.

You should just drop the support to IE8.
But, if you really want to use IE8, are you sure you can't use POST for that?
According to this, the limit is on the URL length, but shouldn't affect the POST length (that's what I made of what's written there, perhaps I'm wrong; but I don't have any IE8 installed, so I can't test that).

Most efficient way to send ascii characters to mailto without reaching URL limit

I have a form which is submitted via mailto to a email server.
As you most know, there is a limitation to the mailto content over which it won't work because it exceeds URL characters limit.
I developed some custom data compression that are domain specific, but it is still not enough (In case all fields are filled, it will bust the limit, this is rare... but rare is bad enough for the client. Never is better.).
I found the Lempel–Ziv–Welch algorithm (http://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Welch) and concluded it would allow me to save 40% of the length average.
Unfortunately, I need of course to call encodeURIComponent to send it to mailto, and as LSW algorightm will return many URL unsupported characters this will in fact make it worse once URL encoded.
Before you tell me it would be easier to make a post to a server using server-side language, let me tell you this is a really unique situation where the form has to be submitted via email via a client-side application, because emails are the only way to connect with the outside world for the end users...
So, do you know any way to compress data efficiently without encodeURIComponent ruining it all ?
Or is there a way to send content to mailto without going through browser ?
I've seen some ways to open Outlook with ActiveX and stuff, but this is pretty browser/email client specific.
Also I checked for options where I save form info in a file using javascript... but the application users are, well let's just say they are not experts at all, and from what I've been told, they could fail to attach the email. (yes, they are that bad)
So I look for the simplest option, where user involvment is almost 0 and where the result is an email sent with the form data, all of that without server-side languages, with a compression algorithm if applicable.
Thanks a lot for your help !

You'll have a hard time getting to "never" with compression, since there will always be strings that a compressor expands instead of compresses. (Basic mathematical property of compression.)
Having said that, there are much better compressors than LZW, depending on the length of your input. You should try zlib and lzma. The binary output of those would then need to be coded using only the allowed URL characters.

Ajax string length limit?

Is there a limit to the length of the parameter that be can added to the url for ajax? I am using Thin server on Ruby, and did an ajax request from the web browser in this format:
io=new XMLHttpRequest();
io.open("GET","http://localhost:3000&v="+encodeURIComponent(JSON.stringify(v)),true);
When the length of the string v exceeds about 7000 bytes, it seems to crash. When less, it seems to work. Is my observation right? Where is the restriction coming from? From Thin, Javascript, or the browser? I use Google Chrome browser.

Is there a limit to the length of the parameter that can added to the url for ajax?
Yes, if you are using a GET request there's a limit which will depend on the client browser. And this limit has nothing to do with AJAX. IIRC it was around 4K for IE but things might have changed. But in any case there's a limit. If you don't want to be limited you should use POST.

Restriction most likely comes from the browser. According to this discussion you should try to keep your URLs under about 2000 characters.

There is a limit to the GET request depending on the character bytes. If you use ASCII it's 256 characters including the url itself. For UTF-8 it's practically the half because 1 utf character is 2bytes long.
You won't have this problem on POST though.