Develop Reference

How to use requests in Python 3 to fetch data from website that utilizes JavaScript and jQuery

I have been playing around with the requests library in Python 3 for quite some time now, and have decided to create a test program. For this program, I'm using the website https://ytmp3.cc/ as an example. But it turns out that a lot is going on, on the client-side it seems.
Some keys and other stuff are being generated, and I have been using Firefox's built-in network monitor, to figure out in which requests this is being made, but without luck.
As far as I know, the requests-library can't keep a "page" open and modify the DOM and content, by making more requests.
Anyone whom could take a look, and give a qualified guess on how the special keys are generated, and how I could possibly get these for my own requests.
Fx when loading the webpage, the first request made is for the root, and the response contains the webpage HTML. What I noticed is that at the bottom, there's an url containing some key and number.
<script id="cs" src="js/converter-1.0.js?o=7_1a-a&=_1519520467"></script>
id 7_1a-a
number _1519520467`
This is used for making the next request, but then a lot of following requests are being made, and some other keys are made as well. But I can't find where these come from since they are not returned by a request.
I know that when inserting a Youtube link, a request will be made to an url, as seen below.
https://d.ymcdn.cc/check.php?callback=jQuery33107639361236859977_1519520481166&v=eVD9j36Ke94&f=mp3&k=7_1a-a&_=1519520481168
This returns the following:
jQuery33107639361236859977_1519520481166({"sid":"21","hash":"2a6b2475b059101480f7f16f2dde67ac","title":"M\u00d8 - Kamikaze (Official Video)","ce":1,"error":""})
From this I can construct the download url, using the hash from above:
https://yyd.ymcdn.cc/ + 2a6b2475b059101480f7f16f2dde67ac (hash) + /eVD9j36Ke94 (youtube video id)
But how do I get
jQuery33107639361236859977_1519520481166&v=eVD9j36Ke94 and 1519520481168
Which I need to create the request?

You can probably save yourself and the operator of that website a lot of headache by just using youtube-dl, specifically with the --extract-audio --audio-format mp3 options. It's probably what that website itself uses.
youtube-dl is written in Python and can easily be used programatically.
If you insist on sending requests to that website for whatever reason, here's how I'd do it:
callback=jQuery33107639361236859977_1519520481166 specifies the name of the callback for the JSONP request. Any name you provide will be printed back out. For example, passing callback=foo will result in the following response:
foo({...})
You can omit it entirely and the server will serve just a JSON response in this case, which is nice.
_=1519520481168 is just to prevent the response being cached. It's randomly generated, just like the above parameter. The website checks for existence, however, so you have to at least pass something in.
The website, like many, checks for a valid Referer header.
Here's a minimal cURL command line to make a request to that website:
curl 'https://d.ymcdn.cc/check.php?v=eVD9j36Ke94&f=mp3&k=aZa4__&_=1' -H 'Referer: https://ytmp3.cc/'

How to map a custom URL to a REST resource in an SPA?

I have a design issue with my SPA, and hope someone can give me some direction. A user profile page is rendered like this:
The browser fetches /some-username.
The server checks to see if the request was a XMLHTTPRequest or not. It is not, and so it simply returns the bundled javascript app to the browser to execute.
The javascript bundle is executed in the browser, it sees the current URL and makes an AJAX request, again to /some-username.
The server sees the XMLHTTPRequest header, looks up the user who has the custom URL "/some-username" and returns the JSON data about the user back to the javascript to render.
This feels wrong. The app should be making RESTful requests to /users/:id to fetch the user data. But how can it know the id that corresponds to the user with the URL /some-username?
It is worth adding an extra HTTP request just to look up the resource identifier? Something like /get_user_id?url=/some-username.

Are you flexible about your API? If so you may change /some-username to /user-id or if you want to include username /user-id/username but ignore username.
As alternative it is also common to make requests in a filter form. Like /users?username=peter
And feel free to use /users/peter if your username identifies the user. Becuase it's actualy the id (that doesn't have to be integer) and then your url is exactly /users/:id

There is nothing "unRESTful" about /some-username. It's just another resource. The response - I hope - contains the canonical URL /user/id anyway, either as a header or as some kind of "self" link.
That's also how you could achieve your goal. Embed the URL in the page either as JavaScript or as a header equivalent (unfortunately you cannot read the headers of the page request with JavaScript):
//header. Can also use a custom header like X-User-Location
<meta http-eqiv="Location" content="/user/id">
//JavaScript
<script>
var userURL = '/user/id
</script>
I recommend keeping your current approach.

Requesting remote XML data with javascript

Ok here's my problem. I'm working on this little site called 10winstreak and I'm trying to detect if a stream is live or not with javascript because our server that we run the site off of cant handle processing every single request with PHP. The basis of detecting if a stream is live or not is you go to their XML file and in one of their tags (if it's live) it will say something along the lines of true and often time the XML file on their site will be empty if a particular stream isn't live. for example if you have a twitch.tv stream for gamespot you go to http://api.justin.tv/api/stream/list.xml?channel=gamespot and if it's got stuff in it then it's live if not then it's not.
so basically my code looks like this:
function check (URL, term){
$.get(URL , function(data){
console.log(data);
//data is whatever the server returns from the request, do whatever is needed with it to show who is live.
var number = data.search(term);
if (number > -1)
{
document.write("Live");
}
else
{
document.write("Offline");
}
});
}
and URL is a url that gets passed in and term is the term to search for in the xml file (usually "true" or "True"). but before anything happens I end up with "XMLHttpRequest cannot load http://api.own3d.tv/liveCheck.php?live_id=6815. Origin (my server's URL) is not allowed by Access-Control-Allow-Origin."
I've looked into it all over the net and i dont seem to be able to find anything that I can use. there's alot of theory stuff but not enough actual code and i dont understand the theory stuff to be able to start typing code out. from what i've seen you have 2 ways to go, use JSONP or add a line somewhere in your sever to allow for cross-domain accessing. neither of which i understand fully nor know how or what to do. It would be alot of help for someone to show me what needs to be done to get rid of this error. of course if you can explain it to a non-coder like me it would be even more awesome but at my current point, as long as the code works for all I care it might as well be magic lol.

You can solve it :)
Take a look at xReader
<script src="http://kincrew.github.com/xReader/xReader.full.js"></script>
<script type="text/javascript">
xReader("http://api.own3d.tv/liveCheck.php?live_id=6815", function(data) {
alert(data.content);
})
</script>
I think you need cacheburst option. but you can be banned from YQL.

I think its because the path is not relative. You may be calling this from a different domain/sub-domain. You can potentially allow other origins to access, which may open up a security hole or you can create a proxy locally.
In PHP creating a proxy is easy: http://blog.proxybonanza.com/programming/php-curl-with-proxy/
Now, instead of directing your request straight to that URL send the request from jQuery to your own local url and have it access it on the server side.
Another option would be to use YQL: http://www.parrisstudios.com/?p=333 (I wrote an article about this a while ago)... In that way you can turn the response into JSON, which can be accessed cross-domain (as can javascript).
You could ask for the API responses to all be returned using a JSONP server and in JSON.

You aren't going to be able to do this via client-side javascript unless they've enabled some way to retrieve their data cross-domain (CORS, JSONP, some flash widgety thing getting read permissions from crossdomain.xml file(s) located on their server...)

Short answer: unless 10winstreak offers a JSONP service, you'll have to do things on the server-side.
Slightly longer answer:
For security reasons browsers won't let you make AJAX requests from www.example.com to www.example2.com (or any other domain except www.example.com). There isn't much you can do about this except use JSONP (and you can only do that if the remote webservice offers it).
Therefore, what you end up needing to do is ask your server "hey what's on that other server?" and (since it's not limited the way a browser is) it can go get the XML from that other server. There are various ways of doing this, either with code or Apache config; not sure what's right for you, but hopefully now you understand the general principle.
P.S. See this question: Wouldn't have been simpler to just discard cookies for cross-domain XHR? if you are curious why browsers do this.
* EDIT *
I just checked out JustinTV's site, and it appears that they already have a PHP library for you to use:
https://github.com/jtvapi/jtv_php_api
This is very likely your best bet (if you want to keep using PHP that is; if not they have libraries for other languages: http://www.justin.tv/p/api).

how to use JavaScript to sniff url header

the url is input by end users as string on my page, so may point to any domains.
JavaScript in current page needs to sniff the url, verify whether it's still valid, and return the types as image, or video, or audio, even considering html5 video audio tag and existent flash embed. And No need to wait for the complete file transfer.
Can someone help, from concept? thanks very much.
i'm aware the cross domain problem on ajax. So no idea on basic how-to.

If what you're asking, is:
Given any URL -> lookup given URL using a javascript ajax request, and determine if it is a video/audio/image - then, once detected, use the URL accordingly, then you can do something like this:
jQuery and AJAX response header
However, you'll not be able to make a request using client-side JavaScript to another domain, as it will require a cross-domain request (where your alternatives are JsonP, or weird headers in the response).
You're better off passing the URL to your own server, and performing the logic there (Via some kind of server-side web request) and passing a payload back to the client, with the required information in JSON or something - e.g.
{payload: 'video'}

Old question, but I recently wrote a utility that might help you out. It's a CORS-enabled MIME-type checker. See the API doc at lecoq.herokuapp.com
Use it like so: example

How is this working?

I was browsing through one site called BSEINDIA.com (http://www.bseindia.com/stockreach/stockreach.htm?scripcd=532667), i Noticed that on click of Get Quote it seems to fire an Ajax request and get the price of selected equities. I tried to segregate this request and fire it separately, but it doesn't seem to work.
I copied over the code from the HTML of same page (http://www.bseindia.com/stockreach/stockreach.htm?scripcd=532667)
Any pointers why is this not working, is there some sort of Authentication going on , i am not even a member of this site??
following is what i am trying to do
<script type="text/javascript">
var oHTTP=getHTTPObject();
var seconds = Math.random().toString(16).substring(2);
if(oHTTP)
{
oHTTP.open("GET","http://www.bseindia.com/DotNetStockReachs/DetailedStockReach.aspx?GUID="+seconds+"&scripcd=532667",true);
oHTTP.onreadystatechange=AJAXRes;
oHTTP.send(null);
}
function AJAXRes()
{
if(oHTTP.readyState==4)alert(oHTTP.responseText);
}
function getHTTPObject(){var obj;
try{obj=new ActiveXObject("Msxml2.XMLHTTP");}
catch(e){try{
obj=new ActiveXObject("Microsoft.XMLHTTP");}
catch(e1){obj=null;}}
if(!obj&& typeof XMLHttpRequest!='undefined'){
try{obj=new XMLHttpRequest();}
catch(e){obj=false;}}return obj;}
</script>
Found out my Answer here
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.referer%28VS.71%29.aspx

Actually, it is fairly easy. When you send an HTTP request, an header called Referrer gets sent with the request. The Referrer is basically the URL of the page which initiated the request.
BSEINDIA checks the Referrer value to make sure that the request is coming from their site. If it is, it sends the data. If not, it sends its 404 page.
You can easily test that theory by disabling the Referrer in your browser. In Firefox, you can do that by typing about:config and setting network.http.sendRefererHeader to 0.
If you still want to get the data, you will need to write a script (in PHP or another language) which will make the request with the proper Referrer and output the results.

There might be some form of IP restriction in place for accessing the files / data needed to save themselves from third party scripts accessing their data through their own scripts. Thats what I'd do.

Possibly Http Referrer. Make sure you do not break any copyright restriction.