Python has a module called httplib which allows for the retrieval of an html resource from a URL. With this code:
httpServ = httplib.HTTPConnection("www.google.com")
httpServ.connect()
httpServ.request('GET',"/search?q=python")
...
httpServ.close()
I am trying to do the same thing in my angular app, but using $http get doesn't allow me to retrieve the html document due to the same origin policy.
Is there anything similar to the python method available in JavaScript?
So, the Same-Origin Policy has nothing to do with JavaScript. It basically says "don't allow scripts on a page to talk to scripts being run by another host."
This is an extremely important security feature. It means that if you put jQuery on your page, and somehow a jQuery CDN got hacked and they changed jQuery to send your passwords to another page, it wouldn't work (so long as the browser properly enforces the Same-Origin Policy).
You don't have this problem when working with Python because Python exclusively runs on the server (from a web-app perspective). Your server can talk to any machine it wants to, but browsers do not (and should not as seen above) give that freedom to webpages.
So, how to solve your problem? Make your GET request to a script running on your server. Have your server do a curl or wget or w/e of google.com, then have your server send the data back to the client.
Related
I'm looking for some help with the following:
I'm building a website that links external scripts (these scripts can be changed anytime by the script owner).
I want to route any web requests that's being sent by these scripts through my backend server as a proxy, so that I can parse the request and response to make sure they are not exfiltrating data from my website.
The idea here is to be able to leverage external scripts that cannot be trusted 100% with my data.
To enforce this, is it possible to intercept web requests made by <script>s from a different <script> that loaded early on?
If not, what is a better way?
If you cannot trust the scripts, this becomes more of a security question. You should get more information about web security in general.
An interesting example of such implementation is Tampermonkey and it's permission model (similar to what Android apps).
Depending on the use-case, you may want to manually approve each js file and enforce it via integrity checking:
https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity
At the end of the day, you cannot programmatically check the security of an external js file. You wither trust its author or you don't.
it is possible to crawl a website within an Angular-App? I am speaking about to call a website from Angular, not crawling an Angular-App. If that so, then I am wondering which IP will be shown on the crawled website. Since JavaScript is client-side, I would suggest, its the IP of the client, not of the server (like probably at nodejs). But all I know, its mostly browser-implemented stuff what we can use in JS, so it is even possible to crawl websites with methods from JavaScript (or Angular)?
Best Regards
Buzz
In theory, you can create an AJAX request to fetch the data with reponse type text/html. That would give you the remote document as a string. The browser wouldn't try to load the JavaScript and CSS in that document, though. That might not be a problem but CORS is. For security reasons, most browsers prevent you from loading data from somewhere else (otherwise, it would be too easy for criminals to put JavaScript into any web page). See here for details: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
If you have control over the second domain, you can configure the server there to send Access-Control-Allow-Origin headers to the browser to allow access from the Angular App.
Note: You could use an iframe to load the other website but when the domains of the current document and the one in the iframe don't match, then you can't access the contents of the iframe from JavaScript.
One way to work around this is to install a proxy on your server. The browser can then ask your server for the pages in question. In this case, the remote web site will get the IP of your server.
I have some HTML/PHP pages that include javascript calls.
Those calls points on JS/PHP methods included into a library (PIWIK) stored onto a distant server.
They are triggered using an http://www.domainname.com/ prefix to point the correct files.
I cannot modify the source code of the library.
When my own HTML/PHP pages are locally previewed within a browser, I mean using a c:\xxxx kind path, not a localhost://xxxx one, the distant script are called and do their process.
I don't want this to happen, only allowing those scripts to execute if they are called from a www.domainname.com page.
Can you help me to secure this ?
One can for sure directly bypass this security modifying the web pages on-the-fly with some browser add-on while browsing the real web site, but it's a little bit harder to achieve.
I've opened an issue onto the PIWIK issue tracker, but I would like to secure and protect my web site and the according statistics as soon as possible from this issue, waiting for a further Piwik update.
EDIT
The process I'd like to put in place would be :
Someone opens a page from anywhere than www.domainname.com
> this page calls a JS method on a distant server (or not, may be copied locally),
> this script calls a php script on the distant server
> the PHP script says "hey, from where damn do yo call me, go to hell !". Or the PHP script just do not execute....
I've tried to play with .htaccess for that, but as any JS script must be on a client, it blocks also the legitimate calls from www.domainname.com
Untested, but I think you can use php_sapi_name() or the PHP_SAPI constant to detect the interface PHP is using, and do logic accordingly.
Not wanting to sound cheeky, but your situation sounds rather scary and I would advise searching for some PHP configuration best practices regarding security ;)
Edit after the question has been amended twice:
Now the problem is more clear. But you will struggle to secure this if the JavaScript and PHP are not on the same server.
If they are not on the same server, you will be reliant on HTTP headers (like the Referer or Origin header) which are fakeable.
But PIWIK already tracks the referer ("Piwik uses first-party cookies to keep track some information (number of visits, original referrer, and unique visitor ID)" so you can discount hits from invalid referrers.
If that is not enough, the standard way of being sure that the request to a web service comes from a verified source is to use a standard Cross-Site Request Forgery prevention technique -- a CSRF "token", sometimes also called "crumb" or "nonce", and as this is analytics software I would be surprised if PIWIK does not do this already, if it is possible with their architecture. I would ask them.
Most web frameworks these days have CSRF token generators & API's you should be able to make use of, it's not hard to make your own, but if you cannot amend the JS you will have problems passing the token around. Again PIWIK JS API may have methods for passing session ID's & similar data around.
Original answer
This can be accomplished with a Content Security Policy to restrict the domains that scripts can be called from:
CSP defines the Content-Security-Policy HTTP header that allows you to create a whitelist of sources of trusted content, and instructs the browser to only execute or render resources from those sources.
Therefore, you can set the script policy to self to only allow scripts from your current domain (the filing system) to be executed. Any remote ones will not be allowed.
Normally this would only be available from a source where you get set HTTP headers, but as you are running from the local filing system this is not possible. However, you may be able to get around this with the http-equiv <meta> tag:
Authors who are unable to support signaling via HTTP headers can use tags with http-equiv="X-Content-Security-Policy" to define their policies. HTTP header-based policy will take precedence over tag-based policy if both are present.
Answer after question edit
Look into the Referer or Origin HTTP headers. Referer is available for most requests, however it is not sent from HTTPS resources in the browser and if the user has a proxy or privacy plugin installed it may block this header.
Origin is available for XHR requests only made cross domain, or even same domain for some browsers.
You will be able to check that these headers contain your domain where you will want the scripts to be called from. See here for how to do this with htaccess.
At the end of the day this doesn't make it secure, but as in your own words will make it a little bit harder to achieve.
We're developing a Dynamics CRM 2011 product that has a button in the ribbon that calls an external API. Currently, for this button to work, the following settings need to be changed in the browser (IE):
We would like to avoid this, because many of the target customers for this product are very security conscious. Is there a way to write the code so that it will not require these permissions to be changed, but still be able to communicate with the external API? The code running when the button is pressed in CRM is HTML and Javascript.
Thanks!
Are you in control of the API? If so, look into CORS. With CORS, all you do is basically add a few extra headers to your request response. If you use an AJAX library (like jQuerys $.ajax), you should be able to continue writing code as is. If not, a good article on how to implement cors in Javascript can be found here: http://eriwen.com/javascript/how-to-cors/
To enable cors, read up on http://enable-cors.org/
I don't know anything about this CRM, but other than JSONP, your best bet is to have a server side script act as a proxy.
So, you would create a script within the same domain as the user interface code. That script will then use a server side language (such as PHP) to perform the request to the cross domain script on your behalf. The server side connection has no restriction on which domain it can access, and all the browser knows is that it is sending a request to a page within the calling domain, which is presumably safe.
How you will do this depends on the exact language of choice, but in general you would just need to send the remote API URL as well as any arguments needed to your server side script, which then rebuilds the request to that URL and passes the result back to the client.
I realize this issue of cross site scripting has been covered, however being new to web development I had a few further questions.
Currently I am testing an html file I wrote on my PC connecting to a RESTFul web service on another machine. I am getting status=0. Is this considered cross-site scripting?
If a server hosts a file with javascript, and that javascript file has XMLHttpRequests to the server's own web services, will that work, or is that bad?
Apologies if any of these questions are stupid.
status=0 can me a variety of things, and without knowing more about how you got to that point, it is very difficult to determine what, exactly, it means. You could be using an iframe, the other computer could genuinely be telling you that the status is 0... we don't know.
The general rule is that it doesn't matter where the JS is from, it will execute the data where it's loaded. This is what makes the Google js archiving api possible (you know, use https://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.js on a whole assortment of locations). And honestly, that is not a security issue.
The security issue comes in when a js file tries to access another domain (or even subdomain), whether through manipulation of an iframe or through XMLHTTPRequest. It's at that point that the browser will "lay the smackdown" on the script.
You will have difficulty communicating with JavaScript from your hard drive (file:///) to any internet protocol (http|https) because of this.
No, that is not cross site scripting. When including script JS file from another server it is rendered in your site so You won't be able to access through XMLHttpRequest site where JS script is originally located.
If that is possible than anybody who host jQuery file, there are many servers including google, would be opened for XMLHttpRequests.
SO, IT'S NOT POSSIBLE.
If you want JSON response from another server you can use pjson. Google it for more info.
And Cross Site Scripting is when someone injects JavaScript code on your site in order to bypass access control.
You can use CORS for that. You can use the same code you use now, but the other server you request the page from via ajax has to sent the following header on that page
Access-Control-Allow-Origin: http://yoursite.example.com
#or to allow all hosts
Access-Control-Allow-Origin: *