I´m using the cloudscraper package (PyPI, Github) for web requests on a site that is protected with Cloudflare.
I am well aware there are challenges that can´t be solved yet with this package, particularly the "v2 challenges" with recaptchas and so on.
However, for me, the package seems to not work at all. When I do a GET request with
s.get(my_url)
where s is a Cloudscraper session object, I often get a HTML page with this title: "Attention Required! | Cloudflare".
This is the standard Cloudflare Javascript challenge, which just checks whether the browser supports JS.
I don´t know why this happens. I made sure that
I have a 'realistic' User agent set, with Chrome set as browser argument in the
cloudscraper.CloudScraper()
constructor.
request are timed and not too fast, I wait between requests
I have all the package requirements installed, meaning besides cloudscraper itself, requests, requests-toolbelt, and js2py as engine.
There is no issues section on the Github repo.
The Javascript check is the simplest challenge that Cloudflare can throw at us. Still, this package, which has the sole purpose of solving some Cloudflare challenges, fails to even get past this simple check.
What am I overlooking? Cloudflare makes web automation a nightmare...
EDIT: Also, the Cloudflare page says 'Please enable Cookies and reload the page.' although normally cookies are automatically accepted by the request session´s RequestsCookieJar.
Related
I'm currently making an open source browser extension that will send requests to my site. This can easily be done with Ajax, a request will be sent to the page action.php.
My site will use PHP, well now the question is, how can I make sure action.php receives the request from the original extension? I mean griefers could easily send false information to the server, or a fork could be used and send incorrect data. I thought of generating a token of some sort, but anyone could recreate it I guess.
How can I prevent this situation?
I have some experience with this myself. I've been building an extension with a login and eventually came to the inevitability that security in an extension is inherently difficult.
The issue is that an extension is just a bundle of JS and HTML that anyone can inspect the values of. This means that anyone determined enough to dig through your code can potentially find out how to bypass anything you have built in.
The solution I eventually came to is that, the extension itself cannot hold any long-lasting secrets. A session with a timeout is the only safe thing to store. The actual login for my extension is done via a website over HTTPS.
If you are trying to do this without any such login, your only recourse is to make it as difficult as possible to determine what needs to be sent by using an algorithm that can generate server verifiable tokens, and then only publishing minified code to the webstore.
EDIT: Reread the question and noticed that you said you are doing this open source. Without some sort of authentication on the webserver via HTTPS, there is little you can do to stop those determined to bypass your protections because they will be on display in your public repository.
For sensitive endpoints like this, it would make sense do to the data processing server-side. The client would only have to query the server to process the data.
I'm doing e2e-tests for an app whose frontend is written in AngularJS, and these tests would typically involve filling in forms, sending the data to the backend, then updating the page and making sure that the data persists. The tests are written in protractor.
One of these tests fails, inconsistently and for no apparent reason, so I would like to get as much information for debugging as possible. So I’ve been wondering whether it’s possible at all to log the xhr POST requests that my frontend is sending to the backend during the test in question, or better yet, whether the data that are being sent by the browser can be captured and examined from within protractor? Perhaps, using the browser object? I googled, and googled, but without success.
Yes, I realise that e2e-tests are intended only to interact with the interface and that ajax requests are too low-level for such kind of tests. Yes, perhaps stubbing the whole backend out and just testing the frontend would have been much better. But please humor me. Is it possible to get the information about what is being posted by the browser to the server during e2e-tests with protractor?
Protractor uses the webdriverjs API to "drive" the browser, so it won't have access to any more information than any other Selenium webdriver app would have. See the docs here: http://docs.seleniumhq.org/docs/03_webdriver.jsp#selenium-webdriver-api-commands-and-operations
Outside of some APIs for controlling the browser (adding cookies, opening new tabs), most of the functionality in Protractor and WebdriverJS comes from running snippets of JavaScript in the browser (e.g., to inspect the DOM). So, I don't think any of that qualifies for intercepting communications between the browser and the server.
I think you might have luck using the Protractor infrastructure for injecting code/modules into the app start (this is the best doc I can find for this feature). You should be able to inject a module that can interpose on the $http calls an log them as they go (or, of course, fully mock them out).
I am working on allowing a chrome extension to post a new entry to my site via post data.
I want to be able to lock it down so only the chrome extension can post. If I get post data from anywhere else I want to reject it.
Does anyone know if/how this is possible or how you would go about doing it?
Unfortunately, validating clients (whether a Chrome extension, an Android app, an iOS app, client-side JavaScript, or some other client) from a web server is an unsolved problem.
There are some things that you can do to deter abuse and mitigate this problem such as:
Requiring user authentication (and rate-limiting usage per-user)
Rate-limiting access on the basis of IP addresses
Requiring tokens to be provided that are handed out in prior requests (this can be used to ensure that certain APIs are called in certain expected orders / patterns).
Showing a CAPTCHA or other challenge for anomolous or over-limit usage
While you can additionally check things such as user agent, referrer URL, or a token that you embed in the Chrome extension, with any distributed application, it is easy to reverse-engineer these and mimick them in a counterfeit app, and so these aren't true solutions.
You can add a simple check in the code.
Following code stops anyone who is trying to access your api outside the chrome extension.
if(substr($_SERVER['HTTP_ORIGIN'],0,19) !== "chrome-extension://") die("Not Allowed")
I am developing a firefox addon, and I need to make https calls. I am given an SSL certificate information (Serial Number, SHA1 Fingerprint, andMD5 Fingerprint). When I try to use the Request module from my main.js I always get a status of 0. I tried the Request module with other http requests which are not secured and it works fine.
So I assume that the 0 status has to do with the SSL certificate.
Although I try to simulate the same requests using Dev-HTTP-Client google chrome plugin and it works fine and I can get proper responses from the https server.
I am not sure but I think I need to use the chrome module Cc["#mozilla.org/security/... to make this work.
If you can guide me with the proper steps to setup the SSL certificate information from inside the addon that would be great
Unfortunately there is no direct way to verify a cert manually and continue with the request. Instead you need to add an override on error yourself and retry.
Attempt to make a regular XMLHttpRequest via nsIXMLHttpRequest. There is enough code and samples around on SO and Google describing how to do it. The requests module won't do, as it hides some necessary details.
Implement nsIBadCertListener2 and stuff it into req.channel.notificationCallbacks (might want to preserve the original callbacks).
If your .notifyCertProblem() get called, that means the cert did not verify. Now it's up to you to verify the cert with your seeded fingerprint (and serial).
If your seeded infos match, add a cert override (that won't work for STS hosts, of course)
Re-spin the request after adding the override, as the first request already got canceled as soon as it hit notifyCertProblem().
Most of the stuff is neatly demonstrated in ErrorPage.jsm of Boot2Gecko (still applies to all other mozilla powered products). That's a cross reference, so click around ;)
Of course, you'll need to use the chrome module.
I should mention that it is deliberate that I'm not giving a complete copy-pasta code solution, only all required pointers, as it is my opinion that a person should be capable enough to work with what I provided, or don't touch security subsystems in the first place.
I am working on a project right now that requires some javascript for the proper functionality we are looking for. The problem we are running into is that the clients of this web-based app can only access the site through a proxy server (due to strict policies). This proxy server is stripping the javascript from the page and we are unable to replicate the exact proxy setting to determine a solution. Has anyone ran into this problem before and found a solution?
If there's a fixed set of client machines, and you can go for Firefox only, you might be able to do something using a Firefox extension or a bookmarklet that fetches the Javascript some other way that the proxy doesn't recognize (e.g. as base64 encoded data). It would however certainly require a load of work, and you may have to program the extension yourself... Probably cheaper to buy a new Proxy.
Some of our customers had these problems and we told them to access our app via HTTPS and that cleared the issue since most proxy won't filter secured traffic.