I am using Casperjs to Login in my Amazon Account and retrieve some data.
But once in a while I get Captchas on the login. So casperjs display to me the captcha and I manually return the solution so it can submit the form.
The problem is that CasperJS gets immediately another captcha, this time it's more difficult. I resolve this too, but another captcha appears... and so on indefinitely...
I don't do anything special, just some casperjs fill and click.
Casperjs loads in the page an external js file with the captcha solution, and then submit.
I am sure that the right captcha is submited.
How can Amazon be so sure to trap me in an infinite loop?
Consider how it looks from their point of view. They can tell a robot is accessing your account based on mouse and keyboard interactions. A human will scan the page and move their mouse randomly while searching for the login buttons. Your script jumps directly to clicking the selector.
When a captcha appears, you fill it in. This does not prove you are a human. This simply proves that your robot can alert you to a captcha for a human to fill in. The rest of the interactions are all done by a robot, and Amazon is fully aware of this. You can answer as many captchas as you like, but the interactions to get this far are still going to be flagged as a robot.
You may want to go down a different route, like having a cookie to start a CasperJS session with your account already logged in. Alternatively, does Amazon provide any sort of API to pull out the value you're interested in?
They're blocking your robot out of geniune love and concern, if that makes you feel any better!
Unfortunately this is not an exact science, so probably there is no such thing as a general, durable solution. Amazon.com uses different techniques to check if you are a robot, including browser fingerprinting, cookie challenges and user behavior profiling (mouse movements and so on).
I would try first to randomize some part of the user agent, only to see if that works. And I would also try a full headless browser like Chromium, using Selenium to allow the script to talk with it.
Can I ask how frequently are you trying to crawl your account? I think it shouldn't be a big deal if you are doing that one a day or so.
Related
I am trying to automate the Registration screen of my Angular app but I am not able to view the Submit button due to Google Captcha. Is there any method to skip the Google Captcha and proceed to click on the Submit button in Protractor? I tried using switching the frames but it didn't work out. Please help
CAPTCHAs (or Completely Automated Public Turing tests to tell Computers and Humans Apart) are designed to be a gate that lets humans through and robots (programs) out.
For reCAPTCHA v2 the story is a little different. You’re still engaging in the same 2-step process as above but you’re sending different data. In this case you need to send the reCAPTCHA sitekey which can be found on the containing , regardless of whether or not the iframe has loaded.
The response you get is a token that needs to be submitted alongside the form and needs to be entered into a hidden text field with the ID of g-recaptcha-response
TL;DR, you can't really bypass CAPTCHAs (that is basically the idea of them)
I have a website where your able to advertise things on my website. The problem is that people are able to do it more than once. Is there a way that people are allowed to visit the website and when they join back they will be redirected to another page saying you have already advertised. People are still able to use vpn's but i have a way to stop that.
How can i use javascript or php to record the users ip first when the visit the website, But if they leave the website or reload the page they will be redirected to another page saying you have already advertised. Is this to much work?
Technically yes, you could use JS and PHP to grab a user's IP address and work with it in a database but proxies and dynamic IPs would make it a very easy check to circumvent. You can also use PHP to create a persistent cookie to identify the user and his/her actions and see if you're getting a returning visitor who posted an ad, but cookies can easily be deleted.
So it's not that what you're trying to do is too much work, it's that it's fairly easily circumvented and not very reliable. Your best bet is an authentication system that requires a valid login to post an ad, logging what the advertisers do, and creating logic which will disallow spammy behavior based on your logs.
You won't be able to stop abuses by very, very determined users but you can make it harder and make them think twice about whether it's worth investing all that time and effort into spamming on your site when there are bound to be much softer targets, giving you the time to deal with the most egregious cases personally instead of trying to stop a torrent of spammy ads.
You cannot stop people doing that 100% for sure.
if you block their IPs they use proxy.
if you use session they change their browsers or reset it to default.
if you block their hardware like in facebook block hard disk serial again they use vpn servers.
if ..
there is no way bro.
Ask for paying instead of making it for free.
Say, a link to a person is sent to a user via email. If the person is already logged into the webpage in his/her browser, clicking on the link takes him/her to the page. However, if he/she is not logged in, he/she should be asked to login in order to access the page. Is there a way to achieve the above functionality using jquery, javascript?
Yes. Build a back-end authentication system, using AJAX and whatever your server-side language is.
From there, develop a hypermedia-style of content-system, and a modular, "widget"-based application delivery model.
Within your hypermedia responses to login (plus passing whatever relevant path information was gained from the e-mail), either redirect the page to a new page (based on the linked response from the server), or download the widgets requested from the server (for whatever application you're displaying media in), and then stream in AJAX content (again, from a URL dictated by the server-response).
This is about as close as you're going to get to security, in terms of delivering things to the client, in real-time, with authentication.
If you were to load the reports/gallery/game/whatever, and put a div over it, and ask for users to log in, then smart users can just kill the div.
If you include the content, or include the application components (JS files), or even include the links to the JS files which will request and display the content, then clever people are again going to disassemble that, in 20 seconds, flat.
The only way I can see to do this is to have a common request-point, to touch the server, and conditionally load your application, based on "next-steps" URLs, passed to the client, based on successful authorization and/or successfully completing whatever the previous step was, plus doing authentication of some form on each request (REST-based tokens+nonces, or otherwise)...
This would keep the content (and any application-structure which might have vulnerabilities) from the client, until you can guarantee that the client has been properly authorized, and the entire application is running inside of multiple enclosed/sandboxed modules, with no direct access to one another, and only instance-based access to a shared-library.
Is it worth the work?
Who knows.
Are we talking about a NORAD nuclear-launch iPhone app, which must run in JavaScript?
Then no, engineering this whole thing for the next six months isn't overboard.
And again, all of this security falls over as soon as one person leaves themselves logged-in, and leaves their phone on the table (biometric authentication as well, then?).
Are we talking about a gallery or discount-offers that you want to prevent people to log into, so you know that only the invited people are using them?
Well, then an 18-month project to engineer, develop, debug and deploy a system like this is probably going to be overkill.
In this case, perhaps you can just do your best to prevent the average person from stealing your content or using your cut-prices, and accept that people who take the time to dig into and reverse-engineer everything are going to find a way to get what they want, 95 times out of 100.
In that case, perhaps just putting a login div overtop of the page IS what you're going to be looking for...
If you're dealing with, say a company back-end, or with company fiscals or end-user, private-data, or anything of the sort, then aside from meeting legal requirements for collection/display/storage, how much extra work you put into the security of the system depends on how much your company's willing to pay to do it.
If it makes you feel better, there are companies out there that pay $60,000-$150,000 a year, to use JS tracking/testing programs from Adobe. Those programs sit right there, on the webpage, most of the time, for anybody to see, as long as you know where to look.
So this isn't exactly an unknown problem.
Yes it is. By authenticating (login) you can store a "loggedIn" cookie which you have to delete by session end (logout or closing the browser). You can use that cookie to check if somebody is logged in or not. If not logged in, than you can display the login page and send the login request with ajax. Btw it is not a good practice to use hybrid applications like that. It is better to use SPA-s with REST service, or implement this on server side.
I am trying to prevent fraud in a webproject I am building.
The project is a game which includes multiple websites.
Each website does a ajax check for with each pageview to a webpage on my server for a status update of the game.
The response page, lets say www.domain.com/response.cfm (it is coldfusion) normally returns nothing, but at a certain point of time within the games timeframe, it will display a JSON string with information.
This information is then used by the script that is included on the websites.
So website A has been viewed 100 times (all of its pages), which will generate 100 ajax calls.
The problem I have is that a robot could check the ajax destination too, and much faster. Now I can detect a robot, or could make it difficult for him by using a session or checking for cookies, BUT...
the biggest issue is that I found out you can do a lot in the Firebug script console, or the Safari console. Probably Chrome too.
With this console, they can even evade the crossdomain restriction. I created a simple script that does a couple of calls to the Ajax page and when I go to the same domain first, and then use the console...there is no crossdomain limitation. And you execute all kind of javascript, so in essence someone like me could commit fraud in the game by using the javascript console which masks him as regular browser user.
My question now is: Does anyone know how to prevent this? I tried to disable the usage of the console but I don't think I can. It may be possible to detect if the console is active and then disable MY scripts so the game doesn't work. But I think they can load the script source in the console manually and then the game does work.
Looks like console is a beautiful thing, but a nightmare for me now to prevent people cheating in the game I am creating.
Hope anyone has suggestions.
ps: of course I am trying to implement som server side checks to detect cheating, but most of the time it is not realtime.
UPDATE 19/3/2012
The fraud that I am trying to prevent is cheating in the game by polling the page that generates logic for the next step of the game. This is a serverscript page which generates json code which will trigger a change on the website the game is played on. For your information, websites the are involved have a script in there header, like google analytics, so they will communicate with my server every pageview.
Polling that serverpage can reveal information which will gain the cheaters knowledge or progress.
So i have to prevent people from getting knowledge ahead of other earnest players by monitoring the serverpage which will reveal information at a certain time. I don't want them auto polling it and when info is revealed, the send themselves a notifcation and check the website.
So what I will do is make sure that if people have to many pageviews per second, they are blocked. Plus you need a cookie to be able to join in and you only get a cookie by logging in. Hopefully this will give me enough tools to make it as robust as possible.
Thanks for all your knowledge, people.
It would be very, very difficult to disable web consoles across the majority of browsers, and anyone who managed to do this would probably be exploiting a browser bug. But read on...
First rule of web programming: You can never trust anything you receive from the web client. Anything that gets sent to your data might have been forged or altered intentionally or unintentionally, and even if you did manage to block a web console, what's to stop me from opening it in a different browser which specifically disallows websites with the console? So that's out. As #DCoder mentions in the comments, there are other methods as well, including browser extensions, which would allow user-defined JavaScript to be executed.
So any checking you do has to be server side. I know you're trying to do some checking already, and it's hard to give advice without having more specifics. That said, one way to do this, as far as I can see right now, is to issue each client an ID and store that in a database somewhere. They can't be sequential IDs, and make sure that they're not trivially forgeable even if someone has a bunch of different IDs (for example, you might want to salt the username, and then hash it). Each time a request is made to the server, only issue a response if the last request was >500 ms ago, and update the database accordingly. Expire the ID after logoff or some time.
The first thing you should think about is securing your server, not the client. It's impossible to hide client code from the client. While it might arguably help prevent a few people who want to cheat from cheating, it's not your primary objective. You have to do this from the server side. This means validating the requests on the server to ensure that they conform to your expectations to some degree.
Game companies will
Require user authentication of some kind so they can identify users
Create some rules about possibilities. For example, the laws of physics should apply, so you know when someone has cheated. Something they can validate as human activity.
Ban people who cheat
If you are not sending data continuously over the network, then you have an issue which is unsolvable unless you are willing to make checks on the server securely and continuously over the course of the game. This will increase server load, but that's the unfortunate cost of preventing cheats.
I know a few sites (such as my bank and my school) that kill a session after their has been idle for a set amount of time. It is my understanding that session activity is determined by users following links or at the very least from some kind of active interaction, like updating a form via ajax. Basically the server gets a request to do something during the session and goes ahead and extends the session time another 15 minutes.
But on some occasions I have lost major amounts of time and info while filling out a text box or reading some long set of instructions along the way.
So why not have an ajax script that listens for keyboard activity and mouse movement and lets the server know that the user is still there and active, even if they aren't clicking a submit button or following a link?
I was wondering if anyone knew of respectable sites that already do this, or if I was overlooking some major security hazard with this idea.
The only thing I can imagine would be risky are the random acts of cats, vibrating electronics nearby, or a hyper child.
But in all of the above, the user is most likely at home and -- unless they are trying to get exploited -- have probably minimized the window and thus these things are very unlikely to trigger as an event.
Does anybody see any other major risks?
Typical AJAX sites are making posts back to the server anyway. These events are renewing the users session already.
If you put these events on keyboard or mouse clicks, how many times are you going to be posting to the server? If I am typing in a form field like I am now, that means you could potentially have a ping to the server for each letter I type; not a very efficient solution. On the other end, what if your user is just sitting their reading or using an external text editor to type their text into and will copy it into your form later.
I think the more typical solution to provide a friendly UI so that long posts do not get dropped because of a session expiration is to use an auto-save feature. Google Docs does this. Every few seconds/minutes, they post the contents of the editor back to the server without the user actually clicking save/submit. The other option is to inform the user that their session is about to expire (could be done with a javascript timeout). Provide a link to ping the server to renew the session.
Your solution lends itself to the same problem: you are relying on user behavior. In the first case, navigating between pages and in the second, mouse clicks.
I used to work for a company that did a lot of online contests where users would have to enter essay content as well as shorter blocks of data; we used to modify the session time out "session.setMaxInactiveInterval()" for the user's session when they hit the "long-winded" page so that they would have more time to edit, and then we would set it back to normal after the submit.
Later at that company and a couple others I worked at I proposed a solution similar to what you are describing, but for various reasons it was never accepted. It was never considered a bad idea, just not one we chose. Basically it was going to be an ajax call on a timer so that just before the session timed out, it would fire off a light-weight ajax "ping" and keep the session alive as long as that page was open. I have never had the chance to implement it in the "real world" so perhaps there are negatives that I have not thought of.
Good luck.