Some background, the environment I am working in is PHP on the server side and Javascript on the client side. To track a user session, I make a JS PUT request to a server every 5 seconds. Through this I get the amount of time they were on the site, scroll depth, click events, etc.
With tools such as MixPanel or Google Anaylytis, they have the same metrics being measured, yet there isn't constant js calls being made. So my question is how are they tracking the user without javascript?
Your question seems to be... "how does google analytics know how long a user spends on a site without sending constant updates"? Well, short answer is that it doesn't (but it can, read on...!)
In the simplest setup, it will create a session when the first page is opened. Then it will track the time until the second page is opened, then the third. But it doesn't know when the third page was closed, so the total time for the session will only be "time page 3 was opened" minus "time page 1 was opened". And that's what you'll see on the analytics page. This is how most analytics systems work and yes, it's innaccurate. But that's how it is.
HOWEVER! There is a way around this, and it's called an "engagement hit" - essentially just tracking when a user does something on your page, e.g. playing a video or clicking a banner. In fact, you can track all sorts of events with Google Analytics and this will all help towards accurately reporting time-spent. This will involve more JS calls being made than just the typical onload event, but they certainly will not be constant (unless you set up some crazy events).
There's some good information about this on the GA docs site. It might be worth having a look and trying to implement something similar.
Related
I have an html5/javascript application in which multiple users can be viewing the same set of data of any given time. For the sake of a real world example, lets say its a calendar type page.
So user1 is looking has the browser open and looking at the calendar page and user2 is also on the calendar page. User2 makes a change to the calendar and i'd like (as quickly as possible) for those changes the be recognized and refreshed on user1's screen. What is the best way to do this?
I'm thinking about have a mysql table for active users that stores the page they are currently on and a timestamp for its last update, then use ajax calls to ping the server every few seconds and check for an updated timestamp, if its newer than what they have client side, the new data gets sent and the page "reloaded." I am putting reloaded in quotes because the actual browser window will not be refreshed, but a function will be called via javascript that will reload the page. Sort of the way stack overflow performs its update checks, but instead of telling the user the page has changed and providing a button for reload, it should happen automatically. If user1 is working away on the calendar, it seems it might be quite annoying for user2's screen to constantly be refreshing...
Is this a horrible idea? Is pinging the server with an ajax request every few seconds going to cause major slow downs? Is there a better way to do this? I would like the views on either users side to be real time because its important that user1 not be able to update an element on the calendar page that user2 has already changed.
Update: based on some web sockets research it doesnt seem like a proper solution. First its not compatible with older browsers and i support ie8+ and second i dont need real time updstes for all users on the site. The site is an account based applicatiin and an account can have multiple users. The data needs to sync between those users only. Any other recommendations would be great.
You need realtime app for this. You should have a look at socketio. Everytime a user log in, you make him listen for changes on the server. Then when something changed on the server, every users listening are notified.
you can find examples on the official website : http://socket.io/
I am trying to prevent fraud in a webproject I am building.
The project is a game which includes multiple websites.
Each website does a ajax check for with each pageview to a webpage on my server for a status update of the game.
The response page, lets say www.domain.com/response.cfm (it is coldfusion) normally returns nothing, but at a certain point of time within the games timeframe, it will display a JSON string with information.
This information is then used by the script that is included on the websites.
So website A has been viewed 100 times (all of its pages), which will generate 100 ajax calls.
The problem I have is that a robot could check the ajax destination too, and much faster. Now I can detect a robot, or could make it difficult for him by using a session or checking for cookies, BUT...
the biggest issue is that I found out you can do a lot in the Firebug script console, or the Safari console. Probably Chrome too.
With this console, they can even evade the crossdomain restriction. I created a simple script that does a couple of calls to the Ajax page and when I go to the same domain first, and then use the console...there is no crossdomain limitation. And you execute all kind of javascript, so in essence someone like me could commit fraud in the game by using the javascript console which masks him as regular browser user.
My question now is: Does anyone know how to prevent this? I tried to disable the usage of the console but I don't think I can. It may be possible to detect if the console is active and then disable MY scripts so the game doesn't work. But I think they can load the script source in the console manually and then the game does work.
Looks like console is a beautiful thing, but a nightmare for me now to prevent people cheating in the game I am creating.
Hope anyone has suggestions.
ps: of course I am trying to implement som server side checks to detect cheating, but most of the time it is not realtime.
UPDATE 19/3/2012
The fraud that I am trying to prevent is cheating in the game by polling the page that generates logic for the next step of the game. This is a serverscript page which generates json code which will trigger a change on the website the game is played on. For your information, websites the are involved have a script in there header, like google analytics, so they will communicate with my server every pageview.
Polling that serverpage can reveal information which will gain the cheaters knowledge or progress.
So i have to prevent people from getting knowledge ahead of other earnest players by monitoring the serverpage which will reveal information at a certain time. I don't want them auto polling it and when info is revealed, the send themselves a notifcation and check the website.
So what I will do is make sure that if people have to many pageviews per second, they are blocked. Plus you need a cookie to be able to join in and you only get a cookie by logging in. Hopefully this will give me enough tools to make it as robust as possible.
Thanks for all your knowledge, people.
It would be very, very difficult to disable web consoles across the majority of browsers, and anyone who managed to do this would probably be exploiting a browser bug. But read on...
First rule of web programming: You can never trust anything you receive from the web client. Anything that gets sent to your data might have been forged or altered intentionally or unintentionally, and even if you did manage to block a web console, what's to stop me from opening it in a different browser which specifically disallows websites with the console? So that's out. As #DCoder mentions in the comments, there are other methods as well, including browser extensions, which would allow user-defined JavaScript to be executed.
So any checking you do has to be server side. I know you're trying to do some checking already, and it's hard to give advice without having more specifics. That said, one way to do this, as far as I can see right now, is to issue each client an ID and store that in a database somewhere. They can't be sequential IDs, and make sure that they're not trivially forgeable even if someone has a bunch of different IDs (for example, you might want to salt the username, and then hash it). Each time a request is made to the server, only issue a response if the last request was >500 ms ago, and update the database accordingly. Expire the ID after logoff or some time.
The first thing you should think about is securing your server, not the client. It's impossible to hide client code from the client. While it might arguably help prevent a few people who want to cheat from cheating, it's not your primary objective. You have to do this from the server side. This means validating the requests on the server to ensure that they conform to your expectations to some degree.
Game companies will
Require user authentication of some kind so they can identify users
Create some rules about possibilities. For example, the laws of physics should apply, so you know when someone has cheated. Something they can validate as human activity.
Ban people who cheat
If you are not sending data continuously over the network, then you have an issue which is unsolvable unless you are willing to make checks on the server securely and continuously over the course of the game. This will increase server load, but that's the unfortunate cost of preventing cheats.
I'm wondering how Google Analytics Real Time user interface works, what's the technique ? Do they use long-polling from the client to keep the UI statistics instantly up to date by delivering realtime information from the server to the client?
I just open Chrome dev tool on network tab and there is a infinite request on https://www.google.com/analytics/realtime/bind
Does anybody know the trick? It works flawless...
The below refers to how the real time data is collected, not how the UI updates. (It looks like the UI is just using AJAX polling on the client-side, though)
No special polling or client-side technique is used. Data collection is the same as it always has been.
Instead, Google Analytics will assume that someone who's triggered a pageview in the last 5 minutes is still an "active" visitor.
From e-nor:
These visitors have been active in the last 5 minutes, any one not active for over 5 minutes is dropped.
I was one of two people who built the first version of the Realtime Analytics UI. We used Closure's BrowserChannel.
I'm starting to run a few ads on a website, and I'm trying to decide the best way to track performance.
Specifically,
What's the most efficient way to count clicks? About the only way I can think of is to link the ad to another page with the ID of the ad as an argument (e.g. adserver.aspx?id=1234). The other page would then update the database and do a redirect to the advertiser's link. However, it seems inefficient to have to load a separate page for this. Are there any other options?
Also, it seems like I might need to know stuff like how many clicks occurred in a given week. But storing a separate database row with a date for every single click seems excessive. Has anyone else done something like this? Would it make sense to maybe create a new row for each week and increment a counter for all clicks that occurred that week?
Any tips appreciated.
I would suggest that your first solution is the best option, in fact it is the design that most similar systems (OpenX, Google AdSense etc) employ. Additionally it helps you better managed your banners and prevents your site leaking search engine spiders.
As for performance that is just a question of having a good design, in a typical design the redirection script will be fairly lightweight so should process requests fairly quickly. It is worth mentioning that you could thread off the DB updates to reduce the redirect request response times.
There is of course another option:
Rather than homebrew your own banner serving scripts look into implementing OpenX instead, it is free and an extremely good piece of software. OpenX can be found here:
http://openx.org/
and here for the open source version you can run on your own server:
http://www.openx.org/publisher/open-source-ad-server
Another option would be to implement something like Google AdSense and save yourself the hassle of finding advertisers etc. Google also provides tools to allow you to sell banner space and then fallback to default AdSense banners if you have no active advertisers (OpenX will also do this and also support integrating AdSense (and other advertisers))
You can do it with JavaScript.
Opening the page:
Send the ad URL to the client
When the user clicks the ad, use JavaScript to open the URL. Or, simply use an anchor with target="_blank".
Logging the click:
Either way, hook a JavaScript function to the ad click event.
When the user clicks the ad, use AJAX to call a web service. The web service will then log the click.
This way the ad opens as soon as possible, then the browser asynchronously communicates with the server for logging.
The question is, what happens if the user clicks the ad, the page opens, then the web service call fails? If you would prefer, you could call the web service first, then and only if the call succeeds, open the ad. Probably not the best from the end user's perspective.
For you question #2: Storing each click gives you the most reporting flexibility later. If you are set on weekly reports, why not set up a weekly process that generates the reports you need, then cleans up the data? You could even skip cleaning up the data for now until space or speed becomes an issue.
Would it make sense to maybe create a new row for each week and increment a counter for all clicks that occurred that week? Probably not. Only one process can update the counter at a time, otherwise you'll lose data. The locking that has to occur to get the value then update it will probably significantly slow down.
I know a few sites (such as my bank and my school) that kill a session after their has been idle for a set amount of time. It is my understanding that session activity is determined by users following links or at the very least from some kind of active interaction, like updating a form via ajax. Basically the server gets a request to do something during the session and goes ahead and extends the session time another 15 minutes.
But on some occasions I have lost major amounts of time and info while filling out a text box or reading some long set of instructions along the way.
So why not have an ajax script that listens for keyboard activity and mouse movement and lets the server know that the user is still there and active, even if they aren't clicking a submit button or following a link?
I was wondering if anyone knew of respectable sites that already do this, or if I was overlooking some major security hazard with this idea.
The only thing I can imagine would be risky are the random acts of cats, vibrating electronics nearby, or a hyper child.
But in all of the above, the user is most likely at home and -- unless they are trying to get exploited -- have probably minimized the window and thus these things are very unlikely to trigger as an event.
Does anybody see any other major risks?
Typical AJAX sites are making posts back to the server anyway. These events are renewing the users session already.
If you put these events on keyboard or mouse clicks, how many times are you going to be posting to the server? If I am typing in a form field like I am now, that means you could potentially have a ping to the server for each letter I type; not a very efficient solution. On the other end, what if your user is just sitting their reading or using an external text editor to type their text into and will copy it into your form later.
I think the more typical solution to provide a friendly UI so that long posts do not get dropped because of a session expiration is to use an auto-save feature. Google Docs does this. Every few seconds/minutes, they post the contents of the editor back to the server without the user actually clicking save/submit. The other option is to inform the user that their session is about to expire (could be done with a javascript timeout). Provide a link to ping the server to renew the session.
Your solution lends itself to the same problem: you are relying on user behavior. In the first case, navigating between pages and in the second, mouse clicks.
I used to work for a company that did a lot of online contests where users would have to enter essay content as well as shorter blocks of data; we used to modify the session time out "session.setMaxInactiveInterval()" for the user's session when they hit the "long-winded" page so that they would have more time to edit, and then we would set it back to normal after the submit.
Later at that company and a couple others I worked at I proposed a solution similar to what you are describing, but for various reasons it was never accepted. It was never considered a bad idea, just not one we chose. Basically it was going to be an ajax call on a timer so that just before the session timed out, it would fire off a light-weight ajax "ping" and keep the session alive as long as that page was open. I have never had the chance to implement it in the "real world" so perhaps there are negatives that I have not thought of.
Good luck.