Develop Reference

How to check the quantity in stock when making concurrent orders?

Let's imagine a DB with two tables: items and orders. The first table contains all items available for sale. The second one keeps track of all orders made by customers. Each item has a column 'quantity', which says how many of these items are available in stock. When making a new order, the backend checks if ordered amount is not greater than amount in stock. If so, the order is not created. Otherwise the order is created and the quantity of available items is updated.
The problem is that when two orders are created simultaneously, both checks are executed at the same time and two orders are created (not knowing about each other). As a result, there are two orders in DB with the total ordered amount larger than the actual quantity in stock.
I've already searched on how to handle this issue and encountered such concepts as transactions, locks, isolation and so on. I got to understand these terms but still didn't get what architectural solution needs to be implemented.
What exactly I need to do to solve this trouble? What SQL query to write for checking the stock before creating an order? Should I wrap it in a transaction and apply some isolation level to it? Or maybe I just need to lock the tables when making an order? Is it possible to just make the order creating operation wait until concurrent order is created? Still have no answers to these questions.
Hope for your help. Thanks!

For low to mid volume you can implement real time processing. For high volume you need to settle for near-real time.
For real time processing there are two options:
Optimistic Locking: The app doesn't lock the records that need to modify but only reads them. When the processing is finishing (ideally after a short period of time) the app updates the records with "concurrency check". Typically the records will feature a version number, a timestamp, or in extreme cases the entire record will be compared. If the update pasess the concurrency check, then all is good, the transaction is committed, and the order is complete. If the concurrency check does not pass, compensating logic needs to take place to retry the action, to recover it somehow, or to consider it failed. The benefit of this strategy is that it can process more orders with the same hardware. The downside is that it's a more complex solution since it needs to take care of extra path, not just the happy path.
Pessimistic Locking: The app reads and locks all the necessary records. All locks needs to be taken in the same sequence by all other competing processes. If all the locks are secured, then the order can be processed safely without the fear of a hiccup at the end of the task. The benefit of this strategy is that is simple to understand and to debug. The downside of this strategy is that it locks are expensive to obtain and can significantly impact the bandwidth of the app -- that is, how many orders per minute it can process.
Finally, for high volume you probably need to settle for near real time -- aka deferred processing. There are two strategies:
If your app can triage the orders by some criteria (region, client, type of products, warehouse, etc.) you can implement a microservice or a set of queues where each instance of the service/queue serves separate clients or stock items. This can provide a decent level of parallelism in this case.
If there can be no triage for the orders, then a single queue can process all the orders, one by one. This can be slow for some apps.

There are many ways to solve the problem. Assuming you are not trying to create a major retail site here, the simplest is to lock the row in items.
The easiest way is probably not even to check until after the subtraction has been done.
UPDATE ITEMS set quantity_avail=quantity_avail - $2
where part_num= $1
returning (quantity_avail)
And then if the returned value is less than 0, do a rollback of the transaction and report to the user that it is now out of stock.
But now what happens if the shipping department drops it on the floor and breaks it while preparing it for shipping?

Secure High score ladder [duplicate]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
[I know there have been similar questions about preventing cheating on high score lists, but no answer didn't really help me for JavaScript based games, so please try to think about my question, before telling me about similar posts. I ask about best practices because the JavaScript is always visible for the user and therefore it is not possible to prevent cheating completly, I just want to make it harder.]
I'm developing a JavaScript based game that works in the browser. I want to make a high score list that contains the user name and the score of all users. To achieve that the browser sends the username and the score to my server (via AJAX).
Submitting fake scores to this list would be fairly easy: One could take a look at the AJAX requests and then make an own AJAX request with a faked score. Using something like a token that has to be send with the other data is pointless, as it will be easy to discover.
My only approach, that would prevent cheating, would be to send a description of every user action to the server and calculate the score there. But this is not really practicable as it would be too much for the server.
I accepted an answer, but in case anyone has other ideas about how to make cheating harder, please create another answer!

I like to play cheat the cheater - something like using a token to authenticate the score that changes every time the update is called... but I accept the cheat score that gets posted using a duplicate token. Then I display that cheat score to only the cheater, so it appears that it worked, but now the cheater is seeing his results in a sandbox.

You pretty much answered your own question. If you want to really make it harder for users to cheat, send game log to the server, where you'll calculate the score.
You don't have to send all the events, just ones that affect result score.
There are some techniques, though, that may help you:
include signature in your request. Something like MD5(secret_key + params). Although, "secret key" will have to be in JS source, it will effectively protect you from simple request interception (see Tamper Data and Charles)
if it's a multiplayer game, accept scores calculated by clients and compare them. Cheaters will be pretty visible (assuming that the majority of users are honest).
you can set a score cap, an "unreachable" result. Everyone who posts score higher than this is a cheater. For example, speed typing game: no one can type correct text at 1500 chars/minute, even 700 is pretty damn hard (though achievable).

On score submit:
Request some token from the server, this should be time based and only valid for about 2 seconds
Only accept submits that include a valid hash of this token, some salt and the score.
This prevents manual tampering with the request as it would timeout the score. If you want to account for high-latency give it a little more time until the timeout.
The hashing function:
Scramble the hashing function inside packed code (http://dean.edwards.name/packer/ really produces nasty to read code) if you use jQuery or some other library just drop the hashing functionality inside the library file and it gets pretty bad to find, escpecially if you use a function name like "h" :)
Handling the score-variable itself:
Well everybody with a debugging console can change the variable on runtime when doing this but if you encapsulate your whole Javascript inside a function and call it nothing is in the global namespace and it's much harder to get to the variables:
(function() {
//your js code here
})();

I have had lots of thoughts about it and, eventually, decided to only have local individual highscores, so cheating is not really beneficial for player and not harmful to others. Yet my game is just a simple minesweeper, and there were people who complained about the lack of competitive table.
Option 2, is approach taken by WebSudoku - to show your place "among the people of internet". So you will not see any other results, and people wont see your results - but you can compare yourself to crowd.
p.s: And seriously - any kid with Firebug/WebInspector can easily hack your JS game and, eventually, to be able to reach very high score.

If you are relying on the client to send the final score to the server, then there is no way (afaik) to prevent a genius from cheating. But I think you might be able to prevent stupid people (and honest people) from cheating, so that only geniuses and their friends will dominate your leaderboards.
There are two ways I can think of
1.) "security through obscurity."
Come up with an algorithm that transforms simple scores into something else (and to transform them back). Then obfuscate it. Complicate it. Write a function that multiplies it by q and divides it by ralph. Apply a bunch of functions to it, and among the 5-15 functions that do random stuff to it, include one that multiplies the number by 19 ( a prime number ). On your server, check to make sure every incoming number (or letter) is divisible by 19, and decode
You have to write a bunch of complex code that transforms simple scores into something crazy-looking. You have to write a series of functions in the least-efficient, most spaghetti-code fashion possible. Use
One thing you cold do is to have a set of disallowed values. I.e., perhaps all points awarded are even. If anyone tries to submit an odd number, they are obviously cheating (and very stupid).
2.) time
You should be able to know when the user started the game. You should have a session started and record when they requested the page. Then you should also be able to tell when they submitted their score. And you should also know what the time series is for max points. I.e. can you get 5 points per minute, 100 per minute, minute^3, etc... If user submits more points than are possible during that time, they are cheating.
You could also strike a balance between server and client processing and make the client should send progress update every x minutes by ajax. And if it fails to report, you assume it's been compromised (much like in Bond movies, when he's infiltrating the enemy's lair and he snaps some guard's neck. When the guard doesn't respond to his next 10-minutely check-in, the alarms will go off).
If you've ever played Zynga Poker, you've probably seen what happens when someone at the table has a slow internet connection.

Depending on the nature of the game, you could use other players to verify the results. In simple games this works great, on others you have to be clever and develop many aspects around this feature. E.g. sometimes is possible to replay and verify results based on logged actions. This trick works specially well for Human versus AI, as long as the model is deterministic.
Another option is redefining the score concept to be more user-centric, this is pretty easy to implement, but tends to be hard to devise, and only applies to a few categories of games.
Purely speculative approaches are also possible, it's sometimes pretty easy to know when some parameters don't fit. It would not avoid cheating, but would moderate it a lot.
The most complicated part is getting a small enough replay log, but since most data isn't random (except for player actions, which, actually aren't that random because depend on the game) it's essentially a matter of getting the design right.
Also, if gameplay is extended enough, for action games and the like you can get a lot of compression from doing some approximation, merging (e.g. motion vectors), and clipping uninteresting stuff.

Ideally you would send your entire event log to the server for checking. Perhaps you can implement a heuristic so you can easily determine if the score is within a set of bounds. For instance, if the total game time is 5 seconds you might expect a much lower score than with a much longer game time.
Alternatively, you could choose to manually check the event log for really high scores (the overall top-X, which should be fairly stable).
You will need a seeded random number generator if you're doing anything with randomness (like random events). Which might be tricky if you hadn't already thought of it.
You can find many more resources but it really just boils down to server-side checking. JavaScript is not unique in this, but likely easiest to exploit because you not only see the client-server communication but also the client-side source code!
HTML5 Multiplayer Game Security Solutions
http://lanyrd.com/2011/jsconf/sfggb/
Games like Starcraft only record the mouse clicks and key presses. The actual commands are then simulated. I expect 'Worms Armageddon' to do something similar but their random events (like the bounciness of bananas) aren't seeded properly so in the instant replay you might get a different result.
You could imagine something similar for MMORPGs. The server calculates your position based on the keypresses, the client merely tries to give a good early interpretation but you may warp around when you're lagging because the server will place you elsewhere on the map because it didn't get the keypress events timely.
If you attack something, the server will check if you're close enough and how much damage you can expect to deal with current stats and equipment.

Record key points in game, then score is submitted with these key points. When people look high scores, they can also see overview of played game, if it looks like it is impossible to play like that without cheating, then people can report these suspicious scores to admins.

I used a system using a time based request having 3 parameters
req number, curr time, score
The req number is returned from server in the response to the update score request , each time this is a new random value.
The curr time is calculated not from computer clock but from start of game and is synced with server using an ajax request.
Update score request is sent after short intervals (around 30 sec max).
Following checks are applied on the server
Time is within 10 seconds range from the server clock.
there has been not more than 40 seconds since the req number was sent.
the score change sent after 30 seconds is possible (within 2 x humanly possible range)
Score is updated only if the above checks are passed or the user gets a disconnection message :(
This is simpler than most methods and works out to eliminate all casual hackers (well, unless they read this and want to go to the trouble of updating score quickly or making a script of their own).

If not cheating is more important than the game itself, try to construct and present your game in a way that it looks like finding the solution to a math problem. So the server will give an instance of the problem to the client (example A: a chess board about to be won in 3 moves, example B: a geometry dash randomly generated level) and the user will have to solve it and post back a solution (example A: the winning moves, example b: the exact timestamps and intensity of jumps to avoid obstacles)
With this approach, it is key that the server doesn't send the same level twice, or else the cheater can plan and "design" his solution in advance. Also, the game information must be randomly generated in the server and not sent via seed, or else the cheater can fake the seed and design his solution with time.
The given time for valid submissions must be also tracked in the server so that they will only have "playing" time and no "designing" time. If the cheater is good enough to design a solution as fast as honest players can win the game, then they are talented enough to win the game honestly and deserve their points.
Back in the server, you will need to check that the submitted solution is valid for that instance.
Of course this approach requires lots of extra work: More instances of games (ideally infinite and non repeating), server side generation, server side validation of submissions, time caps, etc.
Note: I know these approach was already suggested in multiple solutions some years ago, I wanted to add my humble contribution.

Timestamp-based conflict resolution without reliable time synchronization

Let's take as an example a js "app", that basically does CRUD, so it creates, updates and deletes (not really) some "records".
In the most basic case, one does not need to resolve conflicts in such an application because the ACID properties of the DBMS are used to eleminate concurrent updates (I'm skimming over a ton of details here, I know). When there's no way to emulate serial execution of updates, one can use timestamps so determine whch update "wins". Even then the client need not worry about timestamps, because they can be generated at request time on the server.
But what if we take it one step further and allow the updates to queue up on the client for some unspecified amount of time (say, to allow the app to work when there's no network connectivity) and then pushed to the server? Then the timestamp can not be generated on the server, since the time when the update was pushed to the server and the actual time when the update was performed may vary greatly.
In the ideal world, where all the clocks are synchronized this is not a problem - just generate a timestamp on the client at the time when the update is performed. But in reality, time often drifts from the "server" time (which is assumed to be perfect, after all, its us configuring the server, what could ever go wrong with it?) or is just plain wrong by hours (possible when you don't set the time zone, but instead update the time / date of the system to match). What would one do to account for reality in such a case?
Perhaps there's some other way of conflict resolution, that may be used in such a case?

Your question has two aspects :
Synchronizing/serializing at server using timestamps via ACID properties of database.
Queues that are with client (delays which server is not aware of).
If you are maintaining queues at client which push to server when it sees fit, then it better have trivial synchronizing. Because it is just defeating the purpose of timestamps, which server relies on.
The scope of ACID is limited here because if clients updates are not realtime, it cannot serialize based on timestamp of request created than or request arrival. It creates a scenario where a request R2 created later than request R1 arrives before R1.
Time is a relative concept, using a local time for client or for server will cause drift for the other. Also it does not scale (inefficient if you have several peer nodes - distributed). And it introduces a single point of failure.
To resolve this vector-clocks were designed. They are logical clocks that increment clock when event occurs on the machine atomically. BASE databases (Basically Available, Soft state, Eventual consistency) use it.
The result is that 1 and 2 will never work out. You should never queue requests which use timestamp for conflict resolution.

Nice challenge. While I appreciated the user568109's anwser, this is how I handled a similar situation in a CQRS/DDD application.
While in a DDD application I had a lot of different commands and queries, in a CRUD application, for each type of "record" we have CREATE, UPDATE and DELETE commands and a READ query.
In my system, on the client I kept track of the previous sync in a tuple containing: UTC Time on the Server, Time on the Client (lets call this LastSync).
READ
Read query won't partecipate to syncronization. Still, in some situation you could have to send to the server a LogRead command to keep track of the informations that were used to take decisions. Such kind of commands did contain entity's type, entity's identifier and LastSync.ServerTime).
CREATE
Create commands are idempotents by definition: they either success or fail (when a record with the same identity already exists). At sync time you will have to either notify the user of the conflict (so that he can handle the situation, eg by changing the identifier) or to fix timestamp as explained later.
UPDATE
Update commands are a bit trickier, since you should probably handle them differently on different type of records. To keep it simple you should be able to impose the users that the last update always wins and design the command to carry only the properties that should be updated (just like a SQL UPDATE statement). Otherwise you'll have to handle automatic/manual merge (but believe me, it's a nightmere: most users won't ever understand it!) Initially my customer required this feature for most of entities, but after a while they accepted that the last update wins to avoid such complexity. Moreover, in case of Update on a Deleted object you should notify the situation to the user and, according to the type of entity updated, apply the update or not.
DELETE
Detele commands should be simple, unless you have to notify the user that an update occurred that could have lead him to keep the record instead of delete it.
You should carefully analyze how to handle each of this command for each of your type of entity (and in the case of UPDATEs you could be forced to handle them differently for different set of properties to update).
SYNC PROCESS
The sync session should start sending to the server a message with
Current Time on the Client
LastSync
This way the server could calculate the offset between its time and the client's time and apply such offset to every command he recieve. Moreover it can check if the offset changed after the LastSync and choose a strategy to handle this change. Note that, this way, the server won't know when the client's clock was adjusted.
At the end of a successful sync (it's up to you to decide what successful means here), the client would update the LastSync tuple.
A final note
This is a quite complex solution. You should carefully ponder with the customer if such complexity give you enough value before starting implementing it.

Is there a harder to break way to obfuscate a javascript function than hieroglyphy?

I'm developing an online chess variant game. I want to create a javascript function that has the purpose of communicating to the server on behalf of the player the exact time spent on a move.
This message will be encrypted, of course, but in order to trust this function, I want to obfuscate it to the point that I can rely on the obfuscation algorithm.
I only know a few obfuscation algorithms, hieroglyphy being the most interesting. But it isn't unbreakable. Speed of execution and size are not critical, I can deduct the time spent by the function that sends the message in that same function, and the size can be even up to 2MB.
I'm pretty sure that there is no unbreakable algorithm because as long as it is required to run in a browser, anyone with enough patience can take it piece by piece and see what it does.
Do I have an alternative that would require more effort and time from a user with bad intentions?
Edit I've done some tests in every browser on WindowsXP and it appears that in FF, IE, Opera and Chrome the setTimeout function will trigger after a delay that is passed as the second parameter, regardless of any changes to system time during the delay. If no other information is presented to suggest otherwise, the logical conclusion would be that time can be measured client-side regardless of system time changes, using the setTimeout function but not the Date() object, up to a precision given by the setTimeout delay time.
Hamish mentioned in an answer below that modifying the browser date/time APIs is trivial. In that case, the javascript code is vulnerable to a modification that will increase the setTimeout real delay time. Some code should be set in place so that the server should start suspecting of cheating someone who has unreasonable lag time. This will always be a problem if lag time isn't included in thinking time.
There's a reason I can't use server side timing. The lag times would sometimes exceed a reasonable amount and that will leave users dissatisfied. And sometimes the lag can make all the difference.
Which brings me back to the original question. I'm looking for the best obfuscation method, where best is measured in the effort an attacker has to make to deobfuscate. Ideally, I would want to change the obfuscation algorithm faster than an attacker can deobfuscate, and then never to use that algorithm again or use it rarely, at a time the attacker won't expect.

I could set my computer's clock to three hours ago and your script would happily send -10800 seconds. NEVER rely on JavaScript to handle information in a trusted manner. Use your server-side code to time the difference between when the player's turn started and when they made their move, and absolutely keep a representation of the game on the server and make sure the move is valid.

Obfuscating your code doesn't help, for two reasons:
Users can still inspect the messages being sent from the browser to the server. You would also have to sign the message somehow, to prevent it being intercepted and modified. Generally, it will be even easier to unpack the message than the function used to generate it.
You're trying to measure the time taken on the move, which means your obfuscated function still has to trust the system clock and the browser date/time APIs. Both are trivial to modify.
A sensible solution would be to measure the time messages are sent and received on your server, and measure the latency of the connection to correct for transmission speeds (if you need to be very accurate).

Maximum size of ajax returned data

Fellow coders, I have asked this question before but did not get a conclusive answer to it. The question is: how much data can i safely return from and ajax post call before i run into some limitation somewhere?
The scenarios are basically like this: front-end makes an ajax call to a php controller/model. the controller returns a bunch or rows from the database or returns some html representing some report which will be stored in a js string var to be displayed later.
I see two limitations here: the size of the data returned through the ajax call and max size the js var can hold.
Anyone knows what the limits are?
thanks

See this answer: Javascript maximum size for types?
In short, unless the browser specified otherwise, variable sizes are not subject to a restriction. As for Ajax: There's no limit, unless defined server-side (such as this one).

I don't think either factor you listed would be an issue. What I would look at are:
The amount of time the user is willing to wait for the response. Also, your server-side programming language or web server might impose a limit on the length of any one request.
The amount of RAM the client has. Even if there is no variable size limit, eventually the computer will run out of space.
In these situations, you are almost always better off delivering smaller chunks of the data at a time and allowing the user to load what data they need (either by granulation [showing summaries and letting them drill down], or pagination / search). No one wants to wait 10 minutes for the site to load, and HTTP doesn't really handle large requests all that well.