I would appreciate your time on answering this question before I commit it to some code.
I've looked through multiple docs on MongoDB and various forums for the answer to this question, however, I would like to know exactly how certain series of operations execute based on my specific circumstances.
Fundamentally, I would like to perform a Read (findOne) operation on a specific document in a collection within its own shard to obtain its current value, modify it, and then store it back into that document (findOneAndUpdate). That sounds easy enough, however, this specific document can potentially get Written upon during the delay between that Read and then Write operation if I understand properly.
For more context, this is a stock market based application. The shard and collection I am referring to here is a user's portfolio which contains information such as the stock ticker, the number of shares owned of that stock, and the total cost they have historically paid for it. The issue here is that when a player sells a share, I have to proportionally adjust the historic total cost by multiplying a ratio of (formerOwnedShares - executedSoldShares)/(formerOwnedShares) where formerOwnedShares is information that needs to be Read, processed, then Updated (i.e. the user currently owns 10 shares and has paid $10 for them over time, they sell 5 shares, now they own 5 shares and have paid $5 to maintain a consistent average cost per share).
I am concerned about the delay between the Read and Update operations. Several other areas of code can operate on this portfolio document from other users (such as other users buying/selling shares with them), so there is a very real possibility that the first Read operation pulls information from an older version of this document that has just recently been updated.
Is there any way to process this sort of situation in one atomic operation in order to maintain perfect concurrency? The issue is clearly the result of having to update a document's value based on that document's most recent value - which I do not think you can do in a single findOneAndUpdate operation.
Thank you very much for your time.
Use a transaction with read concern snapshot, which is an implementation of MVCC, to utilize the same base document version for both operations (find and find-and-modify).
Is there any way to process this sort of situation in one atomic operation in order to maintain perfect concurrency?
This question makes no sense. First, what do you mean by "perfect concurrency". Second, concurrency is not the same as serialization (in fact, they are exactly opposite).
A baseline approach to performing multiple operations that appear sequential in a concucrrent environment is to 1) maintain a "version" field in the document, 2) upon 2nd and subsequent operations, condition the updates on the version not changing, 3) if updates failed retry the entire process until they succeed.
Transactions with snapshot read concern hopefully automate all of this for you.
Related
Let's imagine a DB with two tables: items and orders. The first table contains all items available for sale. The second one keeps track of all orders made by customers. Each item has a column 'quantity', which says how many of these items are available in stock. When making a new order, the backend checks if ordered amount is not greater than amount in stock. If so, the order is not created. Otherwise the order is created and the quantity of available items is updated.
The problem is that when two orders are created simultaneously, both checks are executed at the same time and two orders are created (not knowing about each other). As a result, there are two orders in DB with the total ordered amount larger than the actual quantity in stock.
I've already searched on how to handle this issue and encountered such concepts as transactions, locks, isolation and so on. I got to understand these terms but still didn't get what architectural solution needs to be implemented.
What exactly I need to do to solve this trouble? What SQL query to write for checking the stock before creating an order? Should I wrap it in a transaction and apply some isolation level to it? Or maybe I just need to lock the tables when making an order? Is it possible to just make the order creating operation wait until concurrent order is created? Still have no answers to these questions.
Hope for your help. Thanks!
For low to mid volume you can implement real time processing. For high volume you need to settle for near-real time.
For real time processing there are two options:
Optimistic Locking: The app doesn't lock the records that need to modify but only reads them. When the processing is finishing (ideally after a short period of time) the app updates the records with "concurrency check". Typically the records will feature a version number, a timestamp, or in extreme cases the entire record will be compared. If the update pasess the concurrency check, then all is good, the transaction is committed, and the order is complete. If the concurrency check does not pass, compensating logic needs to take place to retry the action, to recover it somehow, or to consider it failed. The benefit of this strategy is that it can process more orders with the same hardware. The downside is that it's a more complex solution since it needs to take care of extra path, not just the happy path.
Pessimistic Locking: The app reads and locks all the necessary records. All locks needs to be taken in the same sequence by all other competing processes. If all the locks are secured, then the order can be processed safely without the fear of a hiccup at the end of the task. The benefit of this strategy is that is simple to understand and to debug. The downside of this strategy is that it locks are expensive to obtain and can significantly impact the bandwidth of the app -- that is, how many orders per minute it can process.
Finally, for high volume you probably need to settle for near real time -- aka deferred processing. There are two strategies:
If your app can triage the orders by some criteria (region, client, type of products, warehouse, etc.) you can implement a microservice or a set of queues where each instance of the service/queue serves separate clients or stock items. This can provide a decent level of parallelism in this case.
If there can be no triage for the orders, then a single queue can process all the orders, one by one. This can be slow for some apps.
There are many ways to solve the problem. Assuming you are not trying to create a major retail site here, the simplest is to lock the row in items.
The easiest way is probably not even to check until after the subtraction has been done.
UPDATE ITEMS set quantity_avail=quantity_avail - $2
where part_num= $1
returning (quantity_avail)
And then if the returned value is less than 0, do a rollback of the transaction and report to the user that it is now out of stock.
But now what happens if the shipping department drops it on the floor and breaks it while preparing it for shipping?
I'm trying to implement my first domain driven application, after going through Eric Evan's book on Domain-Driven Design. I'm a bit confused on how to go about this.
In my application, a user can purchase a service for getting them certain number of views on a video they post in Youtube, which is fulfilled by the other users of my app who watch those videos(Basically a replica of the many youtube promoter apps already available, for learning).
Say the service is represented in the app as an entity called WatchTime aggregate. The WatchTime entity contains some information like the id of user who purchased this service, the max number of views purchased, number of views already fulfilled, and points earned by someone who views the video once.
I decided to go with 3 bounded contexts, one for authentication, one for handling the watchtimes, like adding or removing them, and one for managing users and their data. Now the user has his personal info and some points that he collected while using the application.
At first I was thinking that all the user data and related actions be in the 3rd context, like adding more points to a user and or reducing his points, but then while making the model, I realized that that if the watch time purchasing service is going to be in the second one, then its going to have to communicate to the third one every time a WatchTime is purchased to tell a service there to reduce points for that purchase. It wouldn't make sense to keep them in two different ones.
So instead what I'm thinking of is have a model of the user in the 2nd bounded context, but with only points and the WatchTimes that this user purchased, so now it doesnt have to call something on the 3rd context.
My question is how to properly seperate things into contexts? Is it like based on the models, or should it be based on the functionality, and all models related to those functionality are going to be in the same context?
And another thing, how to ensure that all the objects of the same entity have the same value and properly persisted in the database? Should only one object representing a particular entity be present at a time, which will be persisted and disposed by the end of a function? Because I was thinking that if two objects representing the same entity be present at the same time, there's a possibility of both having different values or changing to different values.
If i sound like im rambling, please let me know if I have to be more clear. Thanks.
Bounded contexts basically define areas of functionality where the ubiquitous language (and thus the model) are the same. In different bounded contexts, "user" can mean different things: in a "user profile" context, you might have their email address but in the "viewing time" context, you'd just have the points granted and viewership purchased.
Re "another thing", in general you need to keep an aggregate strongly consistent and only allow an update to succeed if the update is cognizant of every prior update which succeeded, including any updates which succeeded after a read from the datastore. This is the single-writer principle.
There are a couple of ways to accomplish this. First, you can use optimistic concurrency control and store a version number with each aggregate. You then update the aggregate in the DB only if the version hasn't changed; otherwise you attempt the operation (performing all the validations etc.) against the new version of the aggregate. This requires some support in the DB for an atomic check of the version and update (e.g. a transaction).
An alternative approach (my personal preference) is to recognize that a DDD aggregate has a high level of mechanical sympathy to the actor model of computation (e.g. both are units of strong consistency). There are implementations of the actor model (e.g. Microsoft Orleans, Akka Cluster Sharding) which allow an aggregate to be represented by at most one actor at a given time (even if there is a cluster of many servers).
Suppose that I have stored procedure that does following:
Selects top 10 records matching a condition. Like say, Select TOP 10 * FROM c WHERE c.complete=false.
It updates the complete flag to true for the 10 documents selected.
Replaces these 10 documents that have the updated flag.
Returns to client these 10 documents.
Suppose, from the client application, I spawn multiple tasks that all run this same stored procedure simultaneously.
Questions:
Is it possible that the two or more simultaneous run of the stored procedure can cause it to return similar documents? Or will they run in complete isolation?
Does Cosmos DB stored procedure lock the data being read?
Results observed:
None of the tasks returned same documents and the return from stored procedure was always a set of distinct documents. But I am not sure whether this behavior will be consistent. I tried running the stored procedure by spawning varying numbers of tasks as high as 20 but could not observe inconsistency.
Questions:
Is it possible that the two or more simultaneous run of the stored procedure can cause it to return similar documents? Or will they run in complete isolation?
Does Cosmos DB stored procedure lock the data being read?
Cosmos DB to guarantee ACID for all operations that are part of a single stored procedure.
In Cosmos DB, JavaScript is hosted in the same memory space as the database. Hence, requests made within stored procedures and triggers execute in the same scope of a database session. This enables Cosmos DB to guarantee ACID for all operations that are part of a single stored procedure/trigger.
You could find the description above from Database program transactions in this official doc.
Update
The consistency level of stored procedure or triggers is determined by the consistency level of the cosmos DB account you set , as stated in the official document.
Cosmos DB offers five consistency levels.
I'm not sure you set which consistency level, probably just default Session consistency level.
Session consistency level only ensure strong consistency in Session life cycle.
So, stored procedures you tested are not running concurrently. There will be some overlap. The subsequent updates will cover the previous updates.
Solutions:
You could try to change your consistency level to Strong. It ensures the strongest consistency, but increases the delay.
Or you could add transaction locks to your business layout to ensure strong consistency.
Hope it helps you.
This does not sound correct according to MSDN:
Data consistency
Stored procedures and triggers are always executed on the primary
replica of the Azure Cosmos DB container. This ensures that reads from
inside stored procedures offer strong consistency. Queries using
user-defined functions can be executed on the primary or any secondary
replica, but you ensure to meet the requested consistency level by
choosing the appropriate replica.
https://learn.microsoft.com/en-us/azure/cosmos-db/programming#database-program-transactions
I was wondering if for a large number of updates (>1000) if saveAll or individual save in a loop should be used. For example, I am making a batch update to multiple objects in a class and both options work. What I am wondering about is whether for large number of updates, if there is limitations on how many objects saveAll can save accurately. I know that there was some limitation when this was running on free Parse.com account years ago, but I would think that limitation should no longer exist since Parse.com doesn't exist anymore.
Any feedback would be greatly appreciated!
There is a 50 objects limitation in one call (updates/saves). It depends a lot on the aftersave/beforesave cloud code. There is no time limit nor request limit. If your deploy is well orchestrated you can push lots of items super fast. For example pm2 with multiple threads in a multicore cpu. You have also have in mind the mongodb hit. Normally it should be a issue saving is not as expensive as finding.
Let's take as an example a js "app", that basically does CRUD, so it creates, updates and deletes (not really) some "records".
In the most basic case, one does not need to resolve conflicts in such an application because the ACID properties of the DBMS are used to eleminate concurrent updates (I'm skimming over a ton of details here, I know). When there's no way to emulate serial execution of updates, one can use timestamps so determine whch update "wins". Even then the client need not worry about timestamps, because they can be generated at request time on the server.
But what if we take it one step further and allow the updates to queue up on the client for some unspecified amount of time (say, to allow the app to work when there's no network connectivity) and then pushed to the server? Then the timestamp can not be generated on the server, since the time when the update was pushed to the server and the actual time when the update was performed may vary greatly.
In the ideal world, where all the clocks are synchronized this is not a problem - just generate a timestamp on the client at the time when the update is performed. But in reality, time often drifts from the "server" time (which is assumed to be perfect, after all, its us configuring the server, what could ever go wrong with it?) or is just plain wrong by hours (possible when you don't set the time zone, but instead update the time / date of the system to match). What would one do to account for reality in such a case?
Perhaps there's some other way of conflict resolution, that may be used in such a case?
Your question has two aspects :
Synchronizing/serializing at server using timestamps via ACID properties of database.
Queues that are with client (delays which server is not aware of).
If you are maintaining queues at client which push to server when it sees fit, then it better have trivial synchronizing. Because it is just defeating the purpose of timestamps, which server relies on.
The scope of ACID is limited here because if clients updates are not realtime, it cannot serialize based on timestamp of request created than or request arrival. It creates a scenario where a request R2 created later than request R1 arrives before R1.
Time is a relative concept, using a local time for client or for server will cause drift for the other. Also it does not scale (inefficient if you have several peer nodes - distributed). And it introduces a single point of failure.
To resolve this vector-clocks were designed. They are logical clocks that increment clock when event occurs on the machine atomically. BASE databases (Basically Available, Soft state, Eventual consistency) use it.
The result is that 1 and 2 will never work out. You should never queue requests which use timestamp for conflict resolution.
Nice challenge. While I appreciated the user568109's anwser, this is how I handled a similar situation in a CQRS/DDD application.
While in a DDD application I had a lot of different commands and queries, in a CRUD application, for each type of "record" we have CREATE, UPDATE and DELETE commands and a READ query.
In my system, on the client I kept track of the previous sync in a tuple containing: UTC Time on the Server, Time on the Client (lets call this LastSync).
READ
Read query won't partecipate to syncronization. Still, in some situation you could have to send to the server a LogRead command to keep track of the informations that were used to take decisions. Such kind of commands did contain entity's type, entity's identifier and LastSync.ServerTime).
CREATE
Create commands are idempotents by definition: they either success or fail (when a record with the same identity already exists). At sync time you will have to either notify the user of the conflict (so that he can handle the situation, eg by changing the identifier) or to fix timestamp as explained later.
UPDATE
Update commands are a bit trickier, since you should probably handle them differently on different type of records. To keep it simple you should be able to impose the users that the last update always wins and design the command to carry only the properties that should be updated (just like a SQL UPDATE statement). Otherwise you'll have to handle automatic/manual merge (but believe me, it's a nightmere: most users won't ever understand it!) Initially my customer required this feature for most of entities, but after a while they accepted that the last update wins to avoid such complexity. Moreover, in case of Update on a Deleted object you should notify the situation to the user and, according to the type of entity updated, apply the update or not.
DELETE
Detele commands should be simple, unless you have to notify the user that an update occurred that could have lead him to keep the record instead of delete it.
You should carefully analyze how to handle each of this command for each of your type of entity (and in the case of UPDATEs you could be forced to handle them differently for different set of properties to update).
SYNC PROCESS
The sync session should start sending to the server a message with
Current Time on the Client
LastSync
This way the server could calculate the offset between its time and the client's time and apply such offset to every command he recieve. Moreover it can check if the offset changed after the LastSync and choose a strategy to handle this change. Note that, this way, the server won't know when the client's clock was adjusted.
At the end of a successful sync (it's up to you to decide what successful means here), the client would update the LastSync tuple.
A final note
This is a quite complex solution. You should carefully ponder with the customer if such complexity give you enough value before starting implementing it.