I am building something like an ActiveRecord class for documents stored in MongoDB (akin to Mongoose). I have two goals:
Intercept all property setters on a document using a Proxy, and automatically create an update query to be sent to Mongo. I've already found a solution for this problem on SO.
Prevent unnecessary reads from the database. I.e. if a function is performed on a document, and that function only ever sets properties, and doesn't ever use an existing property of the document, then I don't need to read the document from the database, I can directly update it. However, if the function uses any of the document's properties, I'd have to read it from the database first, and only then continue on with the code. Example:
// Don't load the document yet, wait for a property 'read'.
const order = new Order({ id: '123abc' });
// Set property.
order.destination = 'USA';
// No property 'read', Order class can just directly send a update query to Mongo ({ $set: { destination: 'USA' } }).
await order.save();
// Don't load the document yet, wait for a property 'read'.
const order = new Order({ id: '123abc' });
// Read 'weight' from the order object/document and then set 'shipmentCost'.
// Now that a 'get' operation is performed, Proxy needs to step in and load the document '123abc' from Mongo.
// 'weight' will be read from the newly-loaded document.
order.shipmentCost = order.weight * 4.5;
await order.save();
How would I go about this? It seems pretty trivial: set a 'get' trap on the document object. If it's the first-ever property 'get', load the document from Mongo and cache it. But how do I fit an async operation into a getter?
arithmetic cannot be async
You can probably initiate an async read from within a getter (I haven't tried it, but it seems legit), but the getter can't wait for the result. So, unless your DB library provides some blocking access calls, this line, where order.weight is fetched just in time and the value used in multiplication, will always be pure fantasy in any lazy-read regime:
order.shipmentCost = order.weight * 4.5
(If your DB library does have blocking reads, I think it will be straightforward to build what you want by using only blocking reads. Try it. I think this is part of what Sequelize's dataLoader does.)
There's no way for multiplication to operate on Promises. There's no way to await an async value that isn't itself async. Even Events, which are not strictly async/await, would require some async facade or a callback pattern, neither of which are blocking, and so neither of which could make that statement work.
This could work, but it forces every caller to manage lazy-loading:
order.shipmentCost = (await order.weight) * 4.5
That approach will deform your whole ecosystem. It would be much better for callers to simply invoke read & save when needed.
Or you might be able to create a generator that works inside getters, but you'd still need to explicitly "prime the pump" for every property's first access, which would make the "fantasy" statement work at the cost of spawning a horrific pre-statement that awaits instead. Again, better to just use read and save.
I think what you're hoping for is impossible within javascript, because blocking and non-blocking behavior is not transparent and cannot be made to be. Any async mechanism will ultimately manifest as not-a-scalar.
You would need to create your own precompiler, like JSX, that could transform fantasy code into async/aware muck.
Serious advice: use an off-the-shelf persistence library instead of growing your own.
The problem space of data persistence is filled with many very hard problems and edge-cases. You'll have to solve more of them than you think.
Unless your entire project is "build better persistence tech," you're not going to build something better than what is out there, which means building your own is just the slowest way to get an inferior solution.
More code you write is more bugs to fix. (And you're writing tests for this magic persistence library, right?)
If you're trying to build a real app and you just need to interface with Mongo, spend 15 minutes shopping on npm and move on. Life is too short. Nobody will care how "cool" is your hand-rolled database layer that's almost like ActiveRecord (except for some opinionated customizations and bugs and missing features -- all of which will act as a barrier to others and even yourself).
Related
This is a question about performance more than anything else.
Node exposes three different types of methods to accomplish various filesystem tasks:
Promises API (async)
Callback API (async)
Synchronous API (sync)
I've read more articles and stackoverflow answers than I can count, all of which claiming to never need the sync methods.
I recently wrote a script which required a couple directories to be made if they didn't already exist. During this, I noticed that if I used the async/await methods (primarily fs.promises.mkdir and fs.promises.access), the event loop would simply continue to the next async bit of code, regardless of the fact that the next bits require those directories. This is expected behavior, after all, it's async.
I understand this could be solved with a nice little callback hell sesh, but that isn't the question, whereas the idea that the promises api can be used over all other methods is.
The question then becomes:
Is it ever better to use Node's filesystem sync methods over the same async methods?
Is it ever truly required in situations like this to block the process?
Or said differently:
Is it possible to completely avoid sync methods and ONLY use the promises api (NOT promises + callbacks)?
It seems like using the sync methods (given my situation above, where the directories are required to be there before any other call is made) can be EXTREMELY useful to write readable, clear code, even though it may negatively impact performance.
With that being said, there's an overwhelming level of information to say that the sync api is completely useless and never required.
Again, this purely caters to the promises api. Yes, callbacks and promises are both async, but the difference between the job and message queues makes the both api's completely different in this context.
PS: For additonal context on examples, I've provided a code sample so you don't have to imagine my example ;)
Thanks! :)
// Checks if dir exists, if not, creates it. (not the actual code, just an example)
// Sync version
if (!fs.existsSync(dirPath)) {
fs.mkdirSync(dirPath);
}
// Async version
try {
await fs.promises.access(dirPath);
} catch {
await fs.promises.mkdir(dirPath);
}
It depends on the situation. The main benefit of the sync methods is that they allow for easier consumption of their results, and the main disadvantage is that they prevent all other code from executing while working.
If you find yourself in a situation where other code not being able to respond to events is not an issue, you might consider it to be reasonable to use the sync methods - if the code in question has no chance of or reason for running in parallel with anything else.
For example, you would definitely not want to use the sync methods inside, say, a server handling a request.
If your code requires reading some configuration files (or creating some folders) when the script first runs, and there aren't enough of them such that parallelism would be a benefit, you can consider using the sync methods.
That said, even if your current implementation doesn't require parallelism, something to keep in mind is that, if the situation changes and you find that you do actually need to allow for parallel processing, you won't have to make any changes to your existing code if you had started out by using the promise-based methods in the first place - and if you understand the language, using the Promises properly should be pretty easy, so if there's a chance of that, you might consider using the Promises anyway.
I was checking Sequelize's examples and documentation the other day, and I came across:
Albums.update(myAlbumDataObject,
{ where:
{ id: req.params.albumId },
returning: true /*or: returning: ['*'], depending on version*/
});
I was very exited when I saw this. A very good way to avoid quite a few lines of code. What I was doing before was to get the object by Model.findOne(), re-set every field to their new values and invoke the instance method .save(), instead of using a static method like that.
Needless to say I was quite happy and satisfied when I saw that such static method existed, disappointing however, was to learn the method only returns the instance it updates if you're running Sequelize with PostgreSQL.
Very sad to learn that, as I'm using MySQL.
The method sure, issues a SQL statement containing the proper UPDADE string in it, but that's it. I don't know if it hit anything, and I don't have a copy of the updated data to return.
Turns out I need a Model.findOne() first, in order to know if that object exists with that id(and/or other filtering parameters), the Model.update() to issue the updates and finally a Model.findByPk() to return the updated model to the layer above (all of it inside a transaction, naturally). That's too much code!
Also during the update, if there's a UniqueConstraintError exception thrown (witch can be quite common), it's errors[] array carries no valid model 'instance', it's just 'undefined', so it complicates matters if you want details about what happened and/or throw custom error messages defined inside the models.
My questions are: Are there workarounds out there better than those I'm already implementing? Any sequelize plugins that may give me that with MySQL? Any sequelize beta code that can give me that? Is there any effort by the part of the sequelelize dev team to give us that? I'd appreciate any help given.
I'm running Sequelize version is 6.7.0, with Node.js v14.17.5.
Ps.: I even realized now that static Model.update() under MySQL will even update something that doesn't exist without complaining about it.
When I'm offline, if I add an object to a path where a cloud function is listening, and then I delete it while still offline, when going online, Firebase servers will receive the object creation, then right after its deletion.
The problem is that it will trigger, on creation, a cloud function. This cloud function will catch some data at another path and will add that data in the object that was created. But because the object was deleted while offline, it ends up being deleted. But the cloud function will recreate it (partially) when adding the data it went to grab somewhere else.
Because I don't want to have to track every single object I create/delete, I thought about checking if the object would still exist right before saving that data. The problem is that when I do so, the object still exist but by the time I save the data into it, it doesn't exist anymore.
What are my options? I thought about adding a 0.5s sleep but I don't think it's the best practice.
First of all, there's not much you can do on the client app to help this situation. Everything you do to compensate for this will be in Cloud Functions.
Second of all, you have to assume that events could be delivered out of order. Deletes could be processed by Cloud Functions before creates. If your code does not handle this case, you can expect inconsistency.
You said "I don't want to have to track every single object I create/delete", but the fact of the matter is that this is the best option if you want consistent handling of events that could happen out of order. There is no easy way out of this situation if you're using Cloud Functions. On top of that, your functions should be idempotent, so they can handle events that could be delivered more than once.
One alternative is to avoid making changes to documents, and instead push "command" objects to Cloud Functions that tell it the things that should change. This might help slightly, but you should also assume that these commands could arrive out of order.
This is all part of the downside of serverless backends. The upside is that you don't have to set up, manage, and deallocate server instances. But your code has to be resilient to these issues.
I am adding caching (redis) into my project and would prefer to code it into the model logic rather than the controller. I would need to overwrite the model method and add the logic for the caching there.
I know I can override certain methods like find and findOne but I'm not sure what to return.
Example (pseudo)
findOne: function () {
cache.get(key, function (err, data) {
if (data === null) // No cache data
// get the data and return it
else
// return the cache data
});
}
The problem is that these model methods dont just return the data, they return an instance of the model itself (for chaining).
Not really sure how to return the data and how to get it if it isn't already set. Has anyone ever done anything like this?
Caching is something that we'd love for Waterline, but at the moment the only way to really get what you want is to create your own adapter. Overriding find and findOne is not really feasible at this point, as there's no good way to access the underlying "parent" methods in case your cache turned up empty and you wanted to proceed with the query.
In your case, forking one of the existing adapters (like sails-mysql) to add Redis caching would probably be more constructive than starting from scratch. If one could add the caching layer as a separate, installable module (i.e. a dependency) of the forked adapter, it would be easier to replicate the functionality across other adapters, and eventually roll into the adapter spec itself. If anyone felt like tackling this it would be a great contribution! You might also ask in the Sails IRC channel (irc://irc.freenode.net/sailsjs) to see if anyone's already working on something similar.
Are there some benefits of using the methods defined on the localStorage object versus accessing the object properties directly? For example, instead of:
var x = localStorage.getItem(key);
localStorage.setItem(key, data);
I have been doing this:
var x = localStorage[key];
localStorage[key] = data;
Is there anything wrong with this?
Not really, they are, basically, exactly the same. One uses encapsulation (getter/setter) to better protect the data and for simple usage. You're supposed to use this style (for security).
The other allows for better usage when names(keys) are unknown and for arrays and loops. Use .key() and .length to iterate through your storage items without knowing their actual key names.
I found this to be a great resource : http://diveintohtml5.info/storage.html
This question might provide more insight as well to some: HTML5 localStorage key order
Addendum:
Clearly there has been some confusion about encapsulation. Check out this quick Wikipedia. But seriously, I would hope users of this site know how to google.
Moving on, encapsulation is the idea that you are making little in and out portals for communication with another system. Say you are making an API package for others to use. Say you have an array of information in that API system that gets updated by user input. You could make users of your API directly put that information in the array... using the array[key] method. OR you could use encapsulation. Take the code that adds it to the array and wrap it in a function (say, a setArray() or setWhateverMakesSense() function) that the user of your API calls to add this type of information. Then, in this set function you can check the data for issues, you can add it to the array in the correct way, in case you need it pushed or shifted onto the array in a certain way...etc. you control how the input from the user gets into the actual program. So, by itself it does not add security, but allows for security to be written by you, the author of the API. This also allows for better versioning/updating as users of your API will not have to rewrite code if you decide to make internal changes. But this is inherent to good OOP anyhow. Basically, in Javascript, any function you write is a part of your API. People are often the author of an API and it's sole user. In this case, the question of whether or not to use the encapsulation functions is moot. Just do what you like best. Because only you will be using it.
(Therefore, in response to Natix's comment below...)
In the case here of JavaScript and the localStorage object, they have already written this API, they are the author, and we are its users. If the JavaScript authors decide to change how localStorage works, then it will be much less likely for you to have to rewrite your code if you used the encapsulation methods. But we all know its highly unlikely that this level of change will ever happen, at least not any time soon. And since the authors didn't have any inherent different safety checks to make here, then, currently, both these ways of using localStorage are essentially the same. Except when you try to get data that doesn't exist. The encapsulated getItem function will return null (instead of undefined). That is one reason that encapsulation is suggested to be used; for more predictable/uniform/safer/easier code. And using null also matches other languages. They don't like us using undefined, in general. Not that it actually matters anyhow, assuming your code is good it's all essentially the same. People tend to ignore many of the "suggestions" in JavaScript, lol! Anyhow, encapsulation (in JavaScript) is basically just a shim. However, if we want to do our own custom security/safety checks then we can easily either: write a second encapsulation around the localStorage encapsulate, or just overwrite/replace the existing encapsulation (shim) itself around localStorage. Because JavaScript is just that awesome.
PT
I think they are exactly the same, the only thing the documenation states is:
Note: Although the values can be set and read using the standard
JavaScript property access method, using the getItem and setItem
methods is recommended.
If using the full shim, however, it states that:
The use of methods localStorage.yourKey = yourValue; and delete
localStorage.yourKey; to set or delete a key is not a secure way with
this code.
and the limited shim:
The use of method localStorage.yourKey in order to get, set or delete
a key is not permitted with this code.
One of the biggest benefits I see is that I don't have to check if a value is undefined or not before I JSON.parse() it, since getItem() returns NULL as opposed to undefined.
As long as you don't use the "dot notation" like window.localStorage.key you are probably OK, as it is not available in Windows Phone 7. I haven't tested with brackets (your second example). Personally I always use the set and get functions (your first example).
Well, there is actually a difference, when there is no local storage available for an item:
localStorage.item returns undefined
localStorage.getItem('item') returns null
One popular use case may be when using JSON.parse() of the return value: the parsing fails for undefined, while it works for null