I have a basic express server that needs to store some global variables during each request handling.
More in depth, request handling involves many operation that need to be stored in a variable such as global.transaction[]
Of course if I use the global scope, every connection will share information of its transaction and I need a global scope because I need to access the transaction array from many other modules, during my execution.
Any suggestion on this problem? I feel like is something very trivial but I'm looking for complicated solutions :)
Many thanks!
UPDATE
This is a case scenario, to be more clear.
On every request I have 3 modules (ModuleA, ModuleB, ModuleC) which read the content of 10 random files in one directory. I want to keep track of the list of file names read by every request, and send back with res.write the list.
So ModuleA/B/C need to access a sort of global variable but the lists of request_1, request_2, request_3 etc... don't have to mix up.
Here is my suggestion avoid global state like fire.
It's the number one maintenance problem in Node servers from my experience.
It makes your code not composable and harder to reuse.
It creates implicit dependencies in your code - you're never sure which piece depends on which and it's not easy to verify.
You want the parts of code that each piece of an application uses to be as explicit as possible. It's a huge issue.
The issue
We want to synchronize state across multiple requests and act accordingly. This is a very big problem in writing software - some say even the biggest. The importance of the way objects in the application communicate can not be overestimated.
Some solutions
There are several ways to accomplish sharing state across requests or server wide in a Node server. It depends on what you want to do. Here are the two most common imo.
I want to observe what the requests do.
I want one request to do things based on what another request did.
1. I want to observe what the requests do
Again, there are many ways to do this. Here are the two I see most.
Using an event emitter
This way requests emit events. The application reads events the requests fire and learns about them accordingly. The application itself could be an event emitter you can observe from the outside.
You can do something like:
request.emit("Client did something silly", theSillyThing);
And then listen to it from the outside if you choose to.
Using an observer pattern
This is like an event emitter but reversed. You keep a list of dependencies on the request and call a handler method on them yourself when something interesting happens on the request.
Personally, I usually prefer an event emitter because I think they usually solve the case better.
2. I want one request to do things based on what another request did.
This is a lot tricker than just listening. again, there are several approaches here. What they have in common is that we put the sharing in a service
Instead of having global state - each request gets access to a service - for example when you read a file you notify the service and when you want a list of read files - you ask the service. Everything is explicit in the dependency.
The service is not global, only dependencies of it. For example, it can coordinate resources and the data, being some form of Repository).
Nice theory! Now what about my use case?
Here are two options for what I would do in your case. It's far from the only solution.
First option:
Each of the modules are an event emitter, whenever they read a file they emit an event.
A service listens to all their events and keeps count.
Requests have access to that service explicitly and can query it for a list of files.
Requests perform writes through the modules themselves and not the added service.
Second option:
Create a service that owns a copy of module1, module2 and module3. (composition)
The service delegates actions to the modules based on what is required from it.
The service keeps the list of files accessed since the requests were made through it.
The request stops using the modules directly - uses the service instead.
Both these approaches have advantages and disadvantages. A more complicated solution might be required (those two are in practice pretty simple to do) where the services are abstracted further but I think this is a good start.
One simple way is storing data on the request object.
Here is an example (using Express):
app.get('/hello.txt', function(req, res){
req.transaction = req.transaction || [];
if (req.transaction.length) {
// something else has already written to this array
}
});
However, I don't really see how you can need this. When you call moduleA or moduleB, you just have to pass an object as argument, and it solves your issue. Maybe you're looking for dependency injection?
using koa ctx.state doc for this scenario, in express I believe this Plugin should serve your needs.
in order to keep some data that will be resused by another request on the save server app, I propose to use session in expresse and avoid any global state or any props drilling from one request to another.
In order to manage session state in express you could use :
session-file-store save the session in a file
express-mongodb-session : save the session in mongoDb
mssql-session-store -> for a relation db
Of course there is another technique ti manage session in NodeJs.
Related
TL;DR: Is it bad practice to use a global variable to prevent code executing unnecessarily, and if so what are the alternatives?
I have an Electron application that reads a stream of real time data from another application and outputs some elements of it on screen.
There are two types of data received, real time (telemetry) data of what is currently going on and more static data that updated every few seconds (sessionInfo).
The first time a sessionInfo packet is received I need to position and size some of the UI elements according to some of the data in it. This data will definitely not change during the course of the application being used, so I do not want the calculations based on it to be executed more than once*.
I need to listen for all sessionInfo packets, there are other things I do with them when received, it is just this specific part of the data that only needs to be considered once.
Given the above, is this a scenario where it would be appropriate to use a global variable to store this information (or even just a flag to say that this info had been processed) and use this to prevent the code executing multiple times? All my reading suggests that Global Variables are never a good idea, but short of allowing this code to execute repeatedly I am unsure what alternatives I have here.
*I recognise that allowing this would probably make no practical difference to my application, but this is a learning experience for me as well as producing something useful so I would like to understand the 'right' approach rather than just bodge something inefficient together and make it work.
Global variables are generally a bad practice for many reasons like:
They pollute the global scope, so if you create a global i, you
can't use that name anywhere else.
They make it impossible or very difficult to unit test your code.
You usually end up needing more of them than you think, to the point where many things become global.
A better practice is to create singleton services. For example, you might create a SessionService class that handles all your auth stuff. Or, perhaps a DataStreamService that holds the state variables of your data stream. To access that singleton, you'd import it into whatever file needs it. Some frameworks go even farther and have a single global data store, like react/redux.
Currently our project is handling logging through static methods in a class. We use a transaction id to track the logs of a request and don't confuse with the logs of another.
But this has one concurrency problem.
request1 set up the transaction id (static)
request2 set up the transaction id (another one)
request1 logs, then it uses the id of request2.
Our architect propose to instantiate a logger and pass it as parameter in all functions, but that is quite annoying since there are a lot of functions calls, so that means adding a new parameter, not meaningful from a functional viewpoint, to hundreds of functions.
Any other solution I can think needs a way to, in any function, differentiate which context it belongs to, I mean, which request called the function.
I am very novice in nodejs, is there anykind of global variable or other mechanism that could me help here?
There is a blog sharing how this can be achieved.
I took a look on the library it used, it's referring to cls-hooked which uses some internal/experimental Node.js features.
Hence, it's not quite recommended to use in production.
Edited
Recently, I discover a library when learning https://zipkin.io. It uses https://github.com/othiym23/node-continuation-local-storage to keep track of value like thread-local storage. Worth to take a look
There is a similar thread that is currently two years old, and it pretty much sums up what I want.
To give you an example, for the C# developers there is a single Session object which stores user's session and is accessible anywhere, in each class, .dll, you name it. It's stored per-request.
I need basically the same thing. My users have simple permissions, they are either admins or have roughly 5 assigned elements. I'm storing these 5 elements in an access token, so basically in the first middleware I have an array of ids which are user's permissions.
What I want is not to pass a permissions object around, especially not to pass them to service/DAL layers which are responsible for querying the database since it's pretty ridiculous for getAllCities function to receive permissions object and a callback as parameters. Or whatever object for that matter.
I cannot store the values in global variables since the globals are, well, global. I don't want to pass the variables to each function, so I would probably fall back to rounding the databases for user's permissions per each call, but that's just.. Meh.
I also don't want to mess to much with dozens of libraries etc., because I believe this is a common problem that people stumble upon and I wonder what are the best solutions.
On the thread linked above there is a robust solution which I would frankly want to avoid. Everything I've used in Node has been plug-and-play, and implementing custom middlewares and pipelines for this "small" thing seems like an overkill.
Two years later, is there a better solution out there to this problem?
You would typically use something like express-session to manage a session for each user and then each incoming request would automatically have access to that session object for that user in the request handler. You can then store any user-specific information in that session object.
Any functions that you want to execute in a given request should either be passed the request object (from which they can get access to the session) or just pass them the session object or some user-specific object you pulled out of the session. In node.js the place for user-specific information is in the request object. There is no "global" space to store it that all other functions can access it.
If you want a function to be able to access request or session specific data, then pass that data to the function. That's how it works. If this appears to be a problem to you, then you have to rethink how your code is architected and perhaps use objects/methods that can maintain state more easily.
node.js does not have a thread per request that can have it's own thread state. Instead, in an http server, you use the request object as the core state for a given request and you pass it to functions that need it. Or, if you're using a user session, you can extract that from the request object and then pass the session around as needed.
I don't want to pass the variables to each function, so I would
probably fall back to rounding the databases for user's permissions
per each call, but that's just.. Meh.
It sounds to me like too much of your code is just plain functions rather than methods on an object. If you think of your code as a user object that has methods on it, then you can form a user object in the initial request and then call methods on that user object for most things you would want to do that are user-specific. The user object then is a nice simple repository for user-specific information (such as permissions) and ALL methods on the object will have access to it without having to pass it to every function.
I repeat, if passing the user state around seems like a burden, then it's probably because your code is not architected in a way that makes it easy. Rather than looking for a work-around for something that is not the way node.js works, you should think about architecting your code to make these kinds of things easy so your architecture more closely matches the needs of your code given the way the underlying system works.
implementing custom middlewares and pipelines for this "small" thing
seems like an overkill.
Session middleware is not a small thing. It's core to any application that maintains user-specific data across requests and wants easy access to it on each new request from that same user. It's generally how you manage user state in node.js.
To give you an example, for the C# developers there is a single
Session object which stores user's session and is accessible anywhere,
in each class, .dll, you name it. It's stored per-request.
That simply isn't how node.js works at the lowest architectural level. And, the sooner you realize that and start designing your code to align with the way node.js actually works (rather than the way some other system works), the sooner you will find node.js coding flows naturally. It's something new to learn, not to be avoided or worked-around.
I'm trying to understand the Flux example chat app. The authors mention this unidirectional data flow:
However, in the example app there are dependencies between Action Creators (ChatMesssageActionCreator) and Stores (MessageStore), and between Stores (MessageStore, ThreadStore) and Web API Utils (ChatMessageUtils), which seems to be against the unidirectional data flow rule:
Is it recommended to follow the given example, or should one design a better pattern?
Update
I figured out that the ChatMessageUtils doesn't belong to Web API Utils, so the two arrows from store shouldn't point there, therefore maybe they're okay.
However the connection between the ActionCreators and the Store seems still strange.
The example is a bit forced, and it was created with the purpose of trying to show how waitFor() works. The WebAPI aspect of the example is pretty half-baked and really should be revised.
However, even though MessageStore.getCreatedMessageData(text) passes a value to the store, it's still a getter. It's not setting data on the store. It's really being used as a utility method, and a good revision (pull request?) would be to move that method to a Utils module.
To improve upon the example for the real world, you might do a couple things:
Call the WebAPIUtils from the store, instead of from the ActionCreators. This is fine as long as the response calls another ActionCreator, and is not handled by setting new data directly on the store. The important thing is for new data to originate with an action. It matters more how data enters the system than how data exits the system.
Alternatively, you might want to have separate client-side vs. server-side IDs for the messages. There might be few advantages of this, like managing optimistic renderings. In that case, you might want to generate a client-side id in a Utils module, and pass that id along with the text to both the dispatched action and the WebAPIUtils.
All that said, yes the example needs revision.
I have a node server for loading certain scripts that can be written by anyone. I understand that when I fire up my Node server, modules load for the first time in the global scope. When one requests a page, it gets loaded by the "start server" callback; and I can use all the already loaded modules per request. But I haven't encountered a script where global variables get changed during request time and affects every single other instance in the process (maybe there is).
My question is, how safe is it, in terms of server crashing, to alter the global data? Also, suppose that that I have written a proper locking mechanism that will "pause" the server for all instances for a very short amount of time until the proper data is loaded.
Node.js is single threaded. So it's impossible for two separate requests to alter a global variable simultaneously. So in theory, it's safe.
However, if you're doing stuff like keep user A's data temporarily in a variable and then when user A later submits another request use that variable be aware that user B may make a request in between potentially altering user A's data.
For such cases keeping global values in arrays or objects is one way of separating user data. Another strategy is to use a closure which is a common practice in callback-intensive or event/promise oriented libraries such as socket.io.
When it comes to multithreading or multiprocessing, message passing style API like node's built-in cluster module has the same guarantees of not clobbering globals since each process have its own global. There are several multithreading modules that's implemented similarly - one node instance per thread. However, shared memory style APIs can't make such guarantees since each thread is now a real OS thread which may preempt each other and clobber each others memory. So if you ever decide to try out one of the multithreading modules, be aware of this issue.
It is possible to implement fake shared memory using message passing though - sort of like how we do it with ajax or socket.io. So I'd personally avoid shared memory style multithreading unless I really, really need to cooperatively work on a very large dataset that would bog down message passing architectures.
Then again, remember, the web is a giant message passing architecture with the messages being HTML and XML and json. So message passing scales to Google size.