My aim is to find a way to dynamically pull in environment-specific config values for various 'tracks' of my React app: development, staging and production. The solution most often prescribed involves the use of environment variables, which I don't find great because:
Some of my config values are sensitive data like API secret keys, database passwords, etc and I'd ideally not be keeping these in plain-text both locally and on a CICD system
Having to manually set env vars is error prone and doesn't scale well (it's a big project that has more than 20 config-related key-value pairs to set). It's also difficult to document which env vars need to be set, so it's not a convenient solution for a multi-collaborator team as everyone needs to keep track of the list and copy-paste the values into their local machines for shared API keys, etc (or worse, hard-coding/checking them into the source code)
I have tried the following 2 general approaches:
Use node-config - it looks promising as it's light, flexible, and extensible (it allows defining base values on default.js and overriding them with development.js, staging.js, production.js or with custom env variables). Most importantly, we can store secrets in a remote service (e.g AWS/GCP Secrets Manager, envkey, etc). This solution works well for my Node backend, but so far not for the frontend app built on React
Use dotenv (or dotenv-safe, to allow documenting the structure of .env file in another one .env.example that is checked into source control). This is not my favored approach as dotenv discourages using multiple .env files for each environment our project needs. Secondly, I'd likely still have to find another way to feed in the env variables into my CICD system. Redefining the env vars on the [remote] build system feels like doing the work twice - the first being on the .env files used for local development.
Both approaches yield a familiar problem: TypeError: fs.readFileSync is not a function. According to this related question, it appears that the underlying issue is that the 'fs' module is not designed to work on the browser (both dotenv and node-config are low level modules that use 'fs' under the hood). If we cannot use fs (or rather, modules that rely on it) on the client side: how do scalable/production-grade React projects typically manage config values in a sane way? I know hashicorp/vault exists but it seems a bit of an overkill as we'd likely have to set up our own infrastructure.
I also wonder if there's any open-source tools out there to solve this common problem...
Neither of the two solutions offered above really met my requirements, first because I'm using a create-react-app project so don't have much control over webpack configuration. Secondly, I'd much prefer to not keep .env files locally (let alone in plain text)
Luckily, I came across https://doppler.com/, a universal secrets management solution that solves my needs as described on the OP:
it's a cloud-based secrets store + manager that comes with a CLI, which allows us to use the same env secrets across the entire pipeline (local development, CICD and production)
projects come loaded with development, staging and production environments that makes it easy to switch easily between different flavors of the app
Because Doppler works by injecting environment variables into the runtime, I can run it like so, with yarn:
doppler run -- yarn start
For server environments that need to first inject the env vars into a bundled app (e.g the firebase emulator), first do a 'doppler-injected' build:
doppler run -- yarn build
And then run the emulator as usual:
firebase emulators:start
Using separate dotenv for example, .env.dev , .env.qa would be my current approach to a scalable react project. I am assuming you are using webpack and are facing issues in fetching the files using fs, as separately fs is a node server side module.
To resolve this TypeError issue, in your webpack config, you can use fs to access the current working directory,
const fs = require('fs');
const path = require('path');
const reactAppDirectory = fs.realpathSync(process.cwd());
const resolveReactApp = (relativePath) => path.resolve(reactAppDirectory, relativePath);
You need to copy all your assets and build into a spearate public folder have a web.config and resolve its path using the above resolveReactApp('public') in contentBase key of devServer section of your webpack config.
You can pass the environment variables using webpack's DefinePlugin or EnvironmentPlugin,
new webpack.DefinePlugin({
'process.env.NODE_ENV': JSON.stringify(process.env.NODE_ENV),
'process.env.DEBUG': JSON.stringify(process.env.DEBUG)
});
new webpack.EnvironmentPlugin(['NODE_ENV', 'DEV']);
To store secrets, I would suggest you check out docker-secrets and using a containerized architecture for deployment, and further when the team expands, it'll be easier to get the project setup as well. Here's a quick setup intro of how to use docker with react and more info on switching from environment variables to docker-secrets for a better system architecture.
Related
I have a node module that I am building, and I want it to be able to execute on the server (in nextjs server side rendering) and the client (call additional lifecycle methods in the UI). This same module also needs to work when used purely as a js library that can be included in a <script> tag on the page. This module depends on the uuid module, which has logic in it to check if it is running in a browser or server context, and use the proper random number generators/crypto libraries that are available in that context.
If I don't specify a target in my webpack config, the bundle works great in a client browser. It includes the webpack browser logic just fine. But it doesn't work in the server case - webpack removed the server capable logic in the uuid module.
If I target: 'node' in my webpack config - it executes just fine as a node module on the server and the client. It seemingly included all of the logic this time. But now it doesn't work if included just as a script tag on the page. I get ReferenceError: require is not defined from the file that depends on the uuid module.
It seems like the uuid module should handle these different environments just fine, but webpack is messing with that. How can I let that module resolve the proper implementation at runtime?
I unfortunately do not have a minimally reproducible example, or additional code to share at this time. I figured someone might have run into this with webpack (or even webpack and the uuid module) and know the solution.
I was trying to do this by building a single version of the package, but I don't think that is possible.
What is possible is building multiple versions, and then hosting the web bundle via unpkg or jsdelivr via an entry in package.json. Those entries can point to the target: 'web' version of the package, while the npm package can point to the target: 'node' version.
It seems good practice for packages to provide some type of demo app, so I'm just wondering what's the cleanest way to organize the file structure?
I want to have one github repo that contains my published NPM module and a simple demo webapp.
Ideally I would like a top level like:
package/
demo/
and have the code just in package/ get distributed to NPM.
I could use the package.json files option like
files: [ 'package' ]
But then all the code will get distributed with that path prefix, eg
node_modules/MyPackageName/package/index.js
Is there a way to modify the path prefix so it changes the top-level directory and removes the extra package/ I used to organize the files?
Sure other people have ways to do this, but I'd prefer not to use two repos - one demo and one package.
Clarification I want to be able to install the package directly from github, as a kind of "poor-mans private NPM". So I don't want to just publish from within the 'package' directory. I think using github URLs you can specify a branch to use, but not a subdirectory.
You can do this with the help of NODE_PATH env variable:
export NODE_PATH='yourdir'/node_modules
Reference
If the NODE_PATH environment variable is set to a colon-delimited list of absolute paths, then Node.js will search those paths for modules if they are not found elsewhere.
On Windows, NODE_PATH is delimited by semicolons (;) instead of colons.
NODE_PATH was originally created to support loading modules from varying paths before the current module resolution algorithm was frozen.
NODE_PATH is still supported, but is less necessary now that the Node.js ecosystem has settled on a convention for locating dependent modules. Sometimes deployments that rely on NODE_PATH show surprising behavior when people are unaware that NODE_PATH must be set. Sometimes a module's dependencies change, causing a different version (or even a different module) to be loaded as the NODE_PATH is searched.
Additionally, Node.js will search in the following list of GLOBAL_FOLDERS:
1: $HOME/.node_modules
2: $HOME/.node_libraries
3: $PREFIX/lib/node
Where $HOME is the user's home directory, and $PREFIX is Node.js's configured node_prefix.
These are mostly for historic reasons.
It is strongly encouraged to place dependencies in the local node_modules folder. These will be loaded faster, and more reliably.
I'm building SPA with Vue and serve it with nodejs and wrapping it in docker container.
Here is the problem, I'm trying to stick to 12 factor app where for configuration it says keep in env file.
VueJS provides configs for different environment in config folder. But, according to 12 factor app config should not be in files based on environment.
In a twelve-factor app, env vars are granular controls, each fully orthogonal to other env vars. They are never grouped together as “environments”, but instead are independently managed for each deploy.
So how can I access nodejs environment variables in VueJS app?
EDIT:
Thanks for the answers.
The whole idea is to change for example api url on run time trough providing different env variable. If I commit the config file with api url, I would have to rebuild the container on commit and deploy it just for this small change.
I could also have a api access key that is different in dev and prod.
I'm looking for the best way possible to do this kind of things in SPA.
SPA applications nowadays usually go through a build step. This means compiling all of your files into [near to] one dist file and an index.html which may be served statically. This creates a clear separation between front-end (VueJS) and backend (NodeJS). The index.html and js files themselves continue to be static files nonetheless.
This is usually what you want since you can scale server and client independently: Serve static files, say, through s3 + cdn and run your nodejs server independently.
I think what you want is a way to pass runtime configuration to the client. I wouldn't get too caught up on the details of actually sharing the envvars per se.
In your case, I see two possible solutions:
1) Implement an API to access whitelisted envvars from your server - You can think of this as a /config endpoint
2) Render the index.html dynamically via nodejs with something like ejs with the prepopulated envvars - You'll have more coupling between frontend and server but you could extend this to much more than envvars and, say, preopolute the frontend with prefetched data.
Regardless on how you do it, you can consider this runtime configuration for the frontend which should not be attempted to be fixed at build time since otherwise you may be expose sensitive data into static files and it is not always guaranteed that you have all the data at this time.
The way you access the env variables should be the same no matter which OS you are using. The way you set them is different though. On windows I would set an environment variable like so
set PASSWORD=somepassword
And then in the code I can access this variable by doing the following
var pw = process.env.PASSWORD;
You should be able to use this the same way in VueJS.
EDIT:
If you use docker-compose you can set the endpoint on the fly by using environment variables too. Then whenever you docker-compose up the endpoint will be updated with the current value of your environment variable. In your shell update the api endpoint to whatever you want it to be.
set API_URL=http://my-api-endpoint
Then in the docker-compose.yml you would access this value like so
version: '3'
services:
app:
image: 'myapp'
environment:
- API_URL=${API_URL}
You would still access the variable in your code using process.env.API_URL as I mentioned in my example above.
our devops team will use https://github.com/tinou98/vue-12factor
However it's really a big question mark how VueJs does not consider that as a mature frontend/spa framework.
We used to build React Apps since more than 6 years with a built-in support of externalizing env.js file ( create-react-app )
This seems like a situation that must be incredibly common, but I've yet to find a good solution to it. I'm a little new to modern Javascript, so please forgive me (or better yet, correct me) if I'm using the wrong terminology here and there.
I'm developing a web application. It's got a server, running as Javascript (ES6, I believe - using import/export) in node.js, using express.js for a router (and built by express-cli). And it's got a client side, also Javascript, mostly Vue modules (and built by vue-cli). I admit I don't really understand a lot of the boilerplate build code express-cli and vue-cli emitted, but it does work. I am not using any of a long list of frameworks that are assumed in the many pages google found for me when I tried to find an answer to this question.
Obviously, the client and server will be sending data structures (instances of various classes) back and forth, which I know how to do. And these ought to be the same class definitions.
I made an attempt to make webpack build both server and client, and that failed, so now I've split the application into two projects, each with its own folder tree, its own package.json, its own node_modules, and I'm using just webpack-dev-server for the client and nodemon/babel for the server. A third folder contains the shared code, which is imported by the client and the server. I got this to work well enough for proof of concept, but getting both sides (and Visual Studio Code) to recognize that the shared code is part of them is turning out to be a challenge, and I'm pretty sure I'm just going down the wrong path.
So, my question is, what is currently considered the best practice (or at least, a good practice) way to structure and build a client/server application of this type? An ideal answer to this question would include both folder structure and enough of the major configuration files to help me figure out how to write mine. A reference to an up-to-date and reliable source of information on this topic would satisfy me nicely.
I suspect that the right answer includes merging everything back into a single project and doing something clever with the webpack config files and maybe project.json... but what exactly that clever thing might be has so far eluded me.
Thanks!
In my opinion, you're right to use separate folders for your node.js(server)/vue.js(client app) as they're effectively 2 separate projects.
Apart from sharing some configs, utilities and validation, the app and server typically have little overlap: they're typically doing 2 very different things, require different build tools, different runtime environments, have different security concerns, and if you consider the future possibility of leveraging your client codebase to create a native IOS/android... app the difference between the 2 codebases will only increase.
For my current project, I have a folder structure whereby my server resides in the project root and my client is in a subfolder /app. Here's an simplified outline of how I've structured my node.js/vue.js project:
constants
config.js (server environment, database connection, api keys etc)
settings.js (business logic)
pricing.js
drivers
auth.js
db-connect.js
email.js
sms.js
socket-io-server.js
models (mongoose database models)
node_modules
routes (express routes)
api.js
auth.js
schema (json schema for automated validation)
login-validation.js
register-validation.js
services
billing.js
validation.js
app (this is my client sub-project sharing some server modules)
public (the output of the webpack client bundle including index.html)
src (ES6 source code, images, SASS/Stylus, fonts etc)
css
html (handlebars templates)
index.hbs
home.hbs
account.hbs
pricing.hbs
img
js
api
client-services
socket-io-client.js
store.js
router.js
vuex-utils.js
dom-utils.js
components (Vue components)
Profile.vue
Payment.vue
index.js (root and entry point for webpack)
webpack.config.js
.env.development.js
.env.production.js
package.json
server.js (node.js server root entry point)
This is by no means prescriptive. Notice, that I've organized the project files largely by function. Here's a link to an article proposing structuring around features.
A shared (client/server) module folder has pros and cons. The main downside I can see is that module sharing changes during development. For that reason my server codebase and modules reside in the project root folder - and some are also imported by the client app. The project root folder then naturally hosts a shared package.json and node_modules folder.
As your app develops, and you gain insight, it's fine to re-structure as the need arises (some IDEs make refactoring easier than others). If the Visual Studio Code IDE or webpack are not working well with your folder structure, it may be a configuration issue or the problem may stem from incorrect require/import paths. IDE inspections may help you find those errors or your IDE may struggle to be useful because of those errors.
My IDE (webstorm) and webpack v4 have no problems with the above folder structure or the location of modules in general (I have re-structured my app in many different ways) and got more adept at optimally configuring my IDE and webpack.
Webpack is very specific about the location of it's configuration file, input/output paths, whether it is executed from the project root or /app folder and it can take a lot of time to get it working properly. Nevertheless, it will locate modules referenced by the correct require/import paths. Here's the part of my webpack.config.js that sets up file entry/output.
const sourcePath = './src';
module.exports = {
entry: [
sourcePath + '/js/index.js'
],
output: {
path: __dirname + '/public', //location of webpack output files
publicPath: '/', //address that browser will request the webpack files
filename: 'js/index.[contenthash].js', //output filename
chunkFilename: 'js/chunk.[contenthash].js' //chunk filename
},
...
I've located my webpack.config.js and execute webpack inside the /app subfolder. My package.json scripts section to start the node.js server and have webpack build & watch my app files is:
"scripts": {
"start": "if [$NODE_ENV == 'production']; then node server.js; else nodemon server.js; fi",
"build": "cd ./app; webpack;"
},
Which allows me to start the node server with:
> npm start
and have webpack watch/re-build my bundle with:
> npm run build
Visual Studio Code has a rich IntelliSense experience and a feature called Automatic Type Acquisition which should allow it to support modules you import regardless of their location. This article provides more information on configuring VS Code for node.js.
I'm working on a Windows Store App (JavaScript/HTML/CSS) that will be deployed directly to devices in our enterprise.
I want to keep the datasources (urls to Restful web APIs) as part of the configuration rather than built into the app itself so that I can set them during deployment (e.g. to set test urls and prod urls).
More generally I want to store text variables in config that is external to the app and can be pulled in by the app somehow.
I thought I could set some environment variables or something but Windows Store Apps can't read them it seems.
Any ideas?
You could certainly make an HTTP request from the app on startup to retrieve a configuration file, but that of course assumes connectivity which may or may not work in your scenario. For a Store-acquired app, this is really the only choice.
In your scenario, however, you'll be doing side-loading through a Powershell, correct? (This is implied in installing directly to devices.) In that case, the Powershell script is running in full trust and will have access to the file system during the process. This means that the script can easily deploy a configuration file into the app's local appdata folder, which the app then picks up when it runs. The app package should also contain a default configuration file that it copies into that appdata folder if such a file doesn't exist on startup.
The documentation for the add-appxpackage script that does the install is here: https://technet.microsoft.com/en-us/library/hh856048.aspx.
Another option you might be able to use is to build different versions of your packages for test and production deployment. It is possible to configure the build process in Visual Studio to selectively bring in different versions of a file depending on your build target (e.g. Debug or Release). I have a blog that describes this technique on http://www.kraigbrockschmidt.com/2014/02/25/differentiate-debug-release-builds-javascript/. This would allow you to package different versions of a configuration file into the package, which you'd then read from the package install location at runtime or copy to appdata if you wanted to make changes at runtime.
I mention this method for building different packages because it's something that doesn't need you to do anything other than change the build target. It accomplishes what you would do with #ifdef precompiler directives in other languages, which aren't available for JavaScript.