I was wondering if anyone here had some ideas or experience getting rid of long, hard-to-maintain relative imports inside of large node.js cloud functions projects. We’ve found that the approach which uses local NPM packages is very sub-optimal because of how quickly we tend to roll out and test new packages and functionality, and refactoring from JS to TS is impossible for us at the moment. We'd love to do it in the near future but are so slammed as it is currently :(
Basically what I’m trying to do in cloud functions is go from const {helperFunction} = require(‘../../../../../../helpers’)
to
const {helperFunction} = require(‘helpers’)
I have been unable to get babel or anything similar to that working in cloud functions. Intuitively I feel like there is an obvious solution to this beyond something like artifact registry or local NPM packages but i’m not seeing it! Any help would be greatly, greatly appreciated :)
For CommonJS modules in general, you have several options. I don't know the Google Cloud environment so you will have to decide which options seem appropriate to it.
The most attractive option to me is #5 as it's just a built-in search up the directory tree, designed specifically to look in the node_modules sub-directories of your parent directories. If you can install the shared modules in that way, then nodejs should be able to find them as long as your directory hierarchy is retained by Google Cloud.
Options #3 and #4 are hacking on the loader which has its own risks, but does give you imlementation flexibility as you could implement your own prefix that looks in a particular spot. But, it's hacking and may or may not work in the Google cloud environment.
Options #1 and #2 rely on environment variables and shared directories which may or may not be relevant in the Google cloud environment.
You can specify the environment variable NODE_PATH as a colon-delimited (or semi-colon on Windows) list of paths to search for modules. Doc here.
In addition, nodejs will search $HOME/.node_modules and $HOME/.node_libraries where $HOME is the user's home directory.
There is a package called module-alias here that is designed specifically to help you solve this. You define module aliases in your package.json, import this one module and then you can use the directory aliases in your require() statements.
You can make your own pre-processor for resolving a module filename that is being loaded by require. You cando this by monkey patching Module._resolveFilename to either modify the filename passed to it or to add additional search paths to the options argument. This is the general concept that the module-alias package (mentioned in point #3 above) uses.
If the actual location of the helper module you want to load is in a node_modules directory somewhere above your current module directory on this volume, it can be found automatically as long as the require is just a filename as in require("helpers"). An example in the doc here describes this:
For example, if the file at /home/ry/projects/foo.js called require('bar.js'), then Node.js would look in the following locations, in this order:
/home/ry/projects/node_modules/bar.js
/home/ry/node_modules/bar.js
/home/node_modules/bar.js
/node_modules/bar.js
You can see how it automatically searches up the directory tree looking in each parent node_modules sub-directory all the way up to the root. If you put these common, shared modules in your main project file's node_modules or in any directory above, then it will be found automatically. This might be the simplest way to do things as it's just a directory structuring and removing of all the ../../ stuff you have in the paths. Clearly these common, shared modules are already located somewhere common - you just need to make sure they're in this search hierarchy so they can be found automatically.
Note this info is for CommonJS modules and may be different for ESM modules.
I have a typescript monorepo with front end, cloud functions, and back end (docker) packages in it. I'd like to add a "common" package that can be used by all of those.
Some production environments like Google Cloud Functions and Docker (at least in one common usage) package up the current app's directory and send it to a remote server, where it does "yarn install" to install all the dependencies. This means the obvious ways of referencing the common package (import * as common from "../../common" and the like) don't work because the build server won't find common there.
Also, sadly, docker and GCF don't follow symlinks, so the yarn link: protocol isn't helpful.
And since I'm working on the common code just as much as the apps, I really don't want to publish it to a private NPM registry every time I make a change. I'd like that change process to be as seamless as possible.
The most seamless way is to just symlink common into each app's node_modules, but (a) yarn deletes those, and (b) the builders don't follow symlinks. So I assume I'll need some kind of build/push process when changing the common code.
It seems like a kind of common thing to want to abstract out common code into its own module that can be used anywhere in the monorepo, but I can't find a solution that works. (For extra points I'd like "jump to definition" in my IDE to go to the actual definition, not a local copy.) Any ideas? Can Yarn 2 help with this use case?
What exactly should I put in .npmignore?
Tests? Stuff like .travis.yml, .jshintrc? Anything that isn't needed when running the module (except the readme)?
I can't find any guidance on this.
As you probably found, NPM doesn't really state specifically what should go in there, rather they have a list of ignored-by-default files. Many people don't even use it as everything in your .gitignore is ignored in npm by default if .npmignore doesn't exist. Additionally, many files are already ignored by default regardless of settings and some files are always excluded from being ignored, as outlined in the link above.
There is not much official on what always should be there because it is basically a subset of .gitignore, but from what I gather from using node for 5-ish years, here's what I've come up with.
Note: By production I mean any time where your module is used by someone and not to develop on the module itself.
Pre-release cross-compiled sources
Pros: If you are using a language that cross-compiles into JavaScript, you can precompile before release and not include .coffee files in your package but keep tracking them in your git repository.
Build file leftovers
Pros: People using things like node-gyp might have object files that get generated during a build that never should go into the package.
Cons: This should always go into the .gitignore anyway. You must place these things inside here if you are using a .npmignore file already as it overrides .gitignore from npm's point of view.
Tests
Pros: Less baggage in your production code.
Cons: You cannot run tests on live environments in the slim chance there is a system-specific failure, such as an out of date version of node running that causes a test to fail.
Continuous integration settings/Meta files
Pros: Again, less baggage. Things such as .travis.yml are not required for using, testing, or viewing the code.
Non-readme docs and code examples
Pros: Less baggage. Some people exist in the school-of-thought where if you cannot express at least minimum viable functionality in your Readme, your module is too big.
Cons: People cannot see exhaustive documentation and code examples on their own file system. They would have to visit the repository (which also requires an internet connection).
Github-pages objects
Pros: You certainly don't need to litter your releases with CNAME files or placeholder index.htmls if you use your module serves double-duty as a gh-pages repository as well.
bower.json and friends
Pros: If you decide to build in your dependencies prior to release, you don't need the end-user to install bower then install more things with that. I would, personally, keep that stuff in the package. When I do an npm install, I should only be relying on npm and no other external sources.
Basically, you should ever use it if there is something you wish to keep out of your npm package but checked-in to your module's repo. It's not a long list of items, but npm would rather build in the functionality than having people stuck with irrelevant objects in their package.
I agree with lante's short and syntetic answer and SamT's big answer:
You should not include your tests in your package.
Your package should only contains production runtime files.
That will make your package more straightforward and faster to be dowloaded.
My contribution to those answers:
.npmignore is the blacklist way to achieve package file selection. But in a more practical way, you can whitelist files you need to include in your package using the files field in your package.json:
{
"files": [
"lib/",
"index.js"
]
}
I think that's simpler, future proof and have better semantics ;)
Just to clarify, anytime someone do npm install your-library, npm will download all source files that the package includes. Those files that were included in the .npmignore file in the source code of the package your-library will be excluded when publishing the lib, so users of your-library won't download them.
Know that people installing your library will need just your library running, anything else will be not necessary.
For example, when someone installs a library, its probably that he/she doesn't care about your .travis.yml or your .jshintrc files, or even some images, Grunt files, documentation, etc.
.npmignore could let your npm package to have less files, and faster to be downloaded
Don't include your tests. Oftentimes tests are like 5x the size of the actual codebase. As long as your tests are on Github, etc, that's good enough.
But what you absolutely should do is test your NPM package in its published format. Create some smoke tests that reside in the actual codebase, but are not part of the test suite.
You can read about testing your package after tarballing it, here:
https://github.com/ORESoftware/r2g
How to test an `npm publish` result, without actually publishing to NPM?
I have a NodeJS project that has some server and client side, both written in ES2015 and compiled to ES5 with Babel, executed by Gulp.
Coming from the Java land, I made the build process put the results in a dist folder, so the directory structure looks like this:
./node_modules
./client
./server
./static
./dist/client # Result of /client compilation
./dist/server # Result of /server compilation
Now this is somewhat ugly, because some things land in a different place, on a different depth. You run into all kinds of problems. Carefully count the dots in imports. Use __dirname vs. process.cwd() to get things relative to the right "root". Run the app with ugly-ish path like node dist/server/server.js. And so on.
How do you typically solve this problem? Would you put compilation results on the same level, in different directory names:
./build-client
./build-server
Something else?
You run into all kinds of problems
Could you enumerate these specifically? I personally don't find this structure problematic.
__dirname
There is no need to use __dirname for imports. If you do an import with a relative path, __dirname is implied. Using the relative path is the idiomatic thing. It's also better as __dirname will be absolute and thus system-specific and thus have deployment/sharing problems.
Although you technically can use this for CommonJS modules, and babel may even support it, ES2015 requires imports to be static strings, so when node eventually supports ES2015 modules natively, anything that isn't a static string will likely be unsupported.
process.cwd()
This cannot be used as it would cause you program to fail most of the time anyone other than you was trying to run it and didn't happen to be in the correct directory. Also this will not work in ES2015.
Overall, my suggestion is
A. If your client and server code are mostly separate, and the relative paths are mostly just ./ and ../ within the directories, it's fine, just adjust to be comfortable with it.
B. If you are commonly importing in between client and server, the code is coupled, and you might reconsider whether client/server is the right axis on which to separate your code. If your app is highly "isomorphic", just restructure you code to reflect the actual coupling that exists across the code and you will naturally minimize the depth of ../../ paths necessary.
My main pain point is really having the output nested in "dist" directory, I wonder how that is usually addressed
You don't need to address this in the source code. Your client side bundler (webpack, browserify, or similar), will use handle third party dependencies from npm automatically using the node require algorithm. If you have a non-npm vendored dependency, just reference that with a relative path and your bundler will handle it.
If you want further help, I suggest you reformat your question from generic to very very specific with the paths and tools and details specified so we can comment directly on your situation instead of something hypothetical/generic.
What is the fundamental difference between bower and npm? Just want something plain and simple. I've seen some of my colleagues use bower and npm interchangeably in their projects.
All package managers have many downsides. You just have to pick which you can live with.
History
npm started out managing node.js modules (that's why packages go into node_modules by default), but it works for the front-end too when combined with Browserify or webpack.
Bower is created solely for the front-end and is optimized with that in mind.
Size of repo
npm is much, much larger than bower, including general purpose JavaScript (like country-data for country information or sorts for sorting functions that is usable on the front end or the back end).
Bower has a much smaller amount of packages.
Handling of styles etc
Bower includes styles etc.
npm is focused on JavaScript. Styles are either downloaded separately or required by something like npm-sass or sass-npm.
Dependency handling
The biggest difference is that npm does nested dependencies (but is flat by default) while Bower requires a flat dependency tree (puts the burden of dependency resolution on the user).
A nested dependency tree means that your dependencies can have their own dependencies which can have their own, and so on. This allows for two modules to require different versions of the same dependency and still work. Note since npm v3, the dependency tree will be flat by default (saving space) and only nest where needed, e.g., if two dependencies need their own version of Underscore.
Some projects use both: they use Bower for front-end packages and npm for developer tools like Yeoman, Grunt, Gulp, JSHint, CoffeeScript, etc.
Resources
Nested Dependencies - Insight into why node_modules works the way it does
This answer is an addition to the answer of Sindre Sorhus. The major difference between npm and Bower is the way they treat recursive dependencies. Note that they can be used together in a single project.
On the npm FAQ: (archive.org link from 6 Sep 2015)
It is much harder to avoid dependency conflicts without nesting
dependencies. This is fundamental to the way that npm works, and has
proven to be an extremely successful approach.
On Bower homepage:
Bower is optimized for the front-end. Bower uses a flat dependency
tree, requiring only one version for each package, reducing page load
to a minimum.
In short, npm aims for stability. Bower aims for minimal resource load. If you draw out the dependency structure, you will see this:
npm:
project root
[node_modules] // default directory for dependencies
-> dependency A
-> dependency B
[node_modules]
-> dependency A
-> dependency C
[node_modules]
-> dependency B
[node_modules]
-> dependency A
-> dependency D
As you can see it installs some dependencies recursively. Dependency A has three installed instances!
Bower:
project root
[bower_components] // default directory for dependencies
-> dependency A
-> dependency B // needs A
-> dependency C // needs B and D
-> dependency D
Here you see that all unique dependencies are on the same level.
So, why bother using npm?
Maybe dependency B requires a different version of dependency A than dependency C. npm installs both versions of this dependency so it will work anyway, but Bower will give you a conflict because it does not like duplication (because loading the same resource on a webpage is very inefficient and costly, also it can give some serious errors). You will have to manually pick which version you want to install. This can have the effect that one of the dependencies will break, but that is something that you will need to fix anyway.
So, the common usage is Bower for the packages that you want to publish on your webpages (e.g. runtime, where you avoid duplication), and use npm for other stuff, like testing, building, optimizing, checking, etc. (e.g. development time, where duplication is of less concern).
Update for npm 3:
npm 3 still does things differently compared to Bower. It will install the dependencies globally, but only for the first version it encounters. The other versions are installed in the tree (the parent module, then node_modules).
[node_modules]
dep A v1.0
dep B v1.0
dep A v1.0 (uses root version)
dep C v1.0
dep A v2.0 (this version is different from the root version, so it will be an nested installation)
For more information, I suggest reading the docs of npm 3
TL;DR: The biggest difference in everyday use isn't nested dependencies... it's the difference between modules and globals.
I think the previous posters have covered well some of the basic distinctions. (npm's use of nested dependencies is indeed very helpful in managing large, complex applications, though I don't think it's the most important distinction.)
I'm surprised, however, that nobody has explicitly explained one of the most fundamental distinctions between Bower and npm. If you read the answers above, you'll see the word 'modules' used often in the context of npm. But it's mentioned casually, as if it might even just be a syntax difference.
But this distinction of modules vs. globals (or modules vs. 'scripts') is possibly the most important difference between Bower and npm. The npm approach of putting everything in modules requires you to change the way you write Javascript for the browser, almost certainly for the better.
The Bower Approach: Global Resources, Like <script> Tags
At root, Bower is about loading plain-old script files. Whatever those script files contain, Bower will load them. Which basically means that Bower is just like including all your scripts in plain-old <script>'s in the <head> of your HTML.
So, same basic approach you're used to, but you get some nice automation conveniences:
You used to need to include JS dependencies in your project repo (while developing), or get them via CDN. Now, you can skip that extra download weight in the repo, and somebody can do a quick bower install and instantly have what they need, locally.
If a Bower dependency then specifies its own dependencies in its bower.json, those'll be downloaded for you as well.
But beyond that, Bower doesn't change how we write javascript. Nothing about what goes inside the files loaded by Bower needs to change at all. In particular, this means that the resources provided in scripts loaded by Bower will (usually, but not always) still be defined as global variables, available from anywhere in the browser execution context.
The npm Approach: Common JS Modules, Explicit Dependency Injection
All code in Node land (and thus all code loaded via npm) is structured as modules (specifically, as an implementation of the CommonJS module format, or now, as an ES6 module). So, if you use NPM to handle browser-side dependencies (via Browserify or something else that does the same job), you'll structure your code the same way Node does.
Smarter people than I have tackled the question of 'Why modules?', but here's a capsule summary:
Anything inside a module is effectively namespaced, meaning it's not a global variable any more, and you can't accidentally reference it without intending to.
Anything inside a module must be intentionally injected into a particular context (usually another module) in order to make use of it
This means you can have multiple versions of the same external dependency (lodash, let's say) in various parts of your application, and they won't collide/conflict. (This happens surprisingly often, because your own code wants to use one version of a dependency, but one of your external dependencies specifies another that conflicts. Or you've got two external dependencies that each want a different version.)
Because all dependencies are manually injected into a particular module, it's very easy to reason about them. You know for a fact: "The only code I need to consider when working on this is what I have intentionally chosen to inject here".
Because even the content of injected modules is encapsulated behind the variable you assign it to, and all code executes inside a limited scope, surprises and collisions become very improbable. It's much, much less likely that something from one of your dependencies will accidentally redefine a global variable without you realizing it, or that you will do so. (It can happen, but you usually have to go out of your way to do it, with something like window.variable. The one accident that still tends to occur is assigning this.variable, not realizing that this is actually window in the current context.)
When you want to test an individual module, you're able to very easily know: exactly what else (dependencies) is affecting the code that runs inside the module? And, because you're explicitly injecting everything, you can easily mock those dependencies.
To me, the use of modules for front-end code boils down to: working in a much narrower context that's easier to reason about and test, and having greater certainty about what's going on.
It only takes about 30 seconds to learn how to use the CommonJS/Node module syntax. Inside a given JS file, which is going to be a module, you first declare any outside dependencies you want to use, like this:
var React = require('react');
Inside the file/module, you do whatever you normally would, and create some object or function that you'll want to expose to outside users, calling it perhaps myModule.
At the end of a file, you export whatever you want to share with the world, like this:
module.exports = myModule;
Then, to use a CommonJS-based workflow in the browser, you'll use tools like Browserify to grab all those individual module files, encapsulate their contents at runtime, and inject them into each other as needed.
AND, since ES6 modules (which you'll likely transpile to ES5 with Babel or similar) are gaining wide acceptance, and work both in the browser or in Node 4.0, we should mention a good overview of those as well.
More about patterns for working with modules in this deck.
EDIT (Feb 2017): Facebook's Yarn is a very important potential replacement/supplement for npm these days: fast, deterministic, offline package-management that builds on what npm gives you. It's worth a look for any JS project, particularly since it's so easy to swap it in/out.
EDIT (May 2019)
"Bower has finally been deprecated. End of story." (h/t: #DanDascalescu, below, for pithy summary.)
And, while Yarn is still active, a lot of the momentum for it shifted back to npm once it adopted some of Yarn's key features.
2017-Oct update
Bower has finally been deprecated. End of story.
Older answer
From Mattias Petter Johansson, JavaScript developer at Spotify:
In almost all cases, it's more appropriate to use Browserify and npm over Bower. It is simply a better packaging solution for front-end apps than Bower is. At Spotify, we use npm to package entire web modules (html, css, js) and it works very well.
Bower brands itself as the package manager for the web. It would be awesome if this was true - a package manager that made my life better as a front-end developer would be awesome. The problem is that Bower offers no specialized tooling for the purpose. It offers NO tooling that I know of that npm doesn't, and especially none that is specifically useful for front-end developers. There is simply no benefit for a front-end developer to use Bower over npm.
We should stop using bower and consolidate around npm. Thankfully, that is what is happening:
With browserify or webpack, it becomes super-easy to concatenate all your modules into big minified files, which is awesome for performance, especially for mobile devices. Not so with Bower, which will require significantly more labor to get the same effect.
npm also offers you the ability to use multiple versions of modules simultaneously. If you have not done much application development, this might initially strike you as a bad thing, but once you've gone through a few bouts of Dependency hell you will realize that having the ability to have multiple versions of one module is a pretty darn great feature. Note that npm includes a very handy dedupe tool that automatically makes sure that you only use two versions of a module if you actually have to - if two modules both can use the same version of one module, they will. But if they can't, you have a very handy out.
(Note that Webpack and rollup are widely regarded to be better than Browserify as of Aug 2016.)
Bower maintains a single version of modules, it only tries to help you select the correct/best one for you.
Javascript dependency management : npm vs bower vs volo?
NPM is better for node modules because there is a module system and you're working locally.
Bower is good for the browser because currently there is only the global scope, and you want to be very selective about the version you work with.
My team moved away from Bower and migrated to npm because:
Programmatic usage was painful
Bower's interface kept changing
Some features, like the url shorthand, are entirely broken
Using both Bower and npm in the same project is painful
Keeping bower.json version field in sync with git tags is painful
Source control != package management
CommonJS support is not straightforward
For more details, see "Why my team uses npm instead of bower".
Found this useful explanation from http://ng-learn.org/2013/11/Bower-vs-npm/
On one hand npm was created to install modules used in a node.js environment, or development tools built using node.js such Karma, lint, minifiers and so on. npm can install modules locally in a project ( by default in node_modules ) or globally to be used by multiple projects. In large projects the way to specify dependencies is by creating a file called package.json which contains a list of dependencies. That list is recognized by npm when you run npm install, which then downloads and installs them for you.
On the other hand bower was created to manage your frontend dependencies. Libraries like jQuery, AngularJS, underscore, etc. Similar to npm it has a file in which you can specify a list of dependencies called bower.json. In this case your frontend dependencies are installed by running bower install which by default installs them in a folder called bower_components.
As you can see, although they perform a similar task they are targeted to a very different set of libraries.
For many people working with node.js, a major benefit of bower is for managing dependencies that are not javascript at all. If they are working with languages that compile to javascript, npm can be used to manage some of their dependencies. however, not all their dependencies are going to be node.js modules. Some of those that compile to javascript may have weird source language specific mangling that makes passing them around compiled to javascript an inelegant option when users are expecting source code.
Not everything in an npm package needs to be user-facing javascript, but for npm library packages, at least some of it should be.