Is there a way to create a meta-package, which package.json:
a) contains a list of dependencies, which are not used to build this
package;
b) enforces NPM to install these deps, when this very
meta-package is added to the project, both for production for the development?
The problem is that I have a few dozens of packages (both, prod and dev) in my boilerplate, which I copy from project to project. I would like to have one private package, which once added to project's package.json tells npm which other prod and dev deps to install. I want to improve the process of using the boilerplate, because when I change something it it, I have (actually that is my will, not the enforcement) to make these changes in all the projects built on the base of this boilerplate and which are still in development.
I have managed to add all the deps I usually use to production section of my meta-package's package.json. However there are two problems with this approach:
1. A lot of unnecessary packages are installed when I build the meta-package (I have to build it as it contains some helpers in ES6/7);
2. I still have to copy/paste the list of production deps into every new project to provide NPM information, what to install on the production server.
Non-npm solutions are welcome as well.
Well, it appeared to be quite easy to do:
1. First of all, it is required to split all the packages used in the boilerplate into two groups - required to run the app (production), and required to build and debug (development).
2. Create the production meta-package (meta-prod). package.json content:
...
"dependencies": {
< Here we have the list of production dependencies. >
},
...
Compile it if necessary and push to git (i.e., github).
3. Create the development meta-package (meta-dev). package.json content:
...
"dependencies": {
< Here we have the list of development dependencies. >
< Please note, that this is in the 'dependencies' section, >
< not the 'devDependencies'. >
},
...
Push it to git.
4. Remove all the dependencies from the boilerplate's package.json and replace it with:
...
"dependencies": {
"meta-prod": "git+https://github.com/your_account/meta-prod"
},
"devDependencies": {
"meta-dev": "git+https://github.com/your_account/meta-dev"
},
...
That's it! Dependencies will be installed when you do npm install and will be properly considered by npm as production and development. Now you can use much smaller package.json boilerplate and update the list of your devs from the single point.
The only downside is that npm does not catch the version change when you use package from git, so to update packages within the app you have to remove meta-packages directories from node_modules and rerun npm install. To fix this you can publish your package to npmjs.com (better to publish it as private package if you are the paid npm customer) and change meta-package version every time you update it.
Here is the boilerplate and meta-packages example:
app boilerplate
meta-prod
meta-dev
Related
Assume you use a library like vue3-datepicker. You realize you need to customize something, and as as first step you want to use a custom fork of it.
The issue is, there is a build step when the package is pushed to npm's registry since the project doesn't use plain JavaScript, but may have vue or typescript files.
In this case, that would be npm run build:component, though that depends on the project.
Just installing the fork from github via:
yarn add <GitHub user name>/<GitHub repository name>#<branch/commit/tag>
hence doesn't suffice as then the ./dist folder doesn't exist.
You'll get really strange errors like:
error: [plugin: vite:dep-scan] Failed to resolve entry for package "vue3-datepicker". The package may have incorrect main/module/exports specified in its package.json: Failed to resolve entry for package "vue3-datepicker". The package may have incorrect main/module/exports specified in its package.json.
As a quick and dirty solution, I removed in my fork the ./dist/ folder from the .gitignore, ran the npm i && npm run build:component in my fork, and pushed it.
Huge downside is, the ./dist/ folder is now part of that repository, after each change in my fork I also have to build the files again and push those as well.
I rather have the build process triggered in my application using my fork. Is there a way from my application to say:
When you install that library, you have to run a certain script once you downloaded all the files?
The solution should be usable for both npm and yarn, in the sense that the fork my be installed by either one in different applications.
A quote from npm-install Docs
If the package being installed contains a prepare script, its dependencies and devDependencies will be installed, and the prepare script will be run, before the package is packaged and installed.
so in your fork's package.json you can add
"scripts": {
// ...
"build:component": "rollup -c build/rollup.config.js",
"prepare": "yarn build:component || npm run build:component"
}
If you want to trigger builds after installation, you can use the postinstall or a build script in your package.json. In this script, you can create directories and do other setups, using shell commands or javascript programs:
{
"scripts": {
"build": "mkdir dist && npm run build:component",
"build:component": "some command"
}
}
I am trying to install jquery and jquery-slimscroll packages to an Angular project (version > 10), but the packages are not seems to be installed properly. So, regarding to this issue, could you please clarify me about the following issues below?
Assume that I have the following client structure:
ClientApp ---
|--- node_modules
|--- package.json
|--- angular.json
1. What is the proper way to add jquery and jquery-slimscroll to the project?
Should I run npm install on the ClientApp folder? Or is there a better way for the latest Angular versions?
2. I think there are several ways to add these packages as script like the following:
in angular.json:
"scripts": [
".../jquery/dist/jquery.min.js",
".../jquery-slimscroll/jquery.slimscroll.js",
]
Is it not a good approach to add them like this? Should I add them in npm as in the 1st question?
3. What about the packages.json? Some packages are added to packages.json instead of angular.json. What is the difference and why some packages are added another?
First of all, you have to understand the function of angular.json and package.json.
Angular.json
A file named angular.json at the root level of an Angular workspace
provides workspace-wide and project-specific configuration defaults
for build and development tools provided by the Angular CLI. Path
values given in the configuration are relative to the root workspace
folder.
Package.json
Initially, this package.json includes a starter set of packages, some
of which are required by Angular and others that support common
application scenarios. You add packages to package.json as your
application evolves. You may even remove some.
The package.json is organized into two groups of packages:
Dependencies are essential to running applications. DevDependencies
are only necessary to develop applications.
So, for the questions of the proper way to add these jquery dependencies, you can do both, the npm install jquery or add to the package.json and if needed to the angular.json.
The difference of both ways is that if you use npm install jquery it will add the dependencies automatically and download the packages to your node_modules, you add it manually, you will need to run npm install anyway, tp get the packages.
I'm trying to figure out correct approach for a javascript monorepo. Imagine monorepo containing packages / libraries:
root
- node_modules
- packages
+ lib-a
* node_modules
+ lib-b
* node_modules
Now let's say both lib-a and lib-b packages use webpack as their build tool.
I see two approaches
Add wepback as dependency to root. Include "build" script in both packages: "build": "webpack -p --config webpack.config.js. webpack.config.js could include root webpack.config.js. Then I could use tool like lerna to run the build from root directory (which means webpack binary is recognized. However I will be unable to run the build in specific packages since webpack is not available there. I could probably change the build script to something like "build": "../../node_modules/.bin/webpack -p --config webpack.config.js
Always include webpack in each package. This means that build script will succeed. This also means that each package will have the same dependency and I should probably watch that each package uses same webpack version.
Basically what I'm getting at is how should packages inside monorepo be structured? If any package is published, should it always be possible to build that package separately.
Your approach #2 is right. You handle each package separately as it was an individual, self-contained package.
The advantage of a monorepo lays not in sharing files through the directory structure but in:
Bootstrapping all dependencies to a single node_modules with flat structure, effectively deduplicating them.
Making your packages available to your other packages through regular package import/require() as they were external dependencies. And, thanks to symlinks to node_modules, your "dependency" packages contain always the latest content without publishing.
Enforcing consistent, always up-to-date, dependency structure in all your packages. As you said "This also means that each package will have the same dependency".
Automation tools to perform different maintainance tasks (like build, publish) on all your packages with a single command.
I know it's not so easy at the beginning, but when you dig into Lerna documentation it's becoming more clear. Besides Lerna main page I recommend reading about hoisting, FAQ and individual commands like bootstrap and publish.
Our current configuration is same as you:
root
- node_modules
- packages
+ lib-a
* node_modules
+ lib-b
* node_modules
We use lerna to handle our project: https://github.com/lerna/lerna
You just need to specify your package folder in the lerna.json
{
"lerna": "3.16.4",
"packages": ["packages/*"],
"version": "0.0.0",
"npmClient": "yarn",
"useWorkspaces": true
}
Then in your package.json scripts you can use the line:
"build": "lerna run build",
This will basically run a build in all packages. So as long as your build script in each package has the proper params and webpack installed it will automatically run the webpack build.
After that you can simply handle working in your designated packages.
I’m building a node.js app that needs to be scalable and maintainable. The idea is to have one repository, but with more than one module inside it.
We are using local modules with npm, but we have a problem is that when we update a modules npm won’t update the dependent services.
here an example of the package.json
// ./app/package.json
{
"dependencies": {
"error-lib": "file:../modules/error-lib"
},
"scripts": {
"start": "node app.js",
"debug": "node --inspect-brk app.js",
},
"main": "app.js",
}
// ../modules/error-lib/pacjake.json
{
"name": "error-lib",
"version": "1.0.0",
"main": "index.ts"
}
I have one problem when a developer updates the error-lib module, an npm install command won't update the old error-lib inside app/node_modules.
I don't want to bump the version of the modules on every change because they follow the same versioning and release lifecycle.
And I don't want to rm -rf modules manually every time.
Any idea on how to automate a repo with many local modules?
TL;DR (Why one repo, many local modules)
We came from a Java background, where we have a single repo multi-module architecture managed by maven.
What we do in maven is to isolate services, common-library, data model in isolated modules.
We have only one repo for all the modules and service since everyone follows the same release cycle with a uniform versioning.
When a developer needs to apply some change he or she can update one of the modules or the services and maven will take care of every dependency and the code will be up to date and ready to rock.
But we want separate modules for many reasons:
This will force us to decouple the functionality and make code reusable since every module can virtually be placed in a separate repo at any time.
If some module will become usable outside of the repo can be externalized and maintained.
We want to achieve the same design for a new part in node.js
Looks like you need $ npm link instead, creating symlinks to your packages within ./app/node_modules. This way, the linked packages are always 'up-to-date'.
Linking a package is a 2-step process:
$ npm link at ../modules/error-lib creates a symbolic package in your global node_modules folder under the name error-lib, specified in ../modules/error-lib/package.json its "name" value.
$ npm link error-lib at ./app to create the second link inside ./app/node_modules to <global-node_modules>/error-lib.
more about npm link
Edit in response to your comment:
I've seen some drawbacks of npm link:
the dependency is not listed in the app package.json, so how I know which dependency I'm linking?
the node_modules is not committed on the repo, so the CI/CD will need to do the linking while listing modules/error-lib inside the app package.json I have a place to take track of the dependencies.
There is any best practice on how to automate npm link or to mitigate the drawbacks?
NPM packages are supposed to be distributed through the npm registry. This means that your error-lib package also should be distributed this way, and be installed as any other package. This will require you to list your package in the package.json file in advance, which is essentially what you want.
If you instead intend to keep everything in one place there is no use in using NPM at all. Just resolve your packages with relative import paths in your software.
Since you said that all the packages follow the same release cycle as your main software, and they live in the same repo, why would you resolve these packages with NPM while they seem more like just part of the software?
error-lib is either a package, or it is not. If it is, it ideally has its own CI/CD pipeline, as large or tiny as required, involving testing, building and distribution. This pipeline is then to be run before the pipeline of your dependent repositories.
I have a downloaded module repo, I want to install it locally, not globally in another directory?
What is an easy way to do this?
you just provide one <folder> argument to npm install, argument should point toward the local folder instead of the package name:
npm install /path
From the npm-link documentation:
In the local module directory:
$ cd ./package-dir
$ npm link
In the directory of the project to use the module:
$ cd ./project-dir
$ npm link package-name
Or in one go using relative paths:
$ cd ./project-dir
$ npm link ../package-dir
This is equivalent to using two commands above under the hood.
Since asked and answered by the same person, I'll add a npm link as an alternative.
from docs:
This is handy for installing your own stuff, so that you can work on it and test it iteratively without having to continually rebuild.
cd ~/projects/node-bloggy # go into the dir of your main project
npm link ../node-redis # link the dir of your dependency
[Edit] As of NPM 2.0, you can declare local dependencies in package.json
"dependencies": {
"bar": "file:../foo/bar"
}
npm pack + package.json
This is what worked for me:
STEP 1: In module project, execute npm pack:
This will build a <package-name>-<version>.tar.gz file.
STEP 2: Move the file to the consumer project
Ideally you can put all such files in a tmp folder in your consumer-project root:
STEP 3: Refer it in your package.json:
"dependencies": {
"my-package": "file:/./tmp/my-package-1.3.3.tar.gz"
}
STEP 4: Install the packages:
npm install or npm i or yarn
Now, your package would be available in your consumer-project's node_modules folder.
Good Luck...
Neither of these approaches (npm link or package.json file dependency) work if the local module has peer dependencies that you only want to install in your project's scope.
For example:
/local/mymodule/package.json:
"name": "mymodule",
"peerDependencies":
{
"foo": "^2.5"
}
/dev/myproject/package.json:
"dependencies":
{
"mymodule": "file:/local/mymodule",
"foo": "^2.5"
}
In this scenario, npm sets up myproject's node_modules/ like this:
/dev/myproject/node_modules/
foo/
mymodule -> /local/mymodule
When node loads mymodule and it does require('foo'), node resolves the mymodule symlink, and then only looks in /local/mymodule/node_modules/ (and its ancestors) for foo, which it doen't find. Instead, we want node to look in /local/myproject/node_modules/, since that's where were running our project from, and where foo is installed.
So, we either need a way to tell node to not resolve this symlink when looking for foo, or we need a way to tell npm to install a copy of mymodule when the file dependency syntax is used in package.json. I haven't found a way to do either, unfortunately :(
Missing the main property?
As previous people have answered npm i --save ../location-of-your-packages-root-directory.
The ../location-of-your-packages-root-directory however must have two things in order for it to work.
package.json in that directory pointed towards
main property in the package.json must be set and working i.g. "main": "src/index.js", if the entry file for ../location-of-your-packages-root-directory is ../location-of-your-packages-root-directory/src/index.js
So I had a lot of problems with all of the solutions mentioned so far...
I have a local package that I want to always reference (rather than npm link) because it won't be used outside of this project (for now) and also won't be uploaded to an npm repository for wide use as of yet.
I also need it to work on Windows AND Unix, so sym-links aren't ideal.
Pointing to the tar.gz result of (npm package) works for the dependent npm package folder, however this causes issues with the npm cache if you want to update the package. It doesn't always pull in the new one from the referenced npm package when you update it, even if you blow away node_modules and re-do your npm-install for your main project.
so.. This is what worked well for me!
Main Project's Package.json File Snippet:
"name": "main-project-name",
"version": "0.0.0",
"scripts": {
"ng": "ng",
...
"preinstall": "cd ../some-npm-package-angular && npm install && npm run build"
},
"private": true,
"dependencies": {
...
"#com/some-npm-package-angular": "file:../some-npm-package-angular/dist",
...
}
This achieves 3 things:
Avoids the common error (at least with angular npm projects) "index.ts is not part of the compilation." - as it points to the built (dist) folder.
Adds a preinstall step to build the referenced npm client package to make sure the dist folder of our dependent package is built.
Avoids issues where referencing a tar.gz file locally may be cached by npm and not updated in the main project without lots of cleaning/troubleshooting/re-building/re-installing.
I hope this is clear, and helps someone out.
The tar.gz approach also sort of works..
npm install (file path) also sort of works.
This was all based off of a generated client from an openapi spec that we wanted to keep in a separate location (rather than using copy-pasta for individual files)
======
UPDATE:
======
There are additional errors with a regular development flow with the above solution, as npm's versioning scheme with local files is absolutely terrible. If your dependent package changes frequently, this whole scheme breaks because npm will cache your last version of the project and then blow up when the SHA hash doesn't match anymore with what was saved in your package-lock.json file, among other issues.
As a result, I recommend using the *.tgz approach with a version update for each change. This works by doing three things.
First:
For your dependent package, use the npm library "ng-packagr". This is automatically added to auto-generated client packages created by the angular-typescript code generator for OpenAPI 3.0.
As a result the project that I'm referencing has a "scripts" section within package.json that looks like this:
"scripts": {
"build": "ng-packagr -p ng-package.json",
"package": "npm install && npm run build && cd dist && npm pack"
},
And the project referencing this other project adds a pre-install step to make sure the dependent project is up to date and rebuilt before building itself:
"scripts": {
"preinstall": "npm run clean && cd ../some-npm-package-angular && npm run package"
},
Second
Reference the built tgz npm package from your main project!
"dependencies": {
"#com/some-npm-package-angular": "file:../some-npm-package-angular/dist/some-npm-package-angular-<packageVersion>.tgz",
...
}
Third
Update the dependent package's version EVERY TIME you update the dependent package. You'll also have to update the version in the main project.
If you do not do this, NPM will choke and use a cached version and explode when the SHA hash doesn't match. NPM versions file-based packages based on the filename changing. It won't check the package itself for an updated version in package.json, and the NPM team stated that they will not fix this, but people keep raising the issue: https://github.com/microsoft/WSL/issues/348
for now, just update the:
"version": "1.0.0-build5",
In the dependent package's package.json file, then update your reference to it in the main project to reference the new filename, ex:
"dependencies": {
"#com/some-npm-package-angular": "file:../some-npm-package-angular/dist/some-npm-package-angular-1.0.0-build5.tgz",
...
}
You get used to it. Just update the two package.json files - version then the ref to the new filename.
Hope that helps someone...
I came across different solution than above while installing custom build package for CKEditor5.
So I uploaded package to app root directory, than:
npm add file:./ckeditor5
In my package.json package is listed as a file:
"ckeditor5-custom-build": "file:ckeditor5",
I think this answer could be relevant to the topic on how to add local package.
For installing local module / package, that not yet on npm or you are developing an npm package and want to test it locally before publishing it. You can try this -
npm i yalc -g
Go to the module/package folder then -
yalc publish
Your packakge is ready to use, now go the project you want to install it -
yalc add <Your package name>
Package will be installed to you project. If you want to remove it -
yalc remove <Your package name>
For more recent versions of npm (I'm using 8.1.3 under macOS Big Sur), the sequence of commands is even easier...
cd /path-where-your-local-project-is/
npm init
This will ask you for some data related to your project and properly initialises your project.json file.
Once that is done, you can install additional modules with:
cd /path-where-your-local-project-is/
npm install --save-dev some-npm-module .
That's all you need!
Note: I believe that the trailing dot is not necessary if you're inside the project directory, but I also think that it doesn't hurt to add it :-)
(I wonder why the official docs still don't explain this...)