Develop Reference

How to install a npm package from github requiring a build step, e.g. when forking a library?

Assume you use a library like vue3-datepicker. You realize you need to customize something, and as as first step you want to use a custom fork of it.
The issue is, there is a build step when the package is pushed to npm's registry since the project doesn't use plain JavaScript, but may have vue or typescript files.
In this case, that would be npm run build:component, though that depends on the project.
Just installing the fork from github via:
yarn add <GitHub user name>/<GitHub repository name>#<branch/commit/tag>
hence doesn't suffice as then the ./dist folder doesn't exist.
You'll get really strange errors like:
error: [plugin: vite:dep-scan] Failed to resolve entry for package "vue3-datepicker". The package may have incorrect main/module/exports specified in its package.json: Failed to resolve entry for package "vue3-datepicker". The package may have incorrect main/module/exports specified in its package.json.
As a quick and dirty solution, I removed in my fork the ./dist/ folder from the .gitignore, ran the npm i && npm run build:component in my fork, and pushed it.
Huge downside is, the ./dist/ folder is now part of that repository, after each change in my fork I also have to build the files again and push those as well.
I rather have the build process triggered in my application using my fork. Is there a way from my application to say:
When you install that library, you have to run a certain script once you downloaded all the files?
The solution should be usable for both npm and yarn, in the sense that the fork my be installed by either one in different applications.

A quote from npm-install Docs
If the package being installed contains a prepare script, its dependencies and devDependencies will be installed, and the prepare script will be run, before the package is packaged and installed.
so in your fork's package.json you can add
"scripts": {
// ...
"build:component": "rollup -c build/rollup.config.js",
"prepare": "yarn build:component || npm run build:component"
}

If you want to trigger builds after installation, you can use the postinstall or a build script in your package.json. In this script, you can create directories and do other setups, using shell commands or javascript programs:
{
"scripts": {
"build": "mkdir dist && npm run build:component",
"build:component": "some command"
}
}

Running js tests - getting "Cannot use import statement outside of a module"

I added the type: module but that didn't help.
I am trying to run mocha or jest tests that use import and export for the source files.
The existing questions about this have specifics that are different from mine and I also find them confusing to follow for someone with my specific situation, especially since I have developed a specific answer with details not relevant to the existing questions, but relevant to other people in my situation.

In the past, you could not use ES modules (i.e. import/export) in Node without transpiling your code using Babel. However, support for ES modules in Node is now a reality, and both Jest and Mocha have also recently added support for using ES modules natively.
It takes more than just adding "type": "module" to your package.json, however.
Steps for using native ES Modules in Jest
As already mentioned, add "type": "module" to your package.json
Install either jest-environment-node or jest-environment-jsdom-sixteen to your development dependencies. For example:
$ npm i -D jest-environment-node
Update the Jest configuration in package.json and add the testEnvironment setting. For example:
"jest": {
"testEnvironment": "jest-environment-jsdom-node"
}
If you are using a version of Node earlier than 13.2, then you will need to add two additional flags when running Node: --experimental-modules and experimental-vm-modules. I use npx to execute the commands, although it's a little verbose:
$ npx --node-arg=--experimental-modules --node-arg=--experimental-vm-modules jest
This will run all your Jest tests using the appropriate Node flags. I'd recommend making this your test script in package.json if this is the way you go.
Now you should be able to use import/export without Babel!
One last important point: when using native ES modules in Node, you have to use the entire import path to your local modules, including the file extensions. For example:
import lib from "./my/lib.js"
Here is the Node documentation on native ES modules, if you want to read about this in more detail: https://nodejs.org/dist/latest-v14.x/docs/api/esm.html
I'd also recommend reading through this Github issue for more details on the Jest implementation of native ES modules: https://github.com/facebook/jest/issues/9430

Per the Jest documentation, if you make sure babel-jest is installed and supply your Babel configuration per the Babel documentation, in a config file or package.json e.g.:
.babelrc.json
{
"presets": [
"#babel/env"
]
}
Then babel-jest will pick it up automatically and you don't need to explicitly pre-build the files to test them. This also means you don't have to set flags on the command line when calling Babel.
Note that if you do want to explicitly pre-build, I'd recommend:
Using a pre<script> script rather than having multiple steps in one line; and
Re-using the build script so you don't have to make changes in two places.
In your case:
"scripts": {
"build": "babel src/ --out-dir lib",
"pretest": "npm run build",
"test": "jest lib/*.test.js"
}

You need to use a compiler and then use the compiled files when running tests.
Many of the references say to add type: module but don't say much more.
To be clear, the basic message:
SyntaxError: Cannot use import statement outside a module
is because you are using import/export and you are trying to run the files directly without compilation.
The mindshift here is getting used to editing the files in one directory and running the tests in another. Alternatively, some solutions offer "in'flight" compilation so this detail is essentially hidden and only the source files are used.
There are a few different approaches to doing this depending on specific needs. Here is one of the simplest approach I've found so far, using Babel for the compilation step:
Install babel npm install babel --save-dev
Add babel commands in package.json scripts, for example:
"scripts": {
"test": "babel src/ --out-dir lib --presets=#babel/env; jest lib/*.test.js",
"build": "babel src/ --out-dir lib --presets=#babel/env"
}, // this was for jest but you can use mocha, etc as needed
// Note that using preset this way eliminates the need for a specific .babel.config.json file
Now, if you run:
jest .
you get SyntaxError: Cannot use import statement outside a module, but if you run npm t you get
Successfully compiled 2 files with Babel.
PASS lib/app.test.js
All tests
✓ Canary test (2ms)
Test Suites: 1 passed, 1 total
Tests: 1 passed, 1 total
Going forward, just remember to:
Edit files in src/
Run tests in lib/

Update package.json in several places after "npm i"

I want to be able to detect any new package installed according to my package.json file, so when ever I do "npm i", it will automatically added to another section on my package.json file.
For example, if I do "npm i axios", It will update in 2 places on my package.json file:
on "dependencies" as usual, and on new section I created: "extDependencies".
Is there any way to detect any new installed packages?

Check this out: npm-scripts documentation
If you want to run a specific script at a specific lifecycle event for
ALL packages, then you can use a hook script.
Place an executable file at node_modules/.hooks/{eventname}, and it’ll
get run for all packages when they are going through that point in the
package lifecycle for any packages installed in that root.
Hook scripts are run exactly the same way as package.json scripts.
That is, they are in a separate child process, with the env described
above.
You could use this to create a postinstall script (bash, python, node.js, etc) that reads the npm_package_name and npm_package_version environment variables and then use those to update the package.json.

Using create react app, and having ES6 dependencies

I wrote an npm package blokus which use ES6 syntax.
I used create-react-app to start a project web-blokus, which depends on blokus.
I can run npm start with no errors, and view my web-blokus app in my browser, and it has all the functionality from using the blokus package.
The problem is that I get an UglifyJS error when running npm build.
static/js/main.8afd34e2.js from UglifyJs
SyntaxError: Name expected [./~/blokus/blokus/blokus.js:3,0]
It appears there is a known situation with UglifyJS not supporting ES6 dependencies (a few relevant issue threads here and here). But I read through these threads as well as a handful of others and I was left pretty confused as to what was planning to be updated and what people were doing as workarounds.
So I wanted to
1) confirm that create-react-app will not work out of the box (once you go to npm build) if your app has any ES6 dependencies
2) ask what people are doing to fix / bypass the issue (do I have to eject, and swap something in for UglifyJS?)
Since create-react-app and ES6 are now so popular, I assume I'm either misunderstanding the limitation, or a standard method of dealing with this limitation is discussed and known.

You can't use ES6 code with create-react-app, or most build systems.
npm packages shouldn't result in ES6 code because of existing tooling, and to a lesser extent, older node versions.
To set up your package, assuming the ES6 code is in the src directory:
npm install --save-dev babel-core babel-cli babel-preset-latest
.babelrc
{
"presets": ["latest"]
}
package.json
"main": "./lib",
"scripts": {
"build": "babel src --out-dir lib"
}
Then do npm run build before publishing.
Create a .gitignore with 'lib' in it, and a .npmignore that's empty. The .npmignore needs to exist.
You can run your tests on the src directory (or lib, doesn't much matter).

Installing a local module using npm?

I have a downloaded module repo, I want to install it locally, not globally in another directory?
What is an easy way to do this?

you just provide one <folder> argument to npm install, argument should point toward the local folder instead of the package name:
npm install /path

From the npm-link documentation:
In the local module directory:
$ cd ./package-dir
$ npm link
In the directory of the project to use the module:
$ cd ./project-dir
$ npm link package-name
Or in one go using relative paths:
$ cd ./project-dir
$ npm link ../package-dir
This is equivalent to using two commands above under the hood.

Since asked and answered by the same person, I'll add a npm link as an alternative.
from docs:
This is handy for installing your own stuff, so that you can work on it and test it iteratively without having to continually rebuild.
cd ~/projects/node-bloggy # go into the dir of your main project
npm link ../node-redis # link the dir of your dependency
[Edit] As of NPM 2.0, you can declare local dependencies in package.json
"dependencies": {
"bar": "file:../foo/bar"
}

npm pack + package.json
This is what worked for me:
STEP 1: In module project, execute npm pack:
This will build a <package-name>-<version>.tar.gz file.
STEP 2: Move the file to the consumer project
Ideally you can put all such files in a tmp folder in your consumer-project root:
STEP 3: Refer it in your package.json:
"dependencies": {
"my-package": "file:/./tmp/my-package-1.3.3.tar.gz"
}
STEP 4: Install the packages:
npm install or npm i or yarn
Now, your package would be available in your consumer-project's node_modules folder.
Good Luck...

Neither of these approaches (npm link or package.json file dependency) work if the local module has peer dependencies that you only want to install in your project's scope.
For example:
/local/mymodule/package.json:
"name": "mymodule",
"peerDependencies":
{
"foo": "^2.5"
}
/dev/myproject/package.json:
"dependencies":
{
"mymodule": "file:/local/mymodule",
"foo": "^2.5"
}
In this scenario, npm sets up myproject's node_modules/ like this:
/dev/myproject/node_modules/
foo/
mymodule -> /local/mymodule
When node loads mymodule and it does require('foo'), node resolves the mymodule symlink, and then only looks in /local/mymodule/node_modules/ (and its ancestors) for foo, which it doen't find. Instead, we want node to look in /local/myproject/node_modules/, since that's where were running our project from, and where foo is installed.
So, we either need a way to tell node to not resolve this symlink when looking for foo, or we need a way to tell npm to install a copy of mymodule when the file dependency syntax is used in package.json. I haven't found a way to do either, unfortunately :(

Missing the main property?
As previous people have answered npm i --save ../location-of-your-packages-root-directory.
The ../location-of-your-packages-root-directory however must have two things in order for it to work.
package.json in that directory pointed towards
main property in the package.json must be set and working i.g. "main": "src/index.js", if the entry file for ../location-of-your-packages-root-directory is ../location-of-your-packages-root-directory/src/index.js

So I had a lot of problems with all of the solutions mentioned so far...
I have a local package that I want to always reference (rather than npm link) because it won't be used outside of this project (for now) and also won't be uploaded to an npm repository for wide use as of yet.
I also need it to work on Windows AND Unix, so sym-links aren't ideal.
Pointing to the tar.gz result of (npm package) works for the dependent npm package folder, however this causes issues with the npm cache if you want to update the package. It doesn't always pull in the new one from the referenced npm package when you update it, even if you blow away node_modules and re-do your npm-install for your main project.
so.. This is what worked well for me!
Main Project's Package.json File Snippet:
"name": "main-project-name",
"version": "0.0.0",
"scripts": {
"ng": "ng",
...
"preinstall": "cd ../some-npm-package-angular && npm install && npm run build"
},
"private": true,
"dependencies": {
...
"#com/some-npm-package-angular": "file:../some-npm-package-angular/dist",
...
}
This achieves 3 things:
Avoids the common error (at least with angular npm projects) "index.ts is not part of the compilation." - as it points to the built (dist) folder.
Adds a preinstall step to build the referenced npm client package to make sure the dist folder of our dependent package is built.
Avoids issues where referencing a tar.gz file locally may be cached by npm and not updated in the main project without lots of cleaning/troubleshooting/re-building/re-installing.
I hope this is clear, and helps someone out.
The tar.gz approach also sort of works..
npm install (file path) also sort of works.
This was all based off of a generated client from an openapi spec that we wanted to keep in a separate location (rather than using copy-pasta for individual files)
======
UPDATE:
======
There are additional errors with a regular development flow with the above solution, as npm's versioning scheme with local files is absolutely terrible. If your dependent package changes frequently, this whole scheme breaks because npm will cache your last version of the project and then blow up when the SHA hash doesn't match anymore with what was saved in your package-lock.json file, among other issues.
As a result, I recommend using the *.tgz approach with a version update for each change. This works by doing three things.
First:
For your dependent package, use the npm library "ng-packagr". This is automatically added to auto-generated client packages created by the angular-typescript code generator for OpenAPI 3.0.
As a result the project that I'm referencing has a "scripts" section within package.json that looks like this:
"scripts": {
"build": "ng-packagr -p ng-package.json",
"package": "npm install && npm run build && cd dist && npm pack"
},
And the project referencing this other project adds a pre-install step to make sure the dependent project is up to date and rebuilt before building itself:
"scripts": {
"preinstall": "npm run clean && cd ../some-npm-package-angular && npm run package"
},
Second
Reference the built tgz npm package from your main project!
"dependencies": {
"#com/some-npm-package-angular": "file:../some-npm-package-angular/dist/some-npm-package-angular-<packageVersion>.tgz",
...
}
Third
Update the dependent package's version EVERY TIME you update the dependent package. You'll also have to update the version in the main project.
If you do not do this, NPM will choke and use a cached version and explode when the SHA hash doesn't match. NPM versions file-based packages based on the filename changing. It won't check the package itself for an updated version in package.json, and the NPM team stated that they will not fix this, but people keep raising the issue: https://github.com/microsoft/WSL/issues/348
for now, just update the:
"version": "1.0.0-build5",
In the dependent package's package.json file, then update your reference to it in the main project to reference the new filename, ex:
"dependencies": {
"#com/some-npm-package-angular": "file:../some-npm-package-angular/dist/some-npm-package-angular-1.0.0-build5.tgz",
...
}
You get used to it. Just update the two package.json files - version then the ref to the new filename.
Hope that helps someone...

I came across different solution than above while installing custom build package for CKEditor5.
So I uploaded package to app root directory, than:
npm add file:./ckeditor5
In my package.json package is listed as a file:
"ckeditor5-custom-build": "file:ckeditor5",
I think this answer could be relevant to the topic on how to add local package.

For installing local module / package, that not yet on npm or you are developing an npm package and want to test it locally before publishing it. You can try this -
npm i yalc -g
Go to the module/package folder then -
yalc publish
Your packakge is ready to use, now go the project you want to install it -
yalc add <Your package name>
Package will be installed to you project. If you want to remove it -
yalc remove <Your package name>

For more recent versions of npm (I'm using 8.1.3 under macOS Big Sur), the sequence of commands is even easier...
cd /path-where-your-local-project-is/
npm init
This will ask you for some data related to your project and properly initialises your project.json file.
Once that is done, you can install additional modules with:
cd /path-where-your-local-project-is/
npm install --save-dev some-npm-module .
That's all you need!
Note: I believe that the trailing dot is not necessary if you're inside the project directory, but I also think that it doesn't hurt to add it :-)
(I wonder why the official docs still don't explain this...)