Develop Reference

How to install npm dependencies which aren't in package.json?

I have problem with npm install. I want to install all dependencies used in my project. In package.json there aren't any dependencies, but in my project files I have for example const mongo=require('mongoose') and in another file const morgan=require('morgan') etc.
I know that when typing npm i --save <dependency_name> it will update my package.json. But I would like to install all dependencies without typing their names explicitly anywhere.
Is there any way to install all dependencies used in the whole project, but which there aren't in package.json?

It is not intended to write your program code first and then let NPM find all dependencies that are used. This is because of a simple reason: Before you can use an external library / package, you'll have to download it, otherwise you could hardly use it at all.
Now, I only see two reasons for your use case:
you copied and pasted some foreign code where you don't have all files, especially no package.json, or
you "inherited" some code of a former employee or team that is not documented and consists of unrelated, inconsistent files.
Anyways, there would be some kinds of solutions for your problem:
The nasty solution: Run npm start and see what error messages you get. All uninstalled dependencies will not be found, so you'll see the package names and can add them manually to your package.json. This will become nasty because you'll have to re-run your program each time. To avoid this, you could have a look at Nodemon which automatically re-starts your program.
The better solution: Open a web IDE of your favor and use the "Global Search" function to find all occurrences of the string require(, or as a Regex: require\((.+)\). This will list you all dependency imports in your program files. If there should be also ECMA 6 imports, also search for import (.+) from (.+). - However, you'll still have to copy and paste all dependency names from all files manually into your package.json file.
The best but most complex solution: Write a Node.js script that scans all your files recursively, beginning in your root project directory. Create a dependency memory variable, like let dependencies = []. Read all *.js files (sync or async) and every time the require or import statement is matched, check whether the dependency is already in your dependencies array. If not, push it. Finally, all your project dependencies will be listed in the dependencies array and you can copy and paste them into your package.json.
Pseudo Node.js Code:
const lineReader = require('line-reader');
let dependencies = [];
const regex = /require\(['|"](.+)['|"]\)/g;
lineReader.eachLine('/path/to/file', function(line) {
const match = regex.exec(line);
if(match) {
if(dependencies.indexOf(match[1]) === -1) {
dependencies.push(match[1]);
}
}
});

No, you can't do that - you will have to add all dependencies by installing them explicitly, like npm install morgan. NPM is not aware of the dependencies you're importing in your files. Another thing is that requiring dependencies that are not listed in package.json is simply wrong and should never take place.

Short answer: You can't
How npm install works is checking all your dependencies listed in the package.json and install them by once. So either you get the package.json from the tutor or you install them one by one

You can scan your project for all required modules.
Assuming your project only uses common.js (require) you can get a list of all modules by doing something like:
egrep -R --exclude-dir=node_modules '=\s*require\s*\(' | awk '{gsub(/^.+require\s*\(\s*./,""); gsub(/.\s*\).*$/,""); print $0}'
A more readable version of the above command is:
#! /bin/bash
egrep -R --exclude-dir=node_modules '=\s*require\s*\(' |
awk '{
gsub(/^.+require\s*\(\s*./,"");
gsub(/.\s*\).*$/,"");
print $0
}'
You can save the script above in a file and execute it as a shell script.
To automatically install the modules just pipe it to xargs:
egrep -R --exclude-dir=node_modules '=\s*require\s*\(' |
awk '{
gsub(/^.+require\s*\(\s*./,"");
gsub(/.\s*\).*$/,"");
print $0
}' |
xargs npm install
I leave supporting ES6 module as homework for the reader.

Why the same package are installed in a different way?

I installed the same package for both my projects. That package(won't link, it's private one) has react-popper as dependency(which in order has create-react-context as dependency), so, when I run project one - everything is ok, but error appears for project two:
ERROR in ./node_modules/react-popper/lib/esm/Manager.js
Module not found: Error: Can't resolve 'create-react-context' in '/../node_modules/react-popper/lib/esm'
After some investigation, I got that node_modules structure is different:
for project one, all react-popper dependencies are saved in project one
node modules, and local folder contain only warning package:
for project two all react-popper dependencies saved in ../react-popper/node_modules local folder:
I've tried some common approaches like reinstalling node modules, clear cache and so on, but the structure is the same. Actually I had a thoughts about webpack and babel versions, but I don't think it can affect node_modules structure itself.
So the question is, which factors can affect it? What should I check?
NOTE: If I manually add create-react-context to project two, it works fine, but it's not a solution.
NOTE: I found out similar issue, but no suggestions there - Why does npm install packages in different directories?, im my case re-creating of yarn.lock also helps, but it's also doesn't look like a right way to solve it. Hope my description is more complete and will help to figure it out.

This is very likely because of the way yarn (as well as npm) tries to deduplicate dependencies. Let's say there are modules A and B which exist in 2 versions (1.0.0 and 2.0.0). B depends on Version 1.0.0 of Module A.
If you install only module B, you will get a node_modules folder like this:
node_modules
- A#1.0.0
- B#2.0.0
But what if you install module A in its latest version (2.0.0)? If npm just updated the version of module A, your existing module B would (potentially) no longer work as it depends on module A. So your node_modules folder will instead look like this (A#1.0.0 is moved inside B's node_modules folder)
node_modules
- A#2.0.0
- B#2.0.0
-- A#1.0.0
Your 2 projects likely have further dependencies, which somehow overlap with react-popper or its dependencies. Due to the nodeJs module resolution mechanism, this usually shouldn't be a problem.
TLDR: The exact structure of the node_modules folder depends on all your dependencies (and devDependencies). yarn/npm will look into every package.json/package-lock.json file of your projects dependencies (and their dependencies) and use this information to calculate a dependency tree with minimal duplication.

How to verify an object instance? instanceof and ....prototype.isPrototypeOf(...) are not reliable [duplicate]

Whenever I make projects, I have to download all dependencies of node modules. Without copying the node_modules, Is there anyway to share the central node_modules in multiple projects?
like the followings, I have to run many commands every time..
npm install gulp-usemin
npm install gulp-wrap
npm install gulp-connect
npm install gulp-watch
npm install gulp-minify-css
npm install gulp-uglify
npm install gulp-concat
npm install gulp-less
npm install gulp-rename
npm install gulp-minify-html

You absolutely can share a node_modules directory amongst projects.
From node's documentation:
If the module identifier passed to require() is not a native module,
and does not begin with '/', '../', or './', then node starts at the
parent directory of the current module, and adds /node_modules, and
attempts to load the module from that location.
If it is not found there, then it moves to the parent directory, and
so on, until the root of the file system is reached.
For example, if the file at '/home/ry/projects/foo.js' called
require('bar.js'), then node would look in the following locations, in
this order:
/home/ry/projects/node_modules/bar.js /home/ry/node_modules/bar.js
/home/node_modules/bar.js /node_modules/bar.js
So just put a node_modules folder inside your projects directory and put in whatever modules you want. Just require them like normal. When node doesn't find a node_modules directory in your project folder, it will check the parent folder automatically. So make your directory structure like this:
-myProjects
--node_modules
--myproject1
---sub-project
--myproject2
So like this, even your sub-project's dependencies can draw on your main node_modules repository.
One drawback to doing it this way is you will have to build out your package.json file manually (unless someone knows a way to automate this with grunt or something). When you install your packages and add the --save arg to an npm install command it automatically appends it to the dependencies section or your package.json, which is convenient.

Try pnpm instead of npm.
pnpm uses hard links and symlinks to save one version of a module only ever once on a disk.
If you have npm installed, you can install in your terminal with:
npm install -g pnpm
To update your existing installations (and sub-directories) use:
pnpm recursive install
Or use the shorthand command (leave off -r if you need to target only one directory)
pnpm -r i
One helpful note: You may find some rare packages don't have all their dependencies defined. They might rely on the flat node_modules file directory structure of npm or yarn installs. If you run into issues of missing dependencies, use this command to hoist all the sub dependencies into a flat-file structure:
pnpm install --shamefully-hoist
It's best to avoid using the --shamefully-hoist flag as it defeats the purpose of using pnpm in the first place, so try using the command pnpm i your-missing-package first (See pnpm FAQ).

I found a trick, just take a look at the Symbolic Links (symlinks) on Windows or Linux, it is working just like shortcuts but more powerful.
Simply you need to make a Junction for your node_modules folder anywhere you want. The junction is nothing but a short cut to your original node_modules folder. Create it inside your project folder where the actual node_modules would have been created if used npm install.
To achieve this you need at least one node_modules real folder then make a Junction to it in the other projects.
On Windows, you can either use the Command Prompt, or use an application. Using the Command Prompt gives you a bit more control, using an application is easier I suggest Link Shell Extension.

Main directory should look like this
node_modules
Project 1
Project 2
Project 3
Project 4
just open the file Project 1/.angular-cli.json
change the schema
"$schema": "./node_modules/#angular/cli/lib/config/schema.json",
to
"$schema": "./../node_modules/#angular/cli/lib/config/schema.json"
and don't forget to create node_modules empty folder inside your project directory

See also npm v7.0.0's support for workspaces
RFC
https://github.com/npm/rfcs/blob/latest/implemented/0026-workspaces.md
Documentation
https://docs.npmjs.com/cli/v7/using-npm/workspaces

By looking at some articles it seems that Lerna
is a good tool for managing multiple projects inside a single directory (monorepo). It supports modules sharing without duplicating the entire packages in every folder and commands to install them in multiple projects.
Javascript monorepos
Monorepos by example
Building large scale apps in a monorepo
pnpm is also a simple and efficient tool, which doesn't duplicate those modules which are already installed for other projects.

Let's assume that having a single node_modules it should contain all the packages for all applications. thus your apps will also share most of the unique package.json entries (just the name should change)
my idea would be to have a single root and multiple src level as below
root\package.json
root\node_modules
root\\..
root\app1\src\\..
root\app2\src\\..
the only issue you might face would be having a backup of json (or tsconfig) for any app and restore them when you work on it or setup your startup scripts to serve any app

Change Directory in Node REPL not working?

Just earlier, I posted my question:
https://stackoverflow.com/questions/28336443/how-to-not-put-my-js-files-in-user-myuser-for-node-js
I have a file, hello.js, located in /Users/MyUser/Desktop/Node/
I can see that my default directory is /Users/MyUser/
Okay, so I get that I need to change my working directory. What I have been able to find so far is to use >process.chrdir('/Users/MyUser/Desktop/Node/');
Cool, that works, but now when I get out of the REPL shell, the directory resets.
The person who responded to my question said that I needed to run >node init and later npm install <name of dependency> --save
My first question: I have ran >node init and see that I can create this package.json file, what does this do exactly?
Secondly: I was told that I need to add dependancies. Could someone please explain to me what this means in Node terms? Does a dependancy simply mean a folder that I want node to include? Do I want to add this Node folder on my Desktop to be able to run my scripts?
I am currently trying to go through the learnyounode courses, however I do not want to have to save all of these test files in my /User/MyUser directory, so any advice would be greatly appreciated.
Thanks

I have ran >node init and see that I can create
this package.json file, what does this do exactly?
npm init is used to create a package.json file interactively. This will ask you a bunch of questions, and then write a package.json for you.
package.json is just a file that handle the project's dependencies and holds various metadata relevant to the project[ project description, version, license information etc]
I was told that I need to add dependencies. Could someone please
explain to me what this means in Node terms?
Lets say you're building an application that is dependent on a number of NPM modules, you can specify them in your package.json file this way:
"dependencies": {
"express": "2.3.12",
"jade": ">= 0.0.1",
"redis": "0.6.0"
}
Now doing npm install would install a package, and any packages that it depends on.
A package is:
a folder containing a program described by a package.json file
a gzipped tarball containing (1)
a url that resolves to (2)
a # that is published on the registry with (3)
a # that points to (4)
a that has a "latest" tag satisfying (5)
a that resolves to (2)
If you need to install a dependency that haven't been included in package.json, simply do npm install <packageName>. Whether or not you want to include this newly installed package in package.json is completely your take. You can also decide how this newly installed package shall appear in your package.json
npm install <packageName> [--save|--save-dev|--save-optional]:
--save: Package will appear in your dependencies.
--save-dev: Package will appear in your devDependencies.
--save-optional: Package will appear in your optionalDependencies.
Does a dependency simply mean a folder that I want node to include?
Umm, partly yes. You may consider dependencies as folders, typically stored in node_modules directory.
Do I want to add this Node folder on my Desktop to be able to run my
scripts?
No, node manages it all. npm install will automatically create node_modules directory and you can refer to those dependencies with
require() in your .js files
var express = require('express');
Node REPL simply provides a way to interactively run JavaScript and see the results. It can be used for debugging, testing, or just trying things out.
process.cwd() points to the directory from which REPL itself has been initiated. You may change it using process.chdir('/path'), but once you close the REPL session and restart, it would always re-instantiate process.cwd() to the directory from which it has been started.
If you are installing some packages/dependencies in node project1 and think those dependencies can also be useful for node project2,
install them again for project2 (to get independentnode_modules directory)
install them globally [using -g flag]. see this
reference packages in project2 as
var referencedDependency = require('/home/User/project1/node_modules/<dependency>')
Simply doing process.chdir('/home/User/project1/node_modules/') in REPL and referencing as
var referencedDependency = require('<dependency>') in your js file wont work.
>process.chdir('/Users/MyUser/Desktop/Node/'); change the working directory only for that particular REPL session.
Hope it helps!

This has nothing to do with node.js but is rather inherent in the design of Unix (which in turn influences the design of shells on other operating systems).
Processes inherit values from their parent's environment but their environments are distinct.
That terse description of how process environments work has a sometimes unexpected behavior: you cannot change your parent's environment. It was designed this way explicitly for security reasons.
What this means is, when you change the working directory in a process and quits that process your shell's working directory will not be affected. Indeed, your shell's working directory isn't affected even when the process (in this case, node REPL) is running.
This exact question is often asked by people writing shell scripts wanting to write a script that CDs into someplace. But it's also common to find this question asked by people writing other languages such as Perl, Tcl, Ruby etc. (even C).
The answer to this question is always the same regardless of language: it's not possible to CD from another program/script/process.
I'm not sure how Windows handles it so it may be possible to do it there. But it's not possible on Unixen.

Running grunt on dependencies

I have 3 projects all using grunt.
Project a depends on project c and b
Project b depends on project c
Project c depends on nothing
Project a and b both require a step that compiles project c (which is a style repo that contains global styles for our org).
I am attempting to run grunt post install for project b.
There are a couple problems here.
Project a and b both try to build project c in their build processes. This takes sometime and I'd rather avoid it.
Project b expects different paths when running alone. loadNpmTasks fails unless I grunt.file.setBase, but then I have other paths that are broken as well. This means I need to manually track all of those down and make sure they are correct in both situations. This is rather flimsy.
As a result, I am thinking I am not doing this correctly or in a "normal way". What is the appropriate way to manage dependencies that use grunt?
Update
The main problem is that I get the following errors no matter what solution I've been choosing:
Local Npm module "grunt-contrib-compass" not found. Is it installed?
Local Npm module "grunt-contrib-handlebars" not found. Is it installed?
Local Npm module "grunt-contrib-requirejs" not found. Is it installed?
Local Npm module "grunt-contrib-watch" not found. Is it installed?
Local Npm module "grunt-notify" not found. Is it installed?
Local Npm module "grunt-curl" not found. Is it installed?
Local Npm module "grunt-shell" not found. Is it installed?
I tried spawn a task and calling grunt via the command line. I tried using various plugins to help. I think the issue is that both the top level and the dependencies require those tasks. This means they get pulled up into the parent node_modules folder. As a result the dependencies are missing the above modules in their node_modules folder.

I've built grunt-grunt with exactly this kind of scenario in mind.
grunt.initConfig({
grunt: {
lintsome: {
gruntfile: 'node_modules/some/Gruntfile.js',
task: 'jshint'
}
}
});
The README contains enough documentation to get you going.