Yarn includes an optional "Plug'n'Play" feature which moves node_modules out of the project directory. In doing so it creates a .pnp.js file with references to various dependency paths on the hard drive.
The file is auto-generated, is several thousand lines long, and appears to reference paths specific to the machine that ran yarn install.
Is .pnp.js meant to be commited with the rest of the code? I can't seem to find any information about this despite the file appearing to be useless to commit. What is conisdered best practice and why?
I found this comment by Arcanis (the lead maintainer of Yarn) on an issue asking about what to include in a .gitignore. It may answer your question:
.yarn/plugins and .yarn/releases contain the Yarn releases used in the current repository (as defined by yarn set version). You will want to keep them versioned (this prevents potential issues if, say, two engineers use different Yarn versions with different features).
.yarn/unplugged should likely always be ignored, since it may contain native builds
.yarn/build-state.yml should likely be ignored as well, as it contains the build infos
If for some reason you version unplugged, it may make sense to keep build-state as well
.yarn/cache may be ignored, but you'll need to run yarn install to regenerate it
Versioning it unlocks what we call Zero-Installs - it's optional, though
.pnp.js (and potentially .pnp.data.json) are in the same boat as the cache. Add it to your repository if you keep your cache in your repository, and ignore it otherwise.
yarn.lock should always be stored within your repository (even if you develop a library)
So to summarize:
If you're using Zero-Installs:
.yarn/unplugged
.yarn/build-state.yml
If you're not using Zero-Installs:
.yarn/*
!.yarn/releases
!.yarn/plugins
.pnp.*
As one of the commenters noted, the official .gitignore recommendations are here, along with excellent commentary. The only critique I have is in the way they showed it there, I think they should have combined the common ways into one, and then separated the special cases for zero-install vs not. Here then is how I have it. But again, note that article, as some of these (e.g. .yarn/sdk) are optionally added or not.
# ------- yarn -------
# see excellent notes at: https://yarnpkg.com/getting-started/qa#which-files-should-be-gitignored
# Also: `yarn.lock` and `.yarnrc.yml` (or it's older counterpart .yarnrc) "should always be stored in your repo"
.yarn/*
!.yarn/patches
!.yarn/plugins
!.yarn/releases
!.yarn/sdks
!.yarn/versions
## --> ADD if using zero-install, otherwise do NOT:
#!.yarn/cache
## --> ELSE ADD if NOT using yarn's zero-install:
.pnp.*
# ------- end yarn -------
Related
Kind of a noob question here...
We've got a private npm module - a library - that needs to be included in other projects. So far, very simple.
Currently, we're simply "remembering" to manually do an npm run buld before pushing changes to our git repo, and then dependent projects when they do an npm run whatever, they're setup to pull from our repo and use the latest version already "compiled" as a module.
So, there are issues with this approach:
It relies on humans being able to perfectly remember to do a build before pushing to origin. (inherently fragile).
VSCode constantly shows me the build-artifacts as if they were source files. Git similarly shows merge conflicts relating to those files - which -- really, aren't source at all. They're compilation artifacts, but I'm not sure I want to .gitignore them - because - well, the point of all of this is to create those artifacts for use in other projects... so they belong in the repo, just not as source files...
So I'm not sure how to untangle this mess.
I want:
A simple way to update the source that doesn't cause git to become upset about merge conflicts for build artifacts, but only for actual source files
A simple way to ensure that the build artifacts are always updated upon push to origin (in fact, I'd prefer that it build & run our mocha tests and refuse to do a push if that fails)
I'm only about 9mos into using git on github - so there's a ton I don't know...
Ideas for better ways to manage this / automate this - are most welcome!
The key to implementation is of course simplicity. If it's easy to do, I'm sure I'll do it, and can get others to do so. But if it's a huge hurdle every time, well, we all know how well that goes over for other devs...
I'm trying to publish a javascript package to an internal repository. When I do yarn publish (or npm publish) it now runs nsp check (otherwise wonderful feature!) to look for known vulnerabilities in any of the dependencies in my project. It finds one and exits in error:
[3/4] Publishing...
$ nsp check
(+) 1 vulnerability found
...
│ More Info │ https://nodesecurity.io/advisories/nnn |
...
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/publish for documentation about this command.
I'd like to ignore this vulnerability and proceed with the publish anyway. How can I do that?
I understand that introducing a way to ignore a vulnerability may be misused, but in this instance the package I'm publishing is an internal tool. I've looked at the dependency's vulnerability and in no way could it be taken advantage of in the context in which I'm using it.
Sidenote: The vulnerability is nested 3 layers deep in a pretty commonly used package (request#2.87.0). The vulnerability itself has been fixed in its own latest version, but because this dependency is three layers deep we have to wait for each subsequent dependent to upgrade. I suspect it will take some time- there's been a ticket open in their Github for 3 months- and I don't want to wait on that.
I've found the answer to this:
Add a file called .nsprc at the top of your project. In the file add:
{
"exceptions": ["https://nodesecurity.io/advisories/nnn"]
}
The URL should match the URL of the vulnerability presented in the nsp check command.
What exactly should I put in .npmignore?
Tests? Stuff like .travis.yml, .jshintrc? Anything that isn't needed when running the module (except the readme)?
I can't find any guidance on this.
As you probably found, NPM doesn't really state specifically what should go in there, rather they have a list of ignored-by-default files. Many people don't even use it as everything in your .gitignore is ignored in npm by default if .npmignore doesn't exist. Additionally, many files are already ignored by default regardless of settings and some files are always excluded from being ignored, as outlined in the link above.
There is not much official on what always should be there because it is basically a subset of .gitignore, but from what I gather from using node for 5-ish years, here's what I've come up with.
Note: By production I mean any time where your module is used by someone and not to develop on the module itself.
Pre-release cross-compiled sources
Pros: If you are using a language that cross-compiles into JavaScript, you can precompile before release and not include .coffee files in your package but keep tracking them in your git repository.
Build file leftovers
Pros: People using things like node-gyp might have object files that get generated during a build that never should go into the package.
Cons: This should always go into the .gitignore anyway. You must place these things inside here if you are using a .npmignore file already as it overrides .gitignore from npm's point of view.
Tests
Pros: Less baggage in your production code.
Cons: You cannot run tests on live environments in the slim chance there is a system-specific failure, such as an out of date version of node running that causes a test to fail.
Continuous integration settings/Meta files
Pros: Again, less baggage. Things such as .travis.yml are not required for using, testing, or viewing the code.
Non-readme docs and code examples
Pros: Less baggage. Some people exist in the school-of-thought where if you cannot express at least minimum viable functionality in your Readme, your module is too big.
Cons: People cannot see exhaustive documentation and code examples on their own file system. They would have to visit the repository (which also requires an internet connection).
Github-pages objects
Pros: You certainly don't need to litter your releases with CNAME files or placeholder index.htmls if you use your module serves double-duty as a gh-pages repository as well.
bower.json and friends
Pros: If you decide to build in your dependencies prior to release, you don't need the end-user to install bower then install more things with that. I would, personally, keep that stuff in the package. When I do an npm install, I should only be relying on npm and no other external sources.
Basically, you should ever use it if there is something you wish to keep out of your npm package but checked-in to your module's repo. It's not a long list of items, but npm would rather build in the functionality than having people stuck with irrelevant objects in their package.
I agree with lante's short and syntetic answer and SamT's big answer:
You should not include your tests in your package.
Your package should only contains production runtime files.
That will make your package more straightforward and faster to be dowloaded.
My contribution to those answers:
.npmignore is the blacklist way to achieve package file selection. But in a more practical way, you can whitelist files you need to include in your package using the files field in your package.json:
{
"files": [
"lib/",
"index.js"
]
}
I think that's simpler, future proof and have better semantics ;)
Just to clarify, anytime someone do npm install your-library, npm will download all source files that the package includes. Those files that were included in the .npmignore file in the source code of the package your-library will be excluded when publishing the lib, so users of your-library won't download them.
Know that people installing your library will need just your library running, anything else will be not necessary.
For example, when someone installs a library, its probably that he/she doesn't care about your .travis.yml or your .jshintrc files, or even some images, Grunt files, documentation, etc.
.npmignore could let your npm package to have less files, and faster to be downloaded
Don't include your tests. Oftentimes tests are like 5x the size of the actual codebase. As long as your tests are on Github, etc, that's good enough.
But what you absolutely should do is test your NPM package in its published format. Create some smoke tests that reside in the actual codebase, but are not part of the test suite.
You can read about testing your package after tarballing it, here:
https://github.com/ORESoftware/r2g
How to test an `npm publish` result, without actually publishing to NPM?
I noticed that some of my pushes to heroku fail because there's a mismatch in capitalization in a dependency in my package.json with what I actually require in javascript. For example, in package.json, I may have something named "somepackage", but in my javascript file I have require("somePackage"). Locally, this works fine, but on heroku it fails. Is there a node setting to enforce case sensitivity so it can fail locally for me?
Please see this discussion on github https://github.com/npm/npm/issues/3914
The preferred naming convention for npm packages is "all-lowercase". you should probably make the necessary changes in your code to follow this convention.
mixing cases brings cross-platform problems mostly because linux has a case-sensitive file system while windows and macOS by default have a case-insensitive file system.
The "all-lowercase" convention was made to avoid surprises such as the ones you describe.
If your are working on Mac OS X, you can check how your file system is configured by following the first answer on https://apple.stackexchange.com/questions/22297/is-bash-in-osx-case-insensitive
I wrote an NPM module that does this and also checks to ensure that all the dependencies referenced in your code are reflected in your package.json file:
nodejs-dep-check
https://www.npmjs.com/package/nodejs-dep-check
the purpose of course is to save you time so that you don't initialize a build or experience a runtime error without a test/check beforehand.
So after a long day at work, I sit down and see an alert for Windows SkyDrive in the system tray:
Files can't be uploaded because the path of this file or folder is too long. Move the item to a different location or shorten its name.
C:\Users\Matthew\SkyDrive\Documents\Projects\Programming\angular-app\server\node_modules\grunt-contrib-nodeunit\node_modules\nodeunit\node_modules\tap\node_modules\runforcover\node_modules\bunker\node_modules\burrito\node_modules\traverse\example\stringify.js
... and for a while, I laughed at that technological limitation.
But then, I wondered: is that amount of directory recursion within a Node project really necessary? It would appear that the paths beyond "angular-app\server\node_modules" are simply dependencies of the project as a whole and might be better expressed as:
C:\Users\Matthew\SkyDrive\Documents\Projects\Programming\angular-app\server\node_modules\grunt-contrib-nodeunit\
C:\Users\Matthew\SkyDrive\Documents\Projects\Programming\angular-app\server\node_modules\nodeunit\
C:\Users\Matthew\SkyDrive\Documents\Projects\Programming\angular-app\server\node_modules\tap\
C:\Users\Matthew\SkyDrive\Documents\Projects\Programming\angular-app\server\node_modules\runforcover\
C:\Users\Matthew\SkyDrive\Documents\Projects\Programming\angular-app\server\node_modules\bunker\
C:\Users\Matthew\SkyDrive\Documents\Projects\Programming\angular-app\server\node_modules\burrito\
C:\Users\Matthew\SkyDrive\Documents\Projects\Programming\angular-app\server\node_modules\traverse\
I hadn't really given it much thought before, as package management in Node seems like magic compared to many platforms.
I would imagine that some large-scale Node.js projects even contain many duplicate modules (having the same or similar versions) which could be consolidated into a lesser amount. It could be argued that:
The increased amount of data stored and transmitted as a result of
duplicate dependencies adds to the cost of developing software.
Shallower directory structures (especially in this context) are
often easier to navigate and understand.
Excessively long path names can cause problems in some computing
environments.
What I am proposing (if such a thing does not exist) is a Node module that:
Recursively scans a Node project, collecting a list of the nested node_modules folders and how deeply they are buried in relation to the root of the project.
Moves the contents of each nested node_modules folder to the main node_modules folder, editing the require() calls of each .js file such that no references are broken.
Handles multiple versions of duplicate dependencies
If nothing else, it would make for an interesting experiment. What do you guys think? What potential problems might I encounter?
See if
npm dedupe
sets you right.
API Doc here
See fenestrate, npm-flatten, flatten-packages, npm dedupe, and multi-stage-installs.
Quoting Sam Mikes from this StackOverflow question:
npm will add dedupe-at-install-time by default. This is significantly more feasible than Node's module system changing, but it is still not exactly trivial, and involves a lot of reworking of some long-entrenched patterns.
This is (finally) currently in the works at npm, going by the name multi-stage-install, and is targeted for npm#3. npm development lead Forrest Norvell is going to spend some time running on Windows in the new year, so please do create windows-related issues on the npm issue tracker < https://github.com/npm/npm/issues >