Develop Reference

How to Dynamically Fetch Files with WebPack and Exclude Directories (node_modules)

TL;DR - We have ran into an issue with Storybook inside of a Monorepo
whose packages are independently versioned where the problem surrounds
Storybook running out of memory when it tries to scan for
*.stories.js due to it scanning every package's node_modules
directory. require.context() doesn't allow for exclusion to the
point where it won't scan the node_modules directories and we
couldn't find a native webpack solution to this problem - nor could we
find any existing discussion points around this problem, whose
solution wasn't just require.context('.', true,
/^\.\/((?!node_modules).)*\.stories\.js$/). We're posting our
discovery and potentially solution to see if anyone in the community
has had to solve a similar problem, and see if we are missing
something/making it harder than it needs to be.
The problem that we are trying to solve for is that Webpack's require.context() method does not allow for excluding specific directories from being scanned. For our project set-up, we are utilizing Lerna to manage a monorepo a JavaScript packages that are all independently versioned. Our file tree looks like this:
.
├── config
│ └── storybook
└── packages
└── vue
└── components
├── example-component
│ ├── node_modules
│ │ └── ...
│ ├── test
│ │ ├── example.stories.js
│ │ └── index.spec.js
│ ├── index.vue
│ └── package.json
└── example-parent
├── example-component-2
│ ├── node_modules
│ │ └── ...
│ ├── test
│ │ ├── example.stories.js
│ │ └── index.spec.js
│ ├── index.vue
│ └── package.json
└── example-component-3
├── node_modules
│ └── ...
├── test
│ ├── example.stories.js
│ └── index.spec.js
├── index.vue
└── package.json
For our storybook configuration, we need to tell Storybook where each of the components' *.stories.js file lives. We want/need the solution to be dynamic in a sense where if we add a new component into the directory tree in the same fashion, Storybook should automatically find the new *.stories.js without having to update Storybook's config file.
The option that Storybook suggests, is to use the require.context(String directory, boolean isDeep, RegEx fileMatch) method that is a part of webpack. However, when isDeep is set as true you are unable to restrict the method from scanning particular directories, like node_modules, due to the fact that the directory parameter is a string and not RegEx. Understandably, a regex match for the directory would make this method more expensive. However, it means that even if I set up my method to look like require.context('.', true, /^\.\/((?!node_modules).)*\.stories\.js$/) WebPack is still going to traverse my node_modules directories but won't match any internal *.stories.js files.
This becomes an issue when we use Lerna to install all the packages' depenencies via lerna bootstrap. Since by default that command will install ALL dependencies, including devDependencies for stuff like automated tests, an individual node_modules directory for a component could be quite large. So as we are adding more and more components, we have ran into a memory issue with running Storybook when components have all of their dependencies installed.
Our initial band-aid solution, was to restrict Lerna from install devDependencies by using lerna bootstrap -- --production --no-optional however this makes us unable to run tests. To run tests, we then have to install all dependencies... which means we can't run Storybook until we clean up the node_modules.
Our current solution is to use a fs type of library/package that allows us to traverse the directory tree with more specificity, and manually ignore node_modules when scanning deep directories. This allows us to still nest components within parent directories for better organization of similar components, and still allows Storybook to automatically find new *.stories.js files. So now our story discovery and Storybook config looks like:
const path = require('path');
const directoryTree = require('directory-tree');
let isPopulatedDir = (item) => item.type === 'directory' && item.children.length > 0;
let isStoryFile = (item) => item.type === 'file' && /\.stories\.js$/.test(item.name);
function parseChildren(children, response) {
for(var i = 0; i < children.length; i++) {
let child = children[i];
if(child.name === 'node_modules') continue;
if (isPopulatedDir(child)) {
parseChildren(child.children, response);
} else if(isStoryFile(child)) {
response.push(child.path);
}
}
return response;
}
function getStories() {
const tree = directoryTree("./packages/vue/components"),
storyPaths = [];
tree.children.forEach(item => {
if (isPopulatedDir(item)) {
storyPaths.push(...parseChildren(item.children, []));
}
});
return storyPaths;
}
module.exports = {
stories: getStories(),
webpackFinal: async (config) => {
config.node = {
fs: "empty"
};
config.resolve.modules = [
...(config.resolve.modules || []),
path.resolve('./'),
];
// Return the altered config
return config;
}
}
This solution works, it doesn't traverse any node_module directories, and allow us to dynamically find the *.stories.js files, no matter how their directory may be set up. But we can't help but feel like this is a problem that other people have had to of ran into, and maybe we are just overlooking a solution that Webpack already offers. So please, if you have had similar obstacles to overcome whose solution may work better for us we'd love to hear your approach or thoughts on our solution.

Generate i18n static HTML using Parcel bundler

I'm using Parceljs to bundle html and js. It works really well with less configuration.
Now, I'm facing i18n issue.
Google recommends using different URLs for each language version of a page.
https://support.google.com/webmasters/answer/182192
So, I want to generate language specific static html from one template like below.
.
├── dist
│ ├── ja
│ │ └── index.html
│ ├── app.c328ef1a.js
│ └── index.html
├── i18n
│ ├── default.json
│ └── ja.json
└── source
├── app.js
└── index.html
source/index.html
<html>
<body>
<h1>__TITLE__</h1>
<script src="/app.js"></script>
</body>
</html>
i18n/default.json
{
"__TITLE__": "Hello world!"
}
i18n/ja.json
{
"__TITLE__": "こんにちは 世界！"
}
Is there a way to deal with this issue using parceljs?
Or, should I write a code for prebuild?
Thank you.

Self answer:
I found a great answer here.
It mentions node-static-i18n package that generates i18n static HTML.
This tool isn't a plugin of parceljs, but it seems to be able to generate expected results.
Welcome yet another answer.

Navigating from Feature file to step defination: Any Plugins

In visual studio cod, How to navigate from feature to step definition. Do we need any additional plugins or any configuration needs to be added. I have downloaded the Cucumber (Gherkin) Full Support plugin but still cannot navigate from .feature to the step definition.

The documentation of Cucumber (Gherkin) Full Support plugin has the explanation for it.
You need to add the below in your settings:
{
"cucumberautocomplete.steps": [
"test/features/step_definitions/*.js",
"node_modules/qa-lib/src/step_definitions/*.js"
],
"cucumberautocomplete.syncfeatures": "test/features/*feature",
"cucumberautocomplete.strictGherkinCompletion": true
}
cucumberautocomplete.steps => provide the path of the step definitions.
cucumberautocomplete.syncfeatures => provide the path of the feature files
After this(might be after a restart), cmd + click(on mac) would take to the step definition.
Thanks,
Naveen

Having installed the extension alexkrechik.cucumberautocomplete, I tried modifying the settings from both the UI of the extension and its corresponding settings JSON (by default, mine were in ~/.config/Code/User/settings.json). But this didn't work because I got this error in the *.feature files: Was unable to find step for "Some feature description".
I noticed I had skipped a step mentioned in the extension docs... By default, it was getting the settings.json from my userspace and not my work(project)space.
For me, the solution was to go to the root directory of my project (usually outside of /src, where you have the package.json and node_modules/) and create a .vscode/ folder. Then, create a settings.json file and paste there the cucumberautocomplete configuration with the paths relative to this brand new file.
Below I show a schema:
myProject/
├── node_modules
├── package.json
├── subdir1
│ ├── src
│ └── test
│ └── e2e
│ └── src
│ ├── features
│ │ └── myfeature1.feature
│ ├── mypageobject1.po.ts
│ └── steps
│ └── mystep1.step.ts
└── .vscode
└── settings.json
An example of configuration would be:
{
"editor.detectIndentation": false,
"window.zoomLevel": 0,
"cucumberautocomplete.steps": [
"subidr1/test/e2e/src/steps/*.steps.ts"
],
"cucumberautocomplete.syncfeatures": "subidr1/test/e2e/src/feature/*.feature"
}
Note that you could use **/*.steps.ts and **/*.feature paths but every time the extension settings file changes, when you Ctr + Click on a feature description, you will need to wait for the editor to resolve the paths. Otherwise, there is no waiting time.

Nodejs: Good practice to just use the index.js to EXPORTS?

I am seeing a pattern on some code I have inherited. Each directory has its JS file but there is also a index.js that actually exports items from the other JS file or files.
I presume this is done so you can see exactly what you are exporting, as the main exports are in index.js and the main code is in the other js file or files.
Is this correct? What is this pattern called ?
Should I continue using this pattern.

Let's say I have the following directory structure:
MyApp
├── app.js
├── test.js
├── package.json
├─┬ controllers
│ ├── index.js
│ ├── signIn.js
│ └── signOut.js
└─┬ views
├── index.js
├── signIn.js
└── signOut.js
Placing the following code inside the index.js files...
// index.js
module.exports = {
signIn: require('./signIn')
, signOut: require('./signOut')
};
...allows you to require an entire directory like...
// test.js
describe('controllers', () => {
// ~/controllers/index.js
const controllers = require('./controllers');
it('performs a sign-in', () => {
...
});
it('performs a sign-out', () => {
...
});
});
The alternative is to require each file individually.
Having an index.js in a directory is not required. You may require a file in a directory without an index.js all the same.
// app.js
const signOut = require('./controllers/signOut.js')
However, it gets tedious as your app grows. I use a package like require-directory as typing out each file in a directory is also tedious and somewhat error prone.
// index.js
module.exports = require('require-directory')(module);
/*
This yields the same result as:
module.exports = {
signIn: require('./signIn')
, signOut: require('./signOut')
, ...
};
*/

ES6 CommonJS Module syntax
Given these two common types of structures...
MyApp
│ // files divided per type (controllers, components, actions, ...)
├─┬ actions
│ ├── index.js
│ ├── signIn.js
│ └── signOut.js
├─┬ components ...
├─┬ reducers ...
├─┬ pages ...
│
│ // files divided per component
├─┬ components ...
│ ├── index.js
│ ├── SimpleComponent.jsx
│ ├── AnotherComponent.duck.jsx // redux "duck" pattern
│ ├─┬ ComplexComponent // large complex logic, own actions, stylesheet, etc.
│ ...
├─┬ pages ...
│ ├── index.js
│ ├─┬ App
│ │ ├── index.js // not necessary here, matter of habit
│ │ ├── App.jsx
│ │ ├── actions.js
│ │ └── reducer.js
│ └─┬ Dashboard
├── another.js
...
You can simply import files in another.js like this
import {signIn, signOut} from './actions'
import {App} from './pages'
import {ComplexComponent} from './components'
instead of this (without index.js files)
import {signIn} from './actions/signIn'
import {signOut} from './actions/signOut'
import {App} from './pages/App/App' //notice the redundancy here
import {ComplexComponent} from './components/ComplexComponent/ComplexComponent'
More reading
ECMAScript 6 modules
import - JavaScript | MDN
Babel transpiler - brings the new imports to your browser now
Structuring React projects
React Redux "Ducks pattern" - a single file approach for components

The other answers provide a lot of great information, but to try and specifically answer your question 'Should I continue using this pattern", I'd say no, at least most of the time.
The thing is, this pattern requires extra effort, as you have to maintain those extra index.js files. In my experience that effort is greater than the effort to simply write one-directory-longer import statements. Plus, you can get the same functionality you'd get from having an index.js without one, by using a module like require-dir.
All that being said, if you are making a library that will be consumed by a large number of people, like a critical module in a large programming department, or a public NPM module, then the effort of an index.js becomes more justified. As long as you have enough people using your modules, your users will (cumulatively) save more time from you adding them than you will lose maintaining them.

I will directly dive into your question on whether to use this pattern or not (as other answers are not sufficient for this).
Assuming that each directory in your code represents a standalone module (doesn't rely on another module to work). Using this pattern will give these advantages:
Better and more organized imports
Separation between internal/external definitions of each module (similar to using private/public on an interface/API)
The problems with this:
It can be very tiresome to keep loose-coupling of the different modules (JS/TS is not pure OOP)
Requires active refactoring to modules definition - more circular dependencies.
Loads more code to memory (even if unused) - though I'm not sure how bad this can be as there are optimizations that usually fix this problem when bundling production code.
Circular dependencies are very problematic, importing the whole module/directory using index.js will import all of its parts (that are declared in index.js) so if you have:
-- moduleA
├-- comp1A // needs comp1B
├-- comp2A
└-- index.js // export both comp1/2
-- moduleB
├-- comp1B
├-- comp2B // needs comp2A
└-- index.js // export both comp1/2
Example case - comp1A needs something from comp1B while comp2B needs something from comp2A
When importing the specific files (without index.js - import something from './moduleB/comp1B') you won't have circular dependencies.
But if you use index.js (import something from './moduleB') you will have circular dependencies.
My recommendation is to use index.js in the right places, and to keep those maintained! Using index.js with small modules will be perfect, but with time they will grow and should be divided. index.js is very bad to use in common/shared/utils/misc/core (whatever you call it when you want to put uncategorized and unrelated code that is used across your whole project) module.

What about this?
module.exports = {
...require('./moduleA'),
...require('./moduleB')
}
(moduleA.a will be overridden by moduleB.a)

How to optimize all javascript files in a directory with requirejs

I have a folder structure like this:
.
├── autocomplete
│   ├── core.js
│   ├── search.js
│   └── user.js
├── build.js
├── collapsible_lists.js
├── griffgrabber
│   ├── canvasobject.js
│   ├── cargame.js
│   ├── car.js
│   ├── griffDrawer.js
│   ├── keylistener.js
│   ├── run.js
│   └── victim.js
├── main.js
├── newsfeed.js
├── require.js
├── shortcut.js
└── sidebar.js
3 directories, 20 files
main.js is the startup file. That file requires a couple of the files, but not all of them. The rest of the files are included with
<script>
require(['shortcut'], function(shortcut){
// ...
})
</script>
in some html files.
This is my build.js file so far:
{
baseUrl: ".",
name: "main",
out: "main-built.js",
}
But it only includes the files that are required by main.js. Is it possible to optimize all the javascript files in one run?

(to expand #Ryan Lynch's suggestion):
Use the include option, as per the documentation:
You can always explicitly add modules that are not found via the optimizer's static analysis by using the include option.
(http://requirejs.org/docs/optimization.html)
{
baseUrl: ".", // ?
appDir: ".", // ?
dir: "output/",
modules: [
{
name: "main",
include: [ "shortcut" ]
}
]
}
More detailed example in the excellent example.build.js (I actually find it more useful than the documentation page)
(sorry, had no time to replicate and test properly to make sure the paths values are correct, I'll try that later and update my answer)

Try including a modules array in your options, so:
{
baseUrl: ".",
out: "main-built.js",
modules: [
{
name: "main"
}
]
}
As per the documentation:
In the modules array, specify the module names that you want to
optimize, in the example, "main". "main" will be mapped to
appdirectory/scripts/main.js in your project. The build system will
then trace the dependencies for main.js and inject them into the
appdirectory-build/scripts/main.js file.

Another - less orthodox - way of achieving this would be to add this "shortcut" module as a dependency to any of the "visible" modules that are discovered by r.js scanning (i.e. "main.js"). This way the entire dependency branch starting at "shortcut" would be included in the output.