Develop Reference

Bundling pouchdb-adapter-memory with Rollup

I am attempting to use the memory adapter for PouchDB. I want to bundle my application (along with dependencies like Pouch and this adapter) using Rollup. In order to reduce this to a minimally reproducible issue imagine this is my application I want to bundle:
import PouchDB from 'pouchdb-browser'
import MemoryAdapterPlugin from 'pouchdb-adapter-memory'
PouchDB.plugin(MemoryAdapterPlugin)
Since I am using a node module I'm obviously going to need the rollup-node-resolve Rollup plugin. Also many of PouchDB's dependent modules are in CJS format so I'm going to also need the rollup-node-commonjs Rollup plugin. Finally PouchDB makes use of some built-in Node modules (such as events) and the memory adapter even more (buffer, etc). I'm not sure these are used at runtime (it may be just in PouchDB's code because it can work in Node or the browser) but to prevent a bundling error I'm going to also include the rollup-plugin-polyfill-node plugin but direct the resolve plugin to prefer built-ins to the browser target if they are needed and available.
With all that in place here is my rollup config:
import commonjs from '#rollup/plugin-commonjs'
import resolve from '#rollup/plugin-node-resolve'
import polyfillNode from 'rollup-plugin-polyfill-node'
export default {
input: 'index.js',
output: {
format: 'iife',
name: 'app',
file: 'bundle.js'
},
plugins: [
commonjs({
requireReturnsDefault: "auto",
}),
polyfillNode(),
resolve({
preferBuiltins: true,
browser: true
}),
]
}
This will bundle. But when I load up the bundle in a browser I get an error about the inherits polyfill not working at this line:
https://github.com/FredKSchott/rollup-plugin-polyfill-node/blob/main/polyfills/inherits.js#L7
It says superCtor is undefined. If I step back a level on the backtrace it comes from this line:
https://github.com/nodejs/readable-stream/blob/main/lib/_stream_duplex.js#L46
The Readable is undefined. When I bundle I get warnings about circular references. I think it fundementally revolves around the fact that some of the PouchDB dependencies use NPM packages as polyfills explicitly (like inherits and readable-stream) while the polyfillNode plugin is also providing some of those same polyfills and they are doing so in an incompatible way. But I don't know how to untangle it.

Finally was able to resolve this so going to provide the answer in case someone else needs to bundle PouchDB memory adapter in Rollup.
The readable streams polyfill is a mess of circular dependencies. This is fine under a CJS format but since Rollup converts CJS to ES format it becomes problematic as Rollup can't figure out the proper order to put things in. This leads to the use of inherits where the superclass is not yet defined. The readable streams polyfill has an open ticket regarding this. The general solution is to fix it upstream in Node and then cut a new copy of the polyfill based on that upstream fix. The upstream fix was made but the polyfill hasn't yet been updated. Once it is this should naturally resolve itself.
In the meantime we can use someone else's fork of the polyfill that has the circular dependencies resolved. This fork is mentioned in that issue but the punchline is to add this to the package.json:
"readable-stream": "npm:vite-compatible-readable-stream#^3.6.0",
This will force readable-stream to resolve to that fork. But that's not the whole story. There are some dependencies of the memory adapter that are locked to a different version of readable-stream. To resolve that we want to add the following to our rollup's resolve plugin:
dedupe: ['readable-stream']
This will force the readable-stream substitute that we have explicitly added to our project to be used anytime a readable stream is needed.
The final issue is that PouchDB is using an ancient version of memdown that also has circular dependency issues. The latest version does not and seems to work with the memory adapter just fine. There is an open ticket at PouchDB to update memdown and this will naturally resolve when that happens.
In the meantime to resolve that we are going follow a similar pattern as above. We will explicitly require memdown into our project although no need for a fork. Just a newer version. Then to force the memory adapter to use this version we add memdown to that dedup option as well.

Grunt and ES6 modules in node - incompatible without the use of .mjs?

So, I'm dabbling a bit with Typescript and Grunt at the moment to see if it's worth it for me. The thing is: Typescript does not compile to *.mjs files but only regular *.js files. Node does support ES6 Modules but only if you either mark them as '*.jsm' files or by setting "type": "module". Setting this top-level field in package.json however has global scope for any *.js file in the same directory and any following ones.
This breaks the Gruntfile.js file as it seems since it uses CommonJS modules, see my very basic Gruntfile as example:
module.exports = function (grunt) {
grunt.initConfig({
ts: {
default: {tsconfig: "./tsconfig.json"}
}
})
grunt.loadNpmTasks("grunt-ts");
grunt.registerTask("default", ["ts"]);
}
Without expecting much success I naively changed the export syntax from module.exports = to export default which expectedly did no work since it didn't make much sense.
Questions
Is there any option to use Grunt with ES6 modules enabled in node?
Is there a proper way to tell TypeScript to compile to *.mjs files?

If you set "type": "module" in your package.json, you need to rename Gruntfile.js to Gruntfile.cjs, and run it with grunt --gruntfile Gruntfile.cjs.

The suggested approach with Babel running before Grunt makes Grunt a bit redundant. Since TypeScript does not yet support exporting ES6 modules to *.mjs files (and you have to use the *.mjs suffix in your import when node should still be running with its CommonJS system) and will probably never fully (see Design Meeting Notes 11/22/2019) I have to conclude that ES6 modules still have serious implications and issues. Changing the file extension is not enough since the extension-less imports fail with node. You'd need to go through every compiled file and change the import to specifically load *.mjs files.
However, the TypeScript Compiler can be set up in a way that it does understand ES6 module syntax and to compile to CommonJS (see TS handbook).
{
"compilerOptions": {
"module": "CommonJS",
// [...]
},
}
This way the TypeScript code be written with ES6 module syntax and the output can be CommonJS compatible without braking other code. As a bonus you can skip the Babel approach and grunt can run TS compiler.

You can using babel-node to compile first. Which will resolve ES6 export and import problem.
npm install --save-dev #babel/core #babel/node
npx babel-node server.js

Codemod for Babel `import` into commonjs `require`

I'm looking for a way to converting a full node-project's Babelimports into CommonJS-style require(). The goal is to get rid of Babel.
Considering node.js has things like async/await built-in nowadays it feels redundant to run Babel. The only thing left that Babel does currently is that it converts the ES6-style imports into require().
I've been searching but can't find any elegant solution to do it semi-automatically. The output when compiling Babel isn't clean enough to just copy without a lot of manual work.
If I have a file with input like this:
import express from 'express'
import bodyParser from 'body-parser'
import authMiddleware from './middlewares/auth'
import { get } from 'lodash'
export const myVar = 1
export default function doSomething() {
// ...
}
.. I'd want an output similar to this
const express = require('express')
const bodyParser = require('body-parser')
const authMiddleware = require('./middlewares/auth').default
const { get } = require('lodash')
export.myVar = 1
export.default = function doSomething() {
// ...
}
Alternatively that it converted the files to the .mjs-syntax for the relative ones and used require() for external stuff.
It's not the first time I have an old node project running Babel where it's turned more-and-more redundant with time, so I'm sure someone has done neat solution to this before.

I dig up the source code of babel-plugin-transform-modules-commonjs. Looks like it's impossible to config babel to output your desired result.
Reason behind is the necessity of helpers like _interopRequireDefault still holds strong, because ES module is not backward-compat to commonjs, notably the export default thing.
Take for example:
// input
import bodyParser from 'body-parser'
import authMiddleware from './middlewares/auth'
// your desired output
const bodyParser = require('body-parser') // <-- no default
const authMiddleware = require('./middlewares/auth').default // <-- default
// actual babel output
var _bodyParser = _interopRequireDefault(require("body-parser"));
var _auth = _interopRequireDefault(require("./middlewares/auth"));
You have no way to tell when to add .default and when not to. Only proper way to handle this is by wrapping require() with _interopRequireDefault and do runtime check.
If compiler does trace the required module and check if it's a ES module or commonjs module, then it can tell if .default is needed. However babel is designed around a single-file-at-a-time model, so no chance it can do that for you.
I think if you can figure out a reliable rule to tell when to add a .default and when not to then perhaps a simple regex-replace will solve your problem.
Side note. I do have some idea to hack it out with a customized babel plugin.
You can fork the babel-plugin-transform-modules-commonjs source, remove the _interopRequireDefault wrapping logic, then you use a resolver to do the aforementioned check-if-requiree-is-esmodule job, then see if .default is needed in output.
But easier said than done, this requires some serious effort.

The simple solution, open your source code in an editor that can go through all files I use VSCode and set a ignore on the node_modules folder and do I regex replace on all files the full way if you need the multiple exports is below.
RegEx way
Search for:
import[\s*]([a-zA-Z0-9,]*)[\s*]from[\s*]['|"]([a-zA-Z0-9\{\},\.\/\\]*)['|"][\s*]
replace with
const $1 = require('$2')
if you use as do this also.
Search for:
import[\s*][a-zA-Z0-9,]*[\s*]as[\s*]([a-zA-Z0-9]*)[\s*]from[\s*]['|"]([a-zA-Z0-9\{\},\.\/\\]*)['|"][\s*]
replace with
const $1 = require('$2')
there are some drawbacks here you can't use multiple exports for that you need The full way
Long way
Ok so for anyone who is interested, here is the process I used to work this out, you can then copy your source out of the build folder into a new location as a new source or overwrite your old src and the remove all of babel from your project (npm prune).
This will leave all the support stuff that module needs including the support for export default _interopRequireDefault() the only way to get rid of this is to create your own plugin that does not do this.
Step 1
Identify what ECMA babel was using for that. So I went to https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import
it shows that import has been part of the spec since ES6 (also known as ECMA2015)
Step 2
The presets are just groups of packages so identify the package for that particular transpile.
Opened my package.json and looked for babel-preset-es2015 found it. went to node_moduels\babel-preset-es2015, opened it's package.json to find
"babel-plugin-transform-es2015-modules-amd": "^6.24.1",
"babel-plugin-transform-es2015-modules-commonjs": "^6.24.1",
"babel-plugin-transform-es2015-modules-systemjs": "^6.24.1",
"babel-plugin-transform-es2015-modules-umd": "^6.24.1",
Step 3
Some testing so using --plugins= argument for babel I tested what each of them did on a small set of 2 files one requiring the other to and test each one I worked out it was the commonjs version that was needed for require();
Step 4
Do the conversion
So make sure you have the following node modules installed babel-cli, babel-core, babel-plugin-transform-es2015-modules-commonjs
Then fire up the CLI and do,
babel --plugins=transform-es2015-modules-commonjs ./src/ --out-dir build/
taken from https://babeljs.io/docs/en/babel-cli

You can use plugin of putout called #putout/plugin-convert-esm-to-commonjs for this purpose. It converts:
import {readFile} from 'fs/promises';
To:
const readFile = require('fs/promises');
To get things done set up the base line with:
npx putout . --disable-all
Enable convert-esm-to-commonjs in configuration file .putout.json:
{
"rules": {
"convert-esm-to-commonjs": "on"
}
}
And apply fixes:
npx putout . --fix

Which tools do I need in order to use ES6 style imports for my project using Gulp?

I have a legacy project which uses Gulp to compile SASS and minify JS etc. (based on Drupal).
I'd like to use ES6 style imports and also be able to import modules that are located in node_modules without referencing the full path (for example import the lodash.debounce module as
import throttle from 'lodash.throttle'
What do I need to achieve that? (besides Gulp and NPM). Babel? Browserify? Lost with all the tooling.
I'd like to avoid webpack as it would increase the complexity of my project (and I don't know if I can use it in parallel with the way Drupal 7 is doing things)

Yep, you want babel. The documentation to set it up in your gulpfile is here: http://babeljs.io/docs/setup/#installation
You'll want to use the es2015 preset to enable the import syntax.
var gulp = require("gulp");
var babel = require("gulp-babel");
gulp.task("default", function () {
return gulp.src("src/app.js")
.pipe(babel({
presets: ['es2015']
}))
.pipe(gulp.dest("dist"));
});

To use import you need a transpiler, for example babel.
Assuming you are building code for the browser then to use import (transpiled to require) you need webpack or browserify which will also allow you to import from node_modules (there are other module builders too, jspm and systemjs, however I know next-to-nothing about them).
For either of these you will also need something to enable babel to be used in the build process. I use webpack with babel-loader.
I found a useful reference here specifically on using ES6 modules with webpack.

How do I install the babel-polyfill library?

I just started to use Babel to compile my ES6 javascript code into ES5. When I start to use Promises it looks like it's not working. The Babel website states support for promises via polyfills.
Without any luck, I tried to add:
require("babel/polyfill");
or
import * as p from "babel/polyfill";
With that I'll get the following error on my app bootstrapping:
Cannot find module 'babel/polyfill'
I searched for the module but it seems I'm missing some fundamental thing here. I also tried to add the old and good bluebird NPM but it looks like it's not working.
How to use the polyfills from Babel?

This changed a bit in babel v6.
From the docs:
The polyfill will emulate a full ES6 environment. This polyfill is automatically loaded when using babel-node.
Installation:
$ npm install babel-polyfill
Usage in Node / Browserify / Webpack:
To include the polyfill you need to require it at the top of the entry point to your application.
require("babel-polyfill");
Usage in Browser:
Available from the dist/polyfill.js file within a babel-polyfill npm release. This needs to be included before all your compiled Babel code. You can either prepend it to your compiled code or include it in a <script> before it.
NOTE: Do not require this via browserify etc, use babel-polyfill.

The Babel docs describe this pretty concisely:
Babel includes a polyfill that includes a custom regenerator runtime
and core.js.
This will emulate a full ES6 environment. This polyfill is
automatically loaded when using babel-node and babel/register.
Make sure you require it at the entry-point to your application, before anything else is called. If you're using a tool like webpack, that becomes pretty simple (you can tell webpack to include it in the bundle).
If you're using a tool like gulp-babel or babel-loader, you need to also install the babel package itself to use the polyfill.
Also note that for modules that affect the global scope (polyfills and the like), you can use a terse import to avoid having unused variables in your module:
import 'babel/polyfill';

For Babel version 7, if your are using #babel/preset-env, to include polyfill all you have to do is add a flag 'useBuiltIns' with the value of 'usage' in your babel configuration. There is no need to require or import polyfill at the entry point of your App.
With this flag specified, babel#7 will optimize and only include the polyfills you needs.
To use this flag, after installation:
npm install --save-dev #babel/core #babel/cli #babel/preset-env
npm install --save #babel/polyfill
Simply add the flag:
useBuiltIns: "usage"
to your babel configuration file called "babel.config.js" (also new to Babel#7), under the "#babel/env" section:
// file: babel.config.js
module.exports = () => {
const presets = [
[
"#babel/env",
{
targets: { /* your targeted browser */ },
useBuiltIns: "usage" // <-----------------*** add this
}
]
];
return { presets };
};
Reference:
usage#polyfill
babel-polyfill#usage-in-node-browserify-webpack
babel-preset-env#usebuiltins
Update Aug 2019:
With the release of Babel 7.4.0 (March 19, 2019) #babel/polyfill is deprecated. Instead of installing #babe/polyfill, you will install core-js:
npm install --save core-js#3
A new entry corejs is added to your babel.config.js
// file: babel.config.js
module.exports = () => {
const presets = [
[
"#babel/env",
{
targets: { /* your targeted browser */ },
useBuiltIns: "usage",
corejs: 3 // <----- specify version of corejs used
}
]
];
return { presets };
};
see example: https://github.com/ApolloTang/stackoverflow-eg--babel-v7.4.0-polyfill-w-core-v3
Reference:
7.4.0 Released: core-js 3, static private methods and partial
application
core-js#3, babel and a look into the future

If your package.json looks something like the following:
...
"devDependencies": {
"babel": "^6.5.2",
"babel-eslint": "^6.0.4",
"babel-polyfill": "^6.8.0",
"babel-preset-es2015": "^6.6.0",
"babelify": "^7.3.0",
...
And you get the Cannot find module 'babel/polyfill' error message, then you probably just need to change your import statement FROM:
import "babel/polyfill";
TO:
import "babel-polyfill";
And make sure it comes before any other import statement (not necessarily at the entry point of your application).
Reference: https://babeljs.io/docs/usage/polyfill/

First off, the obvious answer that no one has provided, you need to install Babel into your application:
npm install babel --save
(or babel-core if you instead want to require('babel-core/polyfill')).
Aside from that, I have a grunt task to transpile my es6 and jsx as a build step (i.e. I don't want to use babel/register, which is why I am trying to use babel/polyfill directly in the first place), so I'd like to put more emphasis on this part of #ssube's answer:
Make sure you require it at the entry-point to your application,
before anything else is called
I ran into some weird issue where I was trying to require babel/polyfill from some shared environment startup file and I got the error the user referenced - I think it might have had something to do with how babel orders imports versus requires but I'm unable to reproduce now. Anyway, moving import 'babel/polyfill' as the first line in both my client and server startup scripts fixed the problem.
Note that if you instead want to use require('babel/polyfill') I would make sure all your other module loader statements are also requires and not use imports - avoid mixing the two. In other words, if you have any import statements in your startup script, make import babel/polyfill the first line in your script rather than require('babel/polyfill').

babel-polyfill allows you to use the full set of ES6 features beyond
syntax changes. This includes features such as new built-in objects
like Promises and WeakMap, as well as new static methods like
Array.from or Object.assign.
Without babel-polyfill, babel only allows you to use features like
arrow functions, destructuring, default arguments, and other
syntax-specific features introduced in ES6.
https://www.quora.com/What-does-babel-polyfill-do
https://hackernoon.com/polyfills-everything-you-ever-wanted-to-know-or-maybe-a-bit-less-7c8de164e423

Like Babel says in the docs, for Babel > 7.4.0 the module #babel/polyfill is deprecated, so it's recommended to use directly core-js and regenerator-runtime libraries that before were included in #babel/polyfill.
So this worked for me:
npm install --save core-js#3.6.5
npm install regenerator-runtime
then add to the very top of your initial js file:
import 'core-js/stable';
import 'regenerator-runtime/runtime';