I am attempting to use the memory adapter for PouchDB. I want to bundle my application (along with dependencies like Pouch and this adapter) using Rollup. In order to reduce this to a minimally reproducible issue imagine this is my application I want to bundle:
import PouchDB from 'pouchdb-browser'
import MemoryAdapterPlugin from 'pouchdb-adapter-memory'
PouchDB.plugin(MemoryAdapterPlugin)
Since I am using a node module I'm obviously going to need the rollup-node-resolve Rollup plugin. Also many of PouchDB's dependent modules are in CJS format so I'm going to also need the rollup-node-commonjs Rollup plugin. Finally PouchDB makes use of some built-in Node modules (such as events) and the memory adapter even more (buffer, etc). I'm not sure these are used at runtime (it may be just in PouchDB's code because it can work in Node or the browser) but to prevent a bundling error I'm going to also include the rollup-plugin-polyfill-node plugin but direct the resolve plugin to prefer built-ins to the browser target if they are needed and available.
With all that in place here is my rollup config:
import commonjs from '#rollup/plugin-commonjs'
import resolve from '#rollup/plugin-node-resolve'
import polyfillNode from 'rollup-plugin-polyfill-node'
export default {
input: 'index.js',
output: {
format: 'iife',
name: 'app',
file: 'bundle.js'
},
plugins: [
commonjs({
requireReturnsDefault: "auto",
}),
polyfillNode(),
resolve({
preferBuiltins: true,
browser: true
}),
]
}
This will bundle. But when I load up the bundle in a browser I get an error about the inherits polyfill not working at this line:
https://github.com/FredKSchott/rollup-plugin-polyfill-node/blob/main/polyfills/inherits.js#L7
It says superCtor is undefined. If I step back a level on the backtrace it comes from this line:
https://github.com/nodejs/readable-stream/blob/main/lib/_stream_duplex.js#L46
The Readable is undefined. When I bundle I get warnings about circular references. I think it fundementally revolves around the fact that some of the PouchDB dependencies use NPM packages as polyfills explicitly (like inherits and readable-stream) while the polyfillNode plugin is also providing some of those same polyfills and they are doing so in an incompatible way. But I don't know how to untangle it.
Finally was able to resolve this so going to provide the answer in case someone else needs to bundle PouchDB memory adapter in Rollup.
The readable streams polyfill is a mess of circular dependencies. This is fine under a CJS format but since Rollup converts CJS to ES format it becomes problematic as Rollup can't figure out the proper order to put things in. This leads to the use of inherits where the superclass is not yet defined. The readable streams polyfill has an open ticket regarding this. The general solution is to fix it upstream in Node and then cut a new copy of the polyfill based on that upstream fix. The upstream fix was made but the polyfill hasn't yet been updated. Once it is this should naturally resolve itself.
In the meantime we can use someone else's fork of the polyfill that has the circular dependencies resolved. This fork is mentioned in that issue but the punchline is to add this to the package.json:
"readable-stream": "npm:vite-compatible-readable-stream#^3.6.0",
This will force readable-stream to resolve to that fork. But that's not the whole story. There are some dependencies of the memory adapter that are locked to a different version of readable-stream. To resolve that we want to add the following to our rollup's resolve plugin:
dedupe: ['readable-stream']
This will force the readable-stream substitute that we have explicitly added to our project to be used anytime a readable stream is needed.
The final issue is that PouchDB is using an ancient version of memdown that also has circular dependency issues. The latest version does not and seems to work with the memory adapter just fine. There is an open ticket at PouchDB to update memdown and this will naturally resolve when that happens.
In the meantime to resolve that we are going follow a similar pattern as above. We will explicitly require memdown into our project although no need for a fork. Just a newer version. Then to force the memory adapter to use this version we add memdown to that dedup option as well.
Related
When I import node:process it works fine. However, when I try to require the same, it gives an error.
This works fine:
import process from 'node:process';
But when I try to require the same, it throws an error:
const process = require('node:process');
Error: Cannot find module 'node:process'
I am curious as to what is the difference between process, which works in both, commonjs and module, vs node:process.
Also, a follow-up, I am using webpack to bundle my js, and I discovered this error when I tried to run my bundled code and realised, that chalk imports node:process, node:os and node:tty. How do I solve that now?
import process from 'node:process'; and import process from 'process'; are equivalent.
The node: exists since version 12 for import.
node: URLs are supported as an alternative means to load Node.js builtin modules. This URL scheme allows for builtin modules to be referenced by valid absolute URL strings.
The idea behind node: is to make clear that it is actually a buildin module and not one installed and to avoid name conflicts, with 3rd party modules.
The node: protocol was first added only for import so a particular node version might support node: with import but not with require.
In v16.13.0 (not sure since which v16 version) you can also use it with require. And was also backported to v14 since v14.18: module: add support for node:‑prefixed require(…) calls
"node:" is an URL scheme for loading ECMAScript modules. As such it started for "import", not "require".
"node:process" is just an alternative name to load the built-in "process" module.
See also Node.js documentation - you can find the lowest supporting Node.js version inside the "History" tag (12.20.0, 14.13.1)
With newer Node.js it should be available for "require" as well (14.18.0, 16.0.0).
Some more details can be found here: node:process always prefers the built-in core module, while process could be loaded from a file.
In a JavaScript project, a file titled index.js can be imported as such:
import SomeComponent from 'components/some-component'
Rather than having to specify index.js.
In my project, I prefer to use a different naming convention: some-component.component.js. This way I can tell what the file is from a glance (rather than having a million index.js).
What I'm trying to achieve is having this same import pattern happen for files with the pattern *.component.js. In other words:
import SomeComponent from 'components/some-component'
Rather than
import SomeComponent from 'components/some-component/some-component.component.js'
I have the following (abbreviated) jsconfig.json:
{
"compilerOptions": {
"baseUrl": "./src",
"paths": {
"*": ["./*"]
}
}
}
Any ideas on how this could be achieved?
Thanks in advance!
Short answer: you can't customize the import resolution using a function (or RegExp). For a web project, you may use a tool like Webpack with a custom loader. But, that is a Webpack specific solution that will not work with other tools (like doing Ctrl-click in VSCode, or using TSC directly).
Long answer:
The resolution of the JavaScript files depends on the tools you use. The standards changed a little bit over the years, and while things are converging to ESM there are still a lot of inconsistencies.
The TypeScript module resolution is described here: https://www.typescriptlang.org/docs/handbook/module-resolution.html
You can have a baseUrl, merge multiple directories as one (rootDirs), or have a limited wildcard path mapping (https://www.typescriptlang.org/docs/handbook/module-resolution.html#path-mapping). You already used it in your question, so probably I'm not telling you anything new. But, that's where the TS configuration options end. (you can customize the compilation module resolution using the TSC API, but not from the CLI/tools config options)
NodeJS can hook a resolution mechanism for modules. For example, Yarn 2+ does that to support a feature called PNP (https://yarnpkg.com/features/pnp). However, TypeScript still needs to resolve types and it will not use your custom loading mechanism for that.
The situation with other tools like Webpack is similar.
The module resolution of Node and TypeScript are specific to those tools. The ESM standard is even more strict: the from part must be an absolute or relative URL, or a "bare identifier" that you need to map using an import map. In other words, you can't have a custom algorithm to resolve paths. The justification for that is described in the Import Maps standard repo: https://github.com/WICG/import-maps#a-programmable-resolution-hook
Conclusion: if you really-really want to do a custom module resolution, you may be able to create your own CLI using the TSC API. That may not work with other tools, and is probably easier to get used to another naming convention.
In an Ember app, when using certain dependencies like moment installed via bower, we have to also import the same in the ember-cli-build.js file:
app.import('bower_components/moment/moment.js');
My question is why is that needed, since I would assume everything inside node_modules as well as bower_components should be available for use inside the app.
Also if that is not the case, how do we identify which dependencies would require such explicit import to be able to use them ?
You don't have to, actually.
There is a package now that lets you 'just import' things: https://github.com/ef4/ember-auto-import
Some reading on the topic of importing: https://discuss.emberjs.com/t/readers-questions-how-far-are-we-from-being-able-to-just-use-any-npm-package-via-the-import-statement/14462?u=nullvoxpopuli
In in-depth answer to your question and the reasons behind why things are the way they are is posted here:
https://discuss.emberjs.com/t/readers-questions-why-does-ember-use-broccoli-and-how-is-it-different-from-webpack-rollup-parcel/15384?u=nullvoxpopuli
(A bit too long for stack overflow, also on mobile, and I wouldn't want to lose all the links and references in a copy-paste)
Hope this helps
Edit:
To answer:
I just wanted to understand "in what cases" do we need to use the import statement in our ember-cli-build (meaning we do not do import for all the dependencies we have in our package/bower.json)...But only for specific ones...I wanted to know what is the criteria or use case for doing import.
Generally, for every package, hence the appeal of the auto-import and / or packagers (where webpack may be used instead of rollup in the future).
Though, it's common for ember-addons to define their own app.import so that you don't need to configure it, like any of these shims, specifically, here is how the c3 charting library is shimmed: https://github.com/mike-north/ember-c3-shim/blob/master/index.js#L7
Importing everything 'manually' like this is a bit of a nuisance, but it is, in part, due to the fact that js packages do not have a consistent distribution format. There is umd, amd, cjs, es6, etc.
with the rollup and broccoli combo, we need to manually specify which format a file is. There are some big advantages to the rollup + broccoli approach, which can be demonstrated here
and here
Sometimes, depending on the transform, you'll need a "vendor-shim".
These are handy when a module has decided it wants to be available on the window / global object instead of available as a module export.
Link: https://simplabs.com/blog/2017/02/13/npm-libs-in-ember-cli.html
(self represents window/global)
however, webpack has already done the work of figuring out how to detect what format a js file is in, and abstracts all of that away from you. webpack is what ember-auto-import uses, and is what allows you to simply
import { stuff} from 'package-name';. The downside to webpack is that you can't pipeline your transforms (which most people may not need, but it can be handy if you're doing Typescript -> Babel -> es5).
Actually: (almost) everything!
Ember does, by default, not add anything to your app except ember addons. There are however some addons that dynamically add stuff to your app like ember-browserify or ember-auto-import.
Also just because you do app.import this does not mean you can use the code with import ... from 'my-package'. The one thing app.import does is it adds the specified file to your vendor.js file. Noting else.
How you use this dependency depends completely on the provided JS file! Ember uses loader.js, an AMD module loader. So if the JS file you app.imported uses AMD (or UMD) this will work and you can import Foo from 'my-package'. (Because this is actually transpiled to AMD imports)
If the provided JS file provides a global you can just use the global.
However there is also the concept of vendor-shims.. Thats basically just a tiny AMD module you can write to export the global as AMD module.
However there are a lot of ember addons that add stuff to your app. For example things like ember-cli-moment-shim just exist to automagically add a dependency to your project. However how it's done completely depends on the addon.
So the rule is:
If its an ember addon read the addon docs but usually you shouldn't app.import
In every other case you manually need to use the library either by app.import or manual broccoli transforms.
The only exception is if you use an addon that tries to generically add dependencies to your project like ember-browserify or ember-auto-import
I'm using Converse.js and it's prebuilt into the RequireJS/AMD syntax. Including the file from a CDN you could use it like require(['converse'], function (converse) { /* .. */ }). How is it possible to use this with webpack? I would like to bundle converse.js with my webpack output.
I have the file on disk, and want to import it like
import converse from './converse.js';
converse.initialize({ .. });
Webpack picks up the file and bundles it correctly, although it's not useable yet as it throws 'initialize is not a function'. What am I missing?
I suspect the way their bundle is built will not work correctly with how Webpack evaluates modules in a limited context.
From their builds, taking the built AMD module via NPM without dependencies should be parsable by Webpack and it will enable you to provide the dependencies to avoid dupes in the final output.
If all else fails, using the script-loader will evaluate the script in the global context and you'd get the same experience as if you'd have followed their usage guidelines to reference it from a CDN, just don't forget to configure the globals for your linter.
I'm trying to use Rollup with Babel's external-helpers. It works, but it's dropping a bunch of babel helpers which I don't even need, for example asyncGenerator.
The docs show a whitelist option but I can't get it to work
rollup.rollup({
entry: 'src/buttonDropdown.es6',
plugins: [
babel({
presets: ['react', ['es2015', { modules: false }], 'stage-2'],
plugins: [['external-helpers', { whitelist: ['asyncGenerator'] }]]
})
]
})
The above has no effect: all Babel helpers are still dropped into my resulting bundle.
What is the correct way of using this feature, and is there a full list of which helpers' names the whitelist array takes?
Or is there some other Rollup plugin I should be using with Rollup to automatically "tree shake" the babel external helpers.
Problem
The babel-plugin-external-helpers plugin is not responsible for injecting those dependencies in the final bundle.
The only thing it controls is that how the generated code will access those functions. For example:
classCallCheck(this, Foo);
// or
babelHelpers.classCallCheck(this, Foo);
It is needed so all rollup-plugin-babel needs to do is to inject babelHelpers in every module.
The documentation is misleading, the whitelist options is not on the external-helpers plugin. It's on the completely separate module and command line tool called babel-external-helpers, which is actually responsible for generating babelHelpers.
It's rollup-plugin-babel what is injecting babelHelpers. And does it using a trick to modularize the final code. It calls babel-external-helpers to generate the helpers, and ignores the whitelist parameter. See my issue requesting to expose an option.
This approach is correct, because rollup will tree-shake the unused helper functions. However some of the helpers (like asyncGenerator) are written in a way that is hard to detect if the initialization has any side effects, thus preventing removal during tree-shaking.
Workaround
I forked rollup-plugin-babel and created a PR which exposes the whitelist option of building babelHelpers in the plugin's options. It can be used this way:
require("rollup").rollup({
entry: "./src/main.js",
plugins: [
require("rollup-plugin-babel")({
"presets": [["es2015", { "modules": false }]],
"plugins": ["external-helpers"],
"externalHelpersWhitelist": ['classCallCheck', 'inherits', 'possibleConstructorReturn']
})
]
}).then(bundle => {
var result = bundle.generate({
format: 'iife'
});
require("fs").writeFileSync("./dist/bundle.js", result.code);
}).then(null, err => console.error(err));
Note that I didn't publish distribution version on npm, you will have to clone the git repo and build it using rollup -c.
Solution
In my opinion the right solution would be to somehow detect or tell rollup that those exports are pure, so can be removed by tree shaking. I will start a discussion about it on github after doing some research.
As I have found in this particular issue in the GitHub page.
The Babel member Hzoo suggests that
Right now the intention of the preset is to allow people to use it without customization - if you want to modify it then you'll have to
just define plugins yourself or make your own preset.
But still if you want to exclude a specific plugin from the default preset then here are some steps.
As suggested by Krucher you can create a fork to the undesirable plugin in the following way
First one is by forking technique
"babel": {
"presets": [
"es2015"
],
"disablePlugins": [
"babel-plugin-transform-es2015-modules-commonjs"
]
}
But if two or more people want to include the es2015-with-commonjs then it would be a problem.For that you have to define your own preset or extend the preset of that module.
The second method would involve the tree-shaking as shown in this article done by Dr. Axel Rauschmayer.
According to the article webpack2 is used with the Babel6.
This helps in removal of the unwanted imports that might have been used anywhere in the project in two ways
First, all ES6 module files are combined into a single bundle file. In that file, exports that were not imported anywhere are not exported, anymore.
Second, the bundle is minified, while eliminating dead code. Therefore, entities that are neither exported nor used inside their modules do not appear in the minified bundle. Without the first step, dead code elimination would never remove exports (registering an export keeps it alive).
Other details can be found in the article.
Simple implemetation is referred as here.
The third method involves creating your own preset for the particular module.
Creating aplugin and greating your own preset can be implemented according to the documentation here
Also as an extra tip you should also use babel-plugin-transforn-runtime
If any of your modules have an external dependancy,the bundle as a whole will have the same external dependancy whether or not you actually used it which may have some side-effects.
There are also a lot of issues with tree shaking of rollup.js as seen in this article
Also as shown in the presets documentation
Enabled by default
These plugins have no effect anymore, as a newer babylon version enabled them by default
- async-functions (since babylon 6.9.1)
- exponentiation-operator (since babylon 6.9.1)
- trailing-function-commas (since babylon 6.9.1)**
Also the concept of whitelisting and blacklisting the plugins has benn brilliantly explained by loganfsmyth here in this thread.
you can pass a whitelist option to specify specific transformations to run, or a blacklist to specific transformations to disable.
You cannot blacklist specific plugins, but you may list only the plugins you want, excluding the ones you do not wish to run.
Update :
According to this article here is an important update -
"The --external-helpers option is now a plugin. To avoid repeated inclusion of Babel’s helper functions, you’ll now need to install and apply the babel-plugin-transform-runtime package, and then require the babel-runtime package within your code (yes, even if you’re using the polyfill)."
Hope this may solve your problem
Hope it may help you.