Develop Reference

How to migrate from background page context design to service worker's passing messages between other contexts design?

I'm very green when it comes to working with service workers and I'm trying to understand how to migrate from functions such as chrome.extension.getBackgroundPage() which most of the extensions I'm working with use, to a design that passes messages between contexts and the service worker according to Google's checklist; but there isn't much of an example to go off of.
Here is a small example of the current design I have. I have a main js page that imports scripts for methods which acts as a global scope when called by getBackgroundPage() on other files. This file also does some other things, but I'm keeping it brief.
Main.js
importScripts("js/store.js", "js/config.js", "js/utility.js", "js/log.js", "js/search.js")
var _Store = _Store || new Store();
var _Config = _Config || new Config(_Store);
let Util = new Utility(_Config, _Store);
I have another file that utilizes Main.js methods, and attempts to build the url on each new tab opening. In this example, its the _Config:
URLBuild.js
let Background = chrome.extension.getBackgroundPage();
let NTURL = '';
if (Background) {
if (Background._Config) {
NTURL = Background._Config.getNTURL();
}
}
if (typeof (NTURL) !== 'undefined' && NTURL !== '' && NTURL !== null) {
NTLocation(chrome.extension.getBackgroundPage()._Config.getNTURL());
}
function NTLocation(url) {
if (typeof (url) !== 'undefined' && url !== '' && url.indexOf('https:') > -1) {
chrome.tabs.create({ "url": url + "&iid=myid_" + chrome.runtime.getManifest().version + "&" });
} else {
chrome.tabs.create({ "url": FallBackMethod() });
}
}
From my understanding, I'm trying to simulate how this is done via service workers by passing messages but I feel that this might be the wrong direction I'm taking from my lack of understanding. Here's my attempt at that:
Main.js
//"server"
addEventListener('message', event => {
console.log(`The client sent me a message: ${event.data}`)
event.source.postMessage(_Config);
})
URLBuild.js
// "client"
let myConfig;
if (navigator.serviceWorker) {
navigator.serviceWorker.register('service-wrapper.js');
navigator.serviceWorker.ready.then(registration => {
registration.active.postMessage("Ask for Config Object");
});
navigator.serviceWorker.addEventListener('message', event => {
myConfig = event.data;
});
}
If this is the incorrect way of approaching things, what does the design of passing messages between other contexts and the service worker look like?

Generate tests based on the result of a promise in Jest JS

Right now I have a simple.test.js file that generates calls to test based on simplified call/response files (so we don't need to write a .test.js for each of these simplified cases). For reference I'll include the file here:
'use strict';
const api = require('./api');
const SCRIPT_NAME_KEY = Symbol('script name key'),
fs = require('fs'),
path = require('path');
const generateTests = (dir) => {
const relPath = path.relative(__dirname, dir);
let query, resultScripts = [], resultSqls = [];
for (let entry of fs.readdirSync(dir)) {
if (entry[0] === '-')
continue;
let fqEntry = path.join(dir, entry);
if (fs.statSync(fqEntry).isDirectory()) {
generateTests(fqEntry);
continue;
}
if (entry === 'query.json')
query = fqEntry;
else if (entry.endsWith('.sql'))
resultSqls.push(fqEntry);
else if (entry.endsWith('.js') && !entry.endsWith('.test.js'))
resultScripts.push(fqEntry);
}
if (!query && resultScripts.length === 0 && resultSqls.length === 0)
return;
if (!query)
throw `${relPath} contains result script(s)/sql(s) but no query.json`;
if (resultScripts.length === 0 && resultSqls.length === 0)
throw `${relPath} contains a query.json file but no result script(s)/sql(s)`;
try {
query = require(query);
} catch (ex) {
throw `${relPath} query.json could not be parsed`;
}
for (let x = 0; x < resultScripts.length; x++) {
let scriptName = path.basename(resultScripts[x]);
console.log('scriptName', scriptName);
try {
resultScripts[x] = require(resultScripts[x]);
} catch (ex) {
throw `${relPath} result script ${scriptName} could not be parsed`;
}
resultScripts[x][SCRIPT_NAME_KEY] = scriptName;
}
test(`ST:${relPath}`, () => api.getSqls(query).then(resp => {
if (resultScripts.length === 0) {
expect(resp.err).toBeFalsy();
expect(resp.data).toBeAllValidSql();
} else {
for (const script of resultScripts)
expect({ n: script[SCRIPT_NAME_KEY], r: script(resp, script[SCRIPT_NAME_KEY]) }).toPass();
}
for (const sql of resultSqls)
expect(resp.data).toIncludeSql(fs.readFileSync(sql, 'utf8'));
}));
};
expect.extend({
toPass(actual) {
const pass = actual.r === void 0 || actual.r === null || !!actual.r.pass;
return {
pass: pass,
message: pass ? null : () => actual.r.message || `${actual.n} check failed!`
}
}
});
generateTests(path.join(__dirname, 'SimpleTests'));
This works really great! It runs immediately when the .test.js file is loaded by Jest and generates a test for each folder containing the valid files.
However, I now have a need to generate a test per record in a database. From what I can tell most of the available modules that provide DB functionality work on the premise of promises (and reasonably so!). So now I need to wait for a query to come back BEFORE I generate the tests.
This is what I'm trying:
'use strict';
const api = require('./api');
api.getAllReportsThroughSideChannel().then((reports) => {
for (const report of reports) {
test(`${report.Name} (${report.Id} - ${report.OwnerUsername})`, () => {
// ...
});
}
});
However when I do this I get:
FAIL ./reports.test.js
● Test suite failed to run
Your test suite must contain at least one test.
at ../node_modules/jest/node_modules/jest-cli/build/TestScheduler.js:256:22
As one might expect, the promise gets created but doesn't get a chance to actually trigger the generation of tests until after Jest has already expected to receive a list of tests from the file.
One thing I considered was to have a test that itself is a promise that checks out all the reports, but then it would fail on the first expect that results in a failure, and we want to get a list of all reports that fail tests. What we really want is a separate test for each.
I guess ultimately the question I want to know is if it is possible for the generation of tests to be done via a promise (rather then the tests themselves).
There is a TON of resources for Jest out there, after searching I didn't find anything that applies to my question, so apologies if I just missed it somehow.

Ok, after a few days of looking through docs, and code, it's looking more and more like this simply can not be done in Jest (or probably more correctly, it goes counter to Jest's testing philosophies).
As such, I have created a step prior to running the jest runtime proper, that simply downloads the results of the query to a file, then I use the file to synchronously generate the test cases.
I would LOVE it if someone can propose a better solution though.

How to rollup multiple directories separately with one output

So, I have a directory:
mods/
-core/
--index.js
--scripts/
---lots of stuff imported by core/index
This works with typical rollup fashion if you want to rollup to for example mods/core/index.min.js
But I have many of these mods/**/ directories and I want to take advantage of the fact that they are rollup'd into iifes. Each mods/**/index.js will, rather than export, assign to a global variable that we presume is provided:
mods/core/index.js
import ui from './scripts/ui/'
global.ui = ui
mods/someMod/scripts/moddedClass.js
export default class moddedClass extends global.ui.something { /* some functionality extension */}
mods/someMod/index.js
import moddedClass from './scripts/moddedClass'
global.ui.something = moddedClass
So hopefully you can see how each mod directory can be rollup'd in typical fashion but, I need to then put the actual iifes inside another one so that:
mods/compiled.js
(function compiled() {
const global = {};
(function core() {
//typical rollup iife
})();
(function someMod() {
//typical rollup iife
})();
//a footer like return global or global.init()
})();
Any help towards this end would be greatly appreciated. The simplest possible answer, I think, is how I can simply get a string value for each mod's iife instead of rollup writing it to a file.
At that point I could just iterate the /mods/ directory, in an order specified by some modlist.json or something, and call rollup on each /mod/index.js, then build the outer iife myself from strings.
However, I suppose this would not be a full solution for sourcemapping? Or can multiple inline sourcemaps be included? With source mapping in mind, I wonder if another build step might be necessary, where each mod is transpiled before this system even gets to it.

Use rollup's bundle.generate api to generate multiple iifes and write them into one file using fs.appendFile.
For the sourcemaps you can use this module（it's from the same author of rollup) https://github.com/rich-harris/sorcery

Okay so the way I ended up solving this was using source-map-concat
It basically does what I described, right out of the box. The only thing I had to do was to asynchronously iterate the mod directory and rollup each mod, before passing the results to source-map-concat, since rollup.rollup returns a Promise.
I also ended up wanting in-line sourcemaps so that the code can be directly injected rather than written to a file, so I used convert-source-map for that.
The only issue left to solve is sub-source mapping. Sorcery would work great for that, if I was generating files, but I would like to keep it as string sources. For now it will at least show me what mod an error came from, but not the sub-file it came from. If anyone has info on how to do a sorcery-style operation on strings let me know.
Here's the relevant final code from my file:
const rollup = require("rollup")
const concat = require("source-map-concat")
const convert = require("convert-source-map")
const fs = require("fs")
const path = require("path")
const modsPath = path.join(__dirname, "mods")
const getNames = _ => JSON.parse(fs.readFileSync(path.join(modsPath, "loadList.json"), "utf8"))
const wrap = (node, mod) => {
node.prepend("\n// File: " + mod.source + "\n")
}
const rolls = {}
const bundles = {}
const rollupMod = (modName, after) => {
let dir = path.join(modsPath, modName),
file = path.join(dir, "index.js")
rollup.rollup({
entry: file,
external: "G",
plugins: []
}).then(bundle => {
rolls[modName] = bundle.generate({
format: "iife",
moduleName: modName,
exports: "none",
useStrict: false,
sourceMap: true
})
after()
})
}
const rollupMods = after => {
let names = getNames(), i = 0,
rollNext = _ => rollupMod(names[i++], _ => i < names.length - 1? rollNext() : after())
rollNext()
}
const bundleCode = after => {
rollupMods(_ => {
let mods = concat(getNames().map(modName => {
let mod = rolls[modName]
return {
source: path.join(modsPath, modName),
code: mod.code,
map: mod.map
}
}), {
delimiter: "\n",
process: wrap
})
mods.prepend("(function(){\n")
mods.add("\n})();")
let result = mods.toStringWithSourceMap({
file: path.basename('.')
})
bundles.code = result.code + "\n" + convert.fromObject(result.map).toComment()
after(bundles.code)
})
}
exports.bundleCode = bundleCode

Listen for hot update events on the client side with webpack-dev-derver?

This is a bit of an edge case but it would be helpful to know.
When developing an extension using webpack-dev-server to keep the extension code up to date, it would be useful to listen to "webpackHotUpdate"
Chrome extensions with content scripts often have two sides to the equation:
Background
Injected Content Script
When using webpack-dev-server with HMR the background page stays in sync just fine. However content scripts require a reload of the extension in order to reflect the changes. I can remedy this by listening to the "webpackHotUpdate" event from the hotEmmiter and then requesting a reload. At present I have this working in a terrible and very unreliably hacky way.
var hotEmitter = __webpack_require__(XX)
hotEmitter.on('webpackHotUpdate', function() {
console.log('Reloading Extension')
chrome.runtime.reload()
})
XX simply represents the number that is currently assigned to the emitter. As you can imagine this changed whenever the build changes so it's a very temporary proof of concept sort of thing.
I suppose I could set up my own socket but that seems like overkill, given the events are already being transferred and I simply want to listen.
I am just recently getting more familiar with the webpack ecosystem so any guidance is much appreciated.

Okay!
I worked this out by looking around here:
https://github.com/facebookincubator/create-react-app/blob/master/packages/react-dev-utils/webpackHotDevClient.js
Many thanks to the create-react-app team for their judicious use of comments.
I created a slimmed down version of this specifically for handling the reload condition for extension development.
var SockJS = require('sockjs-client')
var url = require('url')
// Connect to WebpackDevServer via a socket.
var connection = new SockJS(
url.format({
// Default values - Updated to your own
protocol: 'http',
hostname: 'localhost',
port: '3000',
// Hardcoded in WebpackDevServer
pathname: '/sockjs-node',
})
)
var isFirstCompilation = true
var mostRecentCompilationHash = null
connection.onmessage = function(e) {
var message = JSON.parse(e.data)
switch (message.type) {
case 'hash':
handleAvailableHash(message.data)
break
case 'still-ok':
case 'ok':
case 'content-changed':
handleSuccess()
break
default:
// Do nothing.
}
}
// Is there a newer version of this code available?
function isUpdateAvailable() {
/* globals __webpack_hash__ */
// __webpack_hash__ is the hash of the current compilation.
// It's a global variable injected by Webpack.
return mostRecentCompilationHash !== __webpack_hash__
}
function handleAvailableHash(data){
mostRecentCompilationHash = data
}
function handleSuccess() {
var isHotUpdate = !isFirstCompilation
isFirstCompilation = false
if (isHotUpdate) { handleUpdates() }
}
function handleUpdates() {
if (!isUpdateAvailable()) return
console.log('%c Reloading Extension', 'color: #FF00FF')
chrome.runtime.reload()
}
When you are ready to use it (during development only) you can simply add it to your background.js entry point
module.exports = {
entry: {
background: [
path.resolve(__dirname, 'reloader.js'),
path.resolve(__dirname, 'background.js')
]
}
}
For actually hooking into the event emitter as was originally asked you can just require it from webpack/hot/emitter since that file exports an instance of the EventEmitter that's used.
if(module.hot) {
var lastHash
var upToDate = function upToDate() {
return lastHash.indexOf(__webpack_hash__) >= 0
}
var clientEmitter = require('webpack/hot/emitter')
clientEmitter.on('webpackHotUpdate', function(currentHash) {
lastHash = currentHash
if(upToDate()) return
console.log('%c Reloading Extension', 'color: #FF00FF')
chrome.runtime.reload()
})
}
This is just a stripped down version straight from the source:
https://github.com/webpack/webpack/blob/master/hot/dev-server.js

I've fine-tuned the core logic of the crx-hotreload package and come up with a build-tool agnostic solution (meaning it will work with Webpack but also with anything else).
It asks the extension for its directory (via chrome.runtime.getPackageDirectoryEntry) and then watches that directory for file changes. Once a file is added/removed/changed inside that directory, it calls chrome.runtime.reload().
If you'd need to also reload the active tab (when developing a content script), then you should run a tabs.query, get the first (active) tab from the results and call reload on it as well.
The whole logic is ~35 lines of code:
/* global chrome */
const filesInDirectory = dir => new Promise(resolve =>
dir.createReader().readEntries(entries =>
Promise.all(entries.filter(e => e.name[0] !== '.').map(e =>
e.isDirectory
? filesInDirectory(e)
: new Promise(resolve => e.file(resolve))
))
.then(files => [].concat(...files))
.then(resolve)
)
)
const timestampForFilesInDirectory = dir => filesInDirectory(dir)
.then(files => files.map(f => f.name + f.lastModifiedDate).join())
const watchChanges = (dir, lastTimestamp) => {
timestampForFilesInDirectory(dir).then(timestamp => {
if (!lastTimestamp || (lastTimestamp === timestamp)) {
setTimeout(() => watchChanges(dir, timestamp), 1000)
} else {
console.log('%c 🚀 Reloading Extension', 'color: #FF00FF')
chrome.runtime.reload()
}
})
}
// Init if in dev environment
chrome.management.getSelf(self => {
if (self.installType === 'development' &&
'getPackageDirectoryEntry' in chrome.runtime
) {
console.log('%c 📦 Watching for file changes', 'color: #FF00FF')
chrome.runtime.getPackageDirectoryEntry(dir => watchChanges(dir))
}
})
You should add this script to your manifest.json file's background scripts entry:
"background": ["reloader.js", "background.js"]
And a Gist with a light explanation in the Readme: https://gist.github.com/andreasvirkus/c9f91ddb201fc78042bf7d814af47121

How do I require() from the console using webpack?

How do I require() / import modules from the console? For example, say I've installed the ImmutableJS npm, I'd like to be able to use functions from the module while I'm working in the console.

Here's another more generic way of doing this.
Requiring a module by ID
The current version of WebPack exposes webpackJsonp(...), which can be used to require a module by ID:
function _requireById(id) {
return webpackJsonp([], null, [id]);
}
or in TypeScript
window['_requireById'] =
(id: number): any => window['webpackJsonp'];([], null, [id]);
The ID is visible at the top of the module in the bundled file or in the footer of the original source file served via source maps.
Requiring a module by name
Requiring a module by name is much trickier, as WebPack doesn't appear to keep any reference to the module path once it has processed all the sources. But the following code seems to do the trick in lot of the cases:
/**
* Returns a promise that resolves to the result of a case-sensitive search
* for a module or one of its exports. `makeGlobal` can be set to true
* or to the name of the window property it should be saved as.
* Example usage:
* _requireByName('jQuery', '$');
* _requireByName('Observable', true)´;
*/
window['_requireByName'] =
(name: string, makeGlobal?: (string|boolean)): Promise<any> =>
getAllModules()
.then((modules) => {
let returnMember;
let module = _.find<any, any>(modules, (module) => {
if (_.isObject(module.exports) && name in module.exports) {
returnMember = true;
return true;
} else if (_.isFunction(module.exports) &&
module.exports.name === name) {
return true;
}
});
if (module) {
module = returnMember ? module.exports[name] : module.exports;
if (makeGlobal) {
const moduleName = makeGlobal === true ? name : makeGlobal as string;
window[moduleName] = module;
console.log(`Module or module export saved as 'window.${moduleName}':`,
module);
} else {
console.log(`Module or module export 'name' found:`, module);
}
return module;
}
console.warn(`Module or module export '${name}'' could not be found`);
return null;
});
// Returns promise that resolves to all installed modules
function getAllModules() {
return new Promise((resolve) => {
const id = _.uniqueId('fakeModule_');
window['webpackJsonp'](
[],
{[id]: function(module, exports, __webpack_require__) {
resolve(__webpack_require__.c);
}},
[id]
);
});
}
This is quick first shot at this, so it's all up for improvement!

Including this in a module will allow require([modules], function) to be used from a browser
window['require'] = function(modules, callback) {
var modulesToRequire = modules.forEach(function(module) {
switch(module) {
case 'immutable': return require('immutable');
case 'jquery': return require('jquery');
}
})
callback.apply(this, modulesToRequire);
}
Example Usage:
require(['jquery', 'immutable'], function($, immutable) {
// immutable and $ are defined here
});
Note: Each switch-statement option should either be something this module already requires, or provided by ProvidePlugin
Sources:
Based on this answer, which can be used to add an entire folder.
Alternative method from Webpack Docs - which allows something like require.yourModule.function()

I found a way that works, for both WebPack 1 and 2. (as long as the source is non-minified)
Repo: https://github.com/Venryx/webpack-runtime-require
Install
npm install --save webpack-runtime-require
Usage
First, require the module at least once.
import "webpack-runtime-require";
It will then add a Require() function to the window object, for use in the console, or anywhere in your code.
Then just use it, like so:
let React = Require("react");
console.log("Retrieved React.Component: " + React.Component);
It's not very pretty (it uses regexes to search the module wrapper functions) or fast (takes ~50ms the first call, and ~0ms after), but both of these are perfectly fine if it's just for hack-testing in the console.
Technique
The below is a trimmed version of the source to show how it works. (see the repo for the full/latest)
var WebpackData;
webpackJsonp([],
{123456: function(module, exports, __webpack_require__) {
WebpackData = __webpack_require__;
}},
[123456]
);
var allModulesText;
var moduleIDs = {};
function GetIDForModule(name) {
if (allModulesText == null) {
let moduleWrapperFuncs = Object.keys(WebpackData.m).map(moduleID=>WebpackData.m[moduleID]);
allModulesText = moduleWrapperFuncs.map(a=>a.toString()).join("\n\n\n");
// these are examples of before and after webpack's transformation: (which the regex below finds the var-name of)
// require("react-redux-firebase") => var _reactReduxFirebase = __webpack_require__(100);
// require("./Source/MyComponent") => var _MyComponent = __webpack_require__(200);
let regex = /var ([a-zA-Z_]+) = __webpack_require__\(([0-9]+)\)/g;
let matches = [];
let match;
while (match = regex.exec(allModulesText))
matches.push(match);
for (let [_, varName, id] of matches) {
// these are examples of before and after the below regex's transformation:
// _reactReduxFirebase => react-redux-firebase
// _MyComponent => my-component
// _MyComponent_New => my-component-new
// _JSONHelper => json-helper
let moduleName = varName
.replace(/^_/g, "") // remove starting "_"
.replace(new RegExp( // convert chars where:
"([^_])" // is preceded by a non-underscore char
+ "[A-Z]" // is a capital-letter
+ "([^A-Z_])", // is followed by a non-capital-letter, non-underscore char
"g"),
str=>str[0] + "-" + str[1] + str[2] // to: "-" + char
)
.replace(/_/g, "-") // convert all "_" to "-"
.toLowerCase(); // convert all letters to lowercase
moduleIDs[moduleName] = parseInt(id);
}
}
return moduleIDs[name];
}
function Require(name) {
let id = GetIDForModule(name);
return WebpackData.c[id].exports;
}

Being able to use require modules in the console is handy for debugging and code analysis. #psimyn's answer is very specific so you aren't likely to maintain that function with all the modules you might need.
When I need one of my own modules for this purpose, I assign a window property to it so I can get at it e.g window.mymodule = whatever_im_exporting;. I use the same trick to expose a system module if I want to play with it e.g:
myservice.js:
let $ = require('jquery');
let myService = {};
// local functions service props etc...
module.exports = myService;
// todo: remove these window prop assignments when done playing in console
window.$ = $;
window.myService = myService;
It is still a bit of a pain, but digging into the bundles, I can't see any way to conveniently map over modules.

The answer from #Rene Hamburger is good but unfortunately doesn't work anymore (at least with my webpack version). So I updated it:
function getWebpackInternals() {
return new Promise((resolve) => {
const id = 'fakeId' + Math.random();
window['webpackJsonp'].push(["web", {
[id]: function(module, __webpack_exports__, __webpack_require__) {
resolve([module, __webpack_exports__, __webpack_require__])
}
},[[id]]]);
});
}
function getModuleByExportName(moduleName) {
return getWebpackInternals().then(([_, __webpack_exports__, __webpack_require__]) => {
const modules = __webpack_require__.c;
const moduleFound = Object.values(modules).find(module => {
if (module && module.exports && module.exports[moduleName]) return true;
});
if (!moduleFound) {
console.log('couldnt find module ' + moduleName);
return;
}
return moduleFound.exports[moduleName];
})
}
getModuleByExportName('ExportedClassOfModule');

expose-loader is, in my opinion, a more elegant solution:
require("expose-loader?libraryName!./file.js");
// Exposes the exports for file.js to the global context on property "libraryName".
// In web browsers, window.libraryName is then available.

Adding the below code to one of your modules will allow you to load modules by id.
window.require = __webpack_require__;
In the console use the following:
require(34)

You could do something similar as psimyn advised by
adding following code to some module in bundle:
require.ensure([], function () {
window.require = function (module) {
return require(module);
};
});
Use require from console:
require("./app").doSomething();
See more

After making an npm module for this (see my other answer), I did a search on npms.io and seem to have found an existing webpack-plugin available for this purpose.
Repo: https://www.npmjs.com/package/webpack-expose-require-plugin
Install
npm install --save webpack-expose-require-plugin
Usage
Add the plugin to your webpack config, then use at runtime like so:
let MyComponent = require.main("./path/to/MyComponent");
console.log("Retrieved MyComponent: " + MyComponent);
See package/repo readme page for more info.
EDIT
I tried the plugin out in my own project, but couldn't get it to work; I kept getting the error: Cannot read property 'resource' of undefined. I'll leave it here in case it works for other people, though. (I'm currently using the solution mentioned above instead)

After both making my own npm package for this (see here), as well as finding an existing one (see here), I also found a way to do it in one-line just using the built-in webpack functions.
It uses WebPack "contexts": https://webpack.github.io/docs/context.html
Just add the following line to a file directly in your "Source" folder:
window.Require = require.context("./", true, /\.js$/);
Now you can use it (eg. in the console) like so:
let MyComponent = Require("./Path/To/MyComponent");
console.log("Retrieved MyComponent: " + MyComponent);
However, one important drawback of this approach, as compared to the two solutions mentioned above, is that it seems to not work for files in the node_modules folder. When the path is adjusted to "../", webpack fails to compile -- at least in my project. (perhaps because the node_modules folder is just so massive)