Develop Reference

Lerna publish gets stuck after first package

Current Behavior
When I run lerna publish, it gets stuck after packaging the first package. This is where it gets stuck:
lerna WARN ENOLICENSE One way to fix this is to add a LICENSE.md file to the root of this repository.
lerna WARN ENOLICENSE See https://choosealicense.com for additional guidance.
(#########⠂⠂⠂⠂⠂⠂⠂⠂⠂) ⠏ publish: verb packed packages/react
This line particularly:
(#########⠂⠂⠂⠂⠂⠂⠂⠂⠂) ⠏ publish: verb packed packages/react
Expected Behavior
It should publish to NPM successfully
Failure Logs / Configuration
lerna.json
{
"packages": [
"packages/*",
"docs"
],
"version": "independent",
"stream": true,
"hoist": true,
"command": {
"bootstrap": {
"npmClientArgs": ["--no-package-lock"]
},
"publish": {
"ignoreChanges": ["**/stories/**", "**/tests/**"]
}
}
}
Environment
System:
OS: macOS 12.3
CPU: (8) arm64 Apple M1 Pro
Binaries:
Node: 14.21.2 - ~/.nvm/versions/node/v14.21.2/bin/node
npm: 6.14.17 - ~/.nvm/versions/node/v14.21.2/bin/npm
Utilities:
Git: 2.32.1 - /usr/bin/git
npmPackages:
lerna: ^5.6.2 => 5.6.2

Lerna publish workflow not publishing as expected

For smoother CI experience, I have made a Github Action workflow to publish the monorepo packages with a prerelease version, everytime any member opens a PR against master with a particular label 'publish'. This workflow
should ideally publish all the changed packages since the last publish with a preid -pr{pr#} e.g., package-pr1049.0. Have added dist-tag and predist-tag also here.
Background:
Before publishing the packages, I also run a MAKE executable(make -j init script) to clean and bootstrap all the packages. Post this, it will fetch the repo, checkout to the required branch, and run the publish command with the PR number parameter.
There are 2 problems I am facing in this workflow:
Publishes all the packages in the first commit of the PR:
To debug the issue , Had also added a logger to see if it has the correct record for last 10 commits, which reflects the correct set of commits.
Second commit onwards, only the changed packages are published which is as expected. refer the log
lerna notice cli v3.20.2
lerna info versioning independent
lerna info ci enabled
lerna info Assuming all packages changed
lerna info getChangelogConfig Successfully resolved preset "conventional-changelog-angular"
Changes:
- #swiggy-private/package-1: 1.0.4 => 1.1.0-pr19000.0
- #swiggy-private/package-2: 1.1.4 => 1.2.0-pr19000.0
- #swiggy-private/package-3: 2.41.2 => 2.42.0-pr19000.0 (private)
Always updates the patch version as 0, and increments the preid (PR number with -pr prefix) PACKAGEv0-pr{##}.0, PACKAGEv0-pr{##}.1, ....
For speeding up debugging process, have limited monorepo packages in lerna.json to only 3 of the packages.
My GH workflow
name: Branch Publish
on:
pull_request:
types: [opened, synchronize, reopened, labeled]
branches:
- master
jobs:
check:
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
author: ${{ steps.step1.outputs.author }}
steps:
- uses: actions/checkout#v2
with:
ref: ${{ github.event.pull_request.head.sha }}
- id: "step1"
run: |
AUTHOR_NAME=$(git show ${{ github.event.pull_request.head.sha }} | grep Author)
echo "::set-output name=author::$AUTHOR_NAME"
init:
if: "!contains(needs.check.outputs.author, 'GitHub Action Branch') && !contains(github.event.head_commit.message, '[skip ci]')"
runs-on: ubuntu-latest
timeout-minutes: 15
needs: [check]
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v1
with:
node-version: "12.x"
- run: git fetch --prune --unshallow
- run: |
make -j init
env:
NPM_TOKEN: ${{ secrets.GH_TOKEN }}
- uses: actions/cache#v1
id: cache-build
with:
path: "."
key: ${{ github.sha }}
release:
if: "contains(github.event.pull_request.labels.*.name, 'publish')"
runs-on: ubuntu-latest
timeout-minutes: 15
needs: [init]
steps:
- uses: actions/checkout#v2
with:
fetch-depth: "0"
- uses: actions/setup-node#v1
with:
node-version: "12.x"
- uses: actions/cache#v1
id: restore-build
with:
path: "."
key: ${{ github.sha }}
- name: Setup Git
uses: webfactory/ssh-agent#v0.4.1
with:
ssh-private-key: ${{ secrets.GHA_DEPLOY_KEY }}
- name: Lerna Publish
if: success()
env:
GH_TOKEN: ${{ secrets.GH_TOKEN }}
NODE_ENV: production
run: |
git config user.email "action#github.com"
git config user.name "GitHub Action Branch"
git remote set-url origin "git#github.com:${{ github.repository }}"
git fetch --depth=1 origin +refs/tags/*:refs/tags/*
git checkout -- .
git log --pretty=oneline -n 10
git checkout --track origin/$(echo $GITHUB_HEAD_REF | cut -d'/' -f 3)
NUMBER=${{ github.event.number }} npm run publish-branch
- name: Possible Package lock update
if: success()
run: |
git config user.email "action#github.com"
git config user.name "GitHub Action Branch"
git remote set-url origin "git#github.com:${{ github.repository }}"
npx lerna clean -y
npx lerna exec -- npm i --package-lock-only --ignore-scripts --no-audit
echo `git add . && git commit -m "chore: package lock update" --no-verify && git push`
Publish command
"publish-branch": "lerna publish --conventional-prerelease --exact --no-changelog --preid pr$NUMBER --dist-tag beta --pre-dist-tag beta --no-verify-access --yes"
Lerna.json
{
"packages": ["*"],
"version": "independent",
"command": {
"publish": {
"npmClient": "npm",
"graphType": "all",
"allowBranch": ["master", "integration", "*"],
"conventionalCommits": true,
"message": "chore(release): publish",
"includeMergedTags": true,
"ignoreChanges": ["**/__tests__/**", "**/*.md"]
}
}
}
Make Script to bootstrap packages
init: clean-all
$(MAKE) create-npmrc-all
npm ci
npm run bootstrap:ci
NODE_ENV=production npm run prepare:all
create-npmrc-all:
echo $(GITHUB_SCOPE_REGISTRY) >> .npmrc
echo $(GITHUB_REGISTRY_TOKEN) >> .npmrc
$(foreach source, $(DIRECTORY), $(call pass-to-npmrc, $(source), $(GITHUB_SCOPE_REGISTRY)))
$(foreach source, $(DIRECTORY), $(call pass-to-npmrc, $(source), $(GITHUB_REGISTRY_TOKEN)))
clean-all:
rm -rf node_modules
$(foreach source, $(SOURCES), \
$(call clean-source-all, $(source)))
rm -rf .npmrc
rm -rf packages/*/.npmrc
rm -rf coverage
rm -rf packages/*/npm-debug*

Instead of creating script from .npmrc file you can just provide those data directly in the pipeline, those 2 line are perfectly working even with the github predefined token. Not really sure what environment you are using, but I think you'll get the point.
name: Setting Up NPM
run: |
npm set#organization:registry=https://npm.pkg.github.com/organization
npm set "//npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}"

Format argument value before passing to a yarn/npm script

I have a storybook start script which I want to run for some specific folder:
"storybook": "start-storybook -p 6006 -s ./src"
This loads all stories from src folder. As the amount of stories becomes larger, I want to run stories only from some of subfolders:
start-storybook -p 6006 -s ./src/components/CommonComponents
start-storybook -p 6006 -s ./src/components/DashboardComponents
How can I format argument value dynamically in order to start storybook like this below?
$ yarn storybook CommonComponents
And it would turn into:
start-storybook -p 6006 -s ./src/components/CommonComponents

storybook task could be a script, and then inside the script you parse the arguments, and call start-storybook
Create a task in package.json (e.q run-storybook) and set it to execute the custom script:
"run-storybook": yarn ./path/to/script.js
#!/bin/env node
// script.js
const { spawn } = require('child_process')
const args = process.argv.slice(2)
const port = args[1]
const component = args[3]
const path = `./src/components/${component}`
spawn('yarn', ['start-storybook', '-p', port, '-s', path], {
stdio: 'inherit',
shell: true
})
Then you can call: yarn run-storybook -p 600 -s yourcomponent
Note: make sure the script is executable: chmod +x /path/to/script.js.

Can we cache yarn globals in github actions

I have some global packages such as serverless framework, ESLint and etc. I've implemented GitHub Actions cache for yarn. Below is my code.
- uses: actions/cache#v1
id: yarn-cache # use this to check for `cache-hit` (`steps.yarn-cache.outputs.cache-hit != 'true'`)
with:
path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-
- name: Adding serverless globally
run: yarn global add serverless
- name: Yarn Install
if: steps.yarn-cache.outputs.cache-hit != 'true'
run: |
echo "cache hit failed"
yarn install
env:
CI: false
But my global packages are not cached. Is there any way to cache Yarn globals?

I'm pasting the build file for the solution,
name: global-test
on:
push:
branches:
- dev
pull_request:
branches:
- dev
jobs:
aws-deployment:
runs-on: ubuntu-latest
steps:
- name: CHECKOUT ACTION
uses: actions/checkout#v2
- name: NODE SETUP ACTION
uses: actions/setup-node#v1
with:
node-version: '12.x'
- name: Get yarn cache directory path
id: yarn-cache-dir-path
run: |
echo "::set-output name=dir::$(yarn cache dir)"
- name: Set yarn global bin path
run: |
yarn config set prefix $(yarn cache dir)
- name: Add yarn bin path to system path
run: |
echo $(yarn global bin) >> $GITHUB_PATH
- name: Set yarn global installation path
run: |
yarn config set global-folder $(yarn cache dir)
- name: CACHE ACTION
uses: actions/cache#v2
env:
cache-version: v1
id: yarn-cache
with:
path: |
${{ steps.yarn-cache-dir-path.outputs.dir }}
**/node_modules
key: ${{ runner.os }}-yarn-${{ env.cache-version }}-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-${{ env.cache-version }}-
${{ runner.os }}-yarn-
${{ runner.os }}-
- name: Installing dependencies
if: steps.yarn-cache.outputs.cache-hit != 'true'
run: |
echo "YARN CACHE CHANGED"
yarn install
- name: Adding serverless globally
if: steps.yarn-cache.outputs.cache-hit != 'true'
run: |
echo "NO CACHE HIT"
yarn global add serverless
I named the steps, so they can be understood.
UPDATED the answer on 2020-12-06

nightwatchjs parallel mode selenium hub docker compose

I'm trying to run tests in parallel written using nightwatchjs in Docker using Selenium Hub. I'm able to get the tests to run in parallel in Docker without Selenium Hub, however, some child processes will timeout causing multiple retries. The results are very inconsistent. I'm hoping to use Selenium Hub or something similar to remove the timeouts and retries so the test results are more consistent, stable, and do not timeout.
However, now when I run docker-compose run --rm nightwatch, using the following code, the selenium server will start in parallel mode and multiple child processes will be started, however, only the first one will execute. Then the other child processes will get Error retrieving a new session from the selenium server. Connection refused! Is selenium server started? Am I missing something to get the nightwatchjs tests to run in parallel without timing out?
nightwatch.conf.js
module.exports = {
src_folders: ['tests'],
output_folder: 'reports',
custom_commands_path: '',
custom_assertions_path: '',
page_objects_path: 'page_objects',
test_workers: true,
live_output: true,
detailed_output: true,
selenium: {
start_process: true,
server_path: './bin/selenium-server-standalone-3.0.1.jar',
log_path: '',
host: '127.0.0.1',
port: 4444,
cli_args: {
'webdriver.chrome.driver' : './node_modules/chromedriver/bin/chromedriver'
}
},
test_settings: {
default: {
launch_url: 'https://example.com',
selenium_port: 4444,
selenium_host: 'hub',
silent: true,
screenshots: {
'enabled': false,
'path': ''
},
desiredCapabilities: {
browserName: 'chrome',
javascriptEnabled: true,
acceptSslCerts: true,
chromeOptions: {
args: [
'--window-size=1024,768',
'--no-sandbox'
]
}
},
globals: {
waitForConditionTimeout: 20000,
asyncHookTimeout: 70000
}
}
};
docker-compose.yml
version: '2'
services:
nightwatch:
build:
context: .
command: /bin/sh -c "node ./node_modules/nightwatch/bin/nightwatch"
links:
- chrome
- hub
volumes:
- .:/opt/nightwatch
chrome:
environment:
VIRTUAL_HOST: node.chrome.docker
HUB_PORT_4444_TCP_ADDR: hub
HUB_PORT_4444_TCP_PORT: 4444
image: selenium/node-chrome:3.1.0-astatine
links:
- hub
hub:
ports:
- 4444:4444
image: selenium/hub:3.1.0-astatine
Dockerfile
FROM java:8-jre
## Node.js setup
RUN curl -sL https://deb.nodesource.com/setup_6.x | bash -
RUN apt-get install -y nodejs
RUN npm config set spin false
WORKDIR /app
COPY . ./
RUN npm install

The docker node images are configured to run only one browser instance. You can change this by overriding environment variables, like so:
chrome:
environment:
VIRTUAL_HOST: node.chrome.docker
HUB_PORT_4444_TCP_ADDR: hub
HUB_PORT_4444_TCP_PORT: 4444
NODE_MAX_INSTANCES: 5
NODE_MAX_SESSION: 5
image: selenium/node-chrome:3.1.0-astatine
links:
- hub
In case you're interested, I discovered this by looking at the Dockerfile source.