For smoother CI experience, I have made a Github Action workflow to publish the monorepo packages with a prerelease version, everytime any member opens a PR against master with a particular label 'publish'. This workflow
should ideally publish all the changed packages since the last publish with a preid -pr{pr#} e.g., package-pr1049.0. Have added dist-tag and predist-tag also here.
Background:
Before publishing the packages, I also run a MAKE executable(make -j init script) to clean and bootstrap all the packages. Post this, it will fetch the repo, checkout to the required branch, and run the publish command with the PR number parameter.
There are 2 problems I am facing in this workflow:
Publishes all the packages in the first commit of the PR:
To debug the issue , Had also added a logger to see if it has the correct record for last 10 commits, which reflects the correct set of commits.
Second commit onwards, only the changed packages are published which is as expected. refer the log
lerna notice cli v3.20.2
lerna info versioning independent
lerna info ci enabled
lerna info Assuming all packages changed
lerna info getChangelogConfig Successfully resolved preset "conventional-changelog-angular"
Changes:
- #swiggy-private/package-1: 1.0.4 => 1.1.0-pr19000.0
- #swiggy-private/package-2: 1.1.4 => 1.2.0-pr19000.0
- #swiggy-private/package-3: 2.41.2 => 2.42.0-pr19000.0 (private)
Always updates the patch version as 0, and increments the preid (PR number with -pr prefix) PACKAGEv0-pr{##}.0, PACKAGEv0-pr{##}.1, ....
For speeding up debugging process, have limited monorepo packages in lerna.json to only 3 of the packages.
My GH workflow
name: Branch Publish
on:
pull_request:
types: [opened, synchronize, reopened, labeled]
branches:
- master
jobs:
check:
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
author: ${{ steps.step1.outputs.author }}
steps:
- uses: actions/checkout#v2
with:
ref: ${{ github.event.pull_request.head.sha }}
- id: "step1"
run: |
AUTHOR_NAME=$(git show ${{ github.event.pull_request.head.sha }} | grep Author)
echo "::set-output name=author::$AUTHOR_NAME"
init:
if: "!contains(needs.check.outputs.author, 'GitHub Action Branch') && !contains(github.event.head_commit.message, '[skip ci]')"
runs-on: ubuntu-latest
timeout-minutes: 15
needs: [check]
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v1
with:
node-version: "12.x"
- run: git fetch --prune --unshallow
- run: |
make -j init
env:
NPM_TOKEN: ${{ secrets.GH_TOKEN }}
- uses: actions/cache#v1
id: cache-build
with:
path: "."
key: ${{ github.sha }}
release:
if: "contains(github.event.pull_request.labels.*.name, 'publish')"
runs-on: ubuntu-latest
timeout-minutes: 15
needs: [init]
steps:
- uses: actions/checkout#v2
with:
fetch-depth: "0"
- uses: actions/setup-node#v1
with:
node-version: "12.x"
- uses: actions/cache#v1
id: restore-build
with:
path: "."
key: ${{ github.sha }}
- name: Setup Git
uses: webfactory/ssh-agent#v0.4.1
with:
ssh-private-key: ${{ secrets.GHA_DEPLOY_KEY }}
- name: Lerna Publish
if: success()
env:
GH_TOKEN: ${{ secrets.GH_TOKEN }}
NODE_ENV: production
run: |
git config user.email "action#github.com"
git config user.name "GitHub Action Branch"
git remote set-url origin "git#github.com:${{ github.repository }}"
git fetch --depth=1 origin +refs/tags/*:refs/tags/*
git checkout -- .
git log --pretty=oneline -n 10
git checkout --track origin/$(echo $GITHUB_HEAD_REF | cut -d'/' -f 3)
NUMBER=${{ github.event.number }} npm run publish-branch
- name: Possible Package lock update
if: success()
run: |
git config user.email "action#github.com"
git config user.name "GitHub Action Branch"
git remote set-url origin "git#github.com:${{ github.repository }}"
npx lerna clean -y
npx lerna exec -- npm i --package-lock-only --ignore-scripts --no-audit
echo `git add . && git commit -m "chore: package lock update" --no-verify && git push`
Publish command
"publish-branch": "lerna publish --conventional-prerelease --exact --no-changelog --preid pr$NUMBER --dist-tag beta --pre-dist-tag beta --no-verify-access --yes"
Lerna.json
{
"packages": ["*"],
"version": "independent",
"command": {
"publish": {
"npmClient": "npm",
"graphType": "all",
"allowBranch": ["master", "integration", "*"],
"conventionalCommits": true,
"message": "chore(release): publish",
"includeMergedTags": true,
"ignoreChanges": ["**/__tests__/**", "**/*.md"]
}
}
}
Make Script to bootstrap packages
init: clean-all
$(MAKE) create-npmrc-all
npm ci
npm run bootstrap:ci
NODE_ENV=production npm run prepare:all
create-npmrc-all:
echo $(GITHUB_SCOPE_REGISTRY) >> .npmrc
echo $(GITHUB_REGISTRY_TOKEN) >> .npmrc
$(foreach source, $(DIRECTORY), $(call pass-to-npmrc, $(source), $(GITHUB_SCOPE_REGISTRY)))
$(foreach source, $(DIRECTORY), $(call pass-to-npmrc, $(source), $(GITHUB_REGISTRY_TOKEN)))
clean-all:
rm -rf node_modules
$(foreach source, $(SOURCES), \
$(call clean-source-all, $(source)))
rm -rf .npmrc
rm -rf packages/*/.npmrc
rm -rf coverage
rm -rf packages/*/npm-debug*
Instead of creating script from .npmrc file you can just provide those data directly in the pipeline, those 2 line are perfectly working even with the github predefined token. Not really sure what environment you are using, but I think you'll get the point.
name: Setting Up NPM
run: |
npm set#organization:registry=https://npm.pkg.github.com/organization
npm set "//npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}"
Related
Current Behavior
When I run lerna publish, it gets stuck after packaging the first package. This is where it gets stuck:
lerna WARN ENOLICENSE One way to fix this is to add a LICENSE.md file to the root of this repository.
lerna WARN ENOLICENSE See https://choosealicense.com for additional guidance.
(#########⠂⠂⠂⠂⠂⠂⠂⠂⠂) ⠏ publish: verb packed packages/react
This line particularly:
(#########⠂⠂⠂⠂⠂⠂⠂⠂⠂) ⠏ publish: verb packed packages/react
Expected Behavior
It should publish to NPM successfully
Failure Logs / Configuration
lerna.json
{
"packages": [
"packages/*",
"docs"
],
"version": "independent",
"stream": true,
"hoist": true,
"command": {
"bootstrap": {
"npmClientArgs": ["--no-package-lock"]
},
"publish": {
"ignoreChanges": ["**/stories/**", "**/tests/**"]
}
}
}
Environment
System:
OS: macOS 12.3
CPU: (8) arm64 Apple M1 Pro
Binaries:
Node: 14.21.2 - ~/.nvm/versions/node/v14.21.2/bin/node
npm: 6.14.17 - ~/.nvm/versions/node/v14.21.2/bin/npm
Utilities:
Git: 2.32.1 - /usr/bin/git
npmPackages:
lerna: ^5.6.2 => 5.6.2
I am trying to setup circeci in NX workspace for react app.
On the step where is executed yarn install
I get next error:
error /home/circleci/project/node_modules/#nrwl/js/node_modules/nx,
/home/circleci/project/node_modules/#nrwl/remix/node_modules/nx:
Command failed. Exit code: 1 Command: node ./bin/init Arguments:
Directory:
/home/circleci/project/node_modules/#nrwl/js/node_modules/nx Output:
NX Cannot read properties of undefined (reading 'endsWith') info Visit https://yarnpkg.com/en/docs/cli/install for documentation about
this command.
Exited with code exit status 1
This is my circeci config
version: 2.1
orbs:
nx: nrwl/nx#1.4.0
jobs:
agent:
resource_class: xlarge
docker:
- image: cimg/node:lts-browsers
parameters:
ordinal:
type: integer
steps:
- checkout
- run:
name: Install dependencies
command: |
yarn install
- run:
name: Start the agent << parameters.ordinal >>
command: yarn nx-cloud start-agent
no_output_timeout: 60m
main:
resource_class: xlarge
docker:
- image: cimg/node:lts-browsers
environment:
NX_CLOUD_DISTRIBUTED_EXECUTION: 'true'
steps:
- checkout
- run:
name: Install dependencies
command: |
yarn install
- nx/set-shas:
main-branch-name: 'main'
- run:
name: Initialize the Nx Cloud distributed CI run
command: yarn nx-cloud start-ci-run
- run:
name: Run workspace lint
command: yarn nx-cloud record -- yarn nx workspace-lint
- run:
name: Check format
command: yarn nx-cloud record -- yarn nx format:check --base=$NX_BASE --head=$NX_HEAD
- run:
name: Run lint
command: yarn nx affected --base=$NX_BASE --head=$NX_HEAD --target=lint --parallel=3
- run:
name: Run test
command: yarn nx affected --base=$NX_BASE --head=$NX_HEAD --target=test --parallel=3 --ci --code-coverage
- run:
name: Run build
command: yarn nx affected --base=$NX_BASE --head=$NX_HEAD --target=build --parallel=3
- run:
name: Stop all agents
command: yarn nx-cloud stop-all-agents
when: always
workflows:
version: 2
ci:
jobs:
- agent:
name: Nx Cloud Agent << matrix.ordinal >>
matrix:
parameters:
ordinal: [1, 2, 3]
- main:
name: Nx Cloud Main
Does anyone had similar problem?
I encount this issue a lot of times.
Just remove the ".cache/nx" folder in your node_modules and re-run your command.
We use jenkins CI tool for automated accessibility testing provided by pa11y. As such i use the below Jenkinsfile to run the tests.
node('mypod') {
container('centos') {
def NODEJS_HOME
env.NODEJS_HOME = "${tool 'Node-12.0.0'}"
env.PATH="${env.NODEJS_HOME}/bin:${env.PATH}"
sh "'${env.NODEJS_HOME}'/bin/node --version"
sh "npm install -g pa11y --unsafe-perm"
sh "pa11y -V"
sh '''curl -O https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
yum -y install ./google-chrome-stable_current_*.rpm
yum -y install libXScrnSaver
yum -y install atk java-atk-wrapper at-spi2-atk gtk3 libXt'''
withCredentials([file(credentialsId: '***', variable: 'pa11yconfig')]) {
sh "cat $pa11yconfig > config.json"
sh "pa11y --config config.json --ignore WCAG2AA.Principle3.Guideline3_2.3_2_2.H32.2 --ignore WCAG2AA.Principle1.Guideline1_4.1_4_3.G145.Fail --ignore WCAG2AA.Principle1.Guideline1_4.1_4_3.G18.Fail --threshold 6 --reporter cli https://$URL > results.json"
}
}
}
It installs the necessary things to run pa11y against the specified URL on linux based node. Windows are too much of a hassle so we use linux for this implementaion.
Also to make this work for the browser to launch we use the below config.json file for pa11y to work.
{
"chromeLaunchConfig": {
"args": [
"--no-sandbox",
"--disable-setuid-sandbox",
"--disable-dev-shm-usage"
]
}
}
All of this works like a charm for any URl we provide.
Now we would like to have some advanced configurations for lets say test if login works or filling a form on a webpage of a site so may be use Actions provided by pa11y.
How should i merge Actions code into this json configuration file to achieve that.
Actions is documented under :-
https://github.com/pa11y/pa11y#actions
Any help or suggestions here would be greatly appreciated!
Something like this:
"chromeLaunchConfig": {
"args": [
"--no-sandbox",
"--disable-setuid-sandbox",
"--disable-dev-shm-usage"
],
},
"reporter": "cli",
"threshold": 6,
"ignore:" [
'WCAG2AA.Principle3.Guideline3_2.3_2_2.H32.2',
'WCAG2AA.Principle1.Guideline1_4.1_4_3.G145.Fail',
'WCAG2AA.Principle1.Guideline1_4.1_4_3.G18.Fail'
]
"actions": [
"navigate to $URL",
"wait for $ThingToHappen"
]
}
(I also included options that you're currently passing to the CLI directly, in case that's of interest to you)
I'm using vue 3 with jest for unit tests. My component is in the .vue file, with js and css in separate files and included in the .vue via src=:
<template>
<div>my library</div>
</template>
<script src="./library.js"></script>
<style src="./library.css"></style>
This works well for the most part, e.g. npm run serve reloads and refreshes everything as I save changes to those files.
However, when I run npm run test:unit the tests are running against a stale (cached?) version of my library.js. My package.json includes in scripts: "test:unit": "vue-cli-service test:unit",. I suspect it is cached because if I add a comment to the .vue file, it runs against the correct version of the .js, but if I remove the comment (so the file matches the previous version) then it runs against the stale .js again.
Possibly interesting is that running vue-cli-service test:unit --watch does re-run the tests when I change the .js file, but it runs against the stale version and not the new version that triggers the re-run.
The only workaround I seem to have is to append to a dummy comment in the .vue file, which is annoying. Or move to SFC which I find annoying because editor support for the different sections isn't as good as it is with separate files.
How can I get npm / vue-cli-service to bypass this apparent caching? Or is there a way to clear the cache?
The script below will reproduce the issue. Note in the output at the bottom that the tests are run three times:
On the first run the output should include "hello created 1".
Then the .js file is edited to change that string so that on the second run the output should be "hello created 2". However, when I run this script it provides "hello created 1" on both test runs.
Then the .vue file is edited to change a dummy comment. On the third run, the output is "hello created 2" as expected.
#!/bin/bash
if [[ -f package.json ]]
then
echo "'cd ..' and rerun this script"
echo "you need to be in the parent directory to the project directory"
exit 1
fi
if [[ ! -d utfail ]]
then
vue create -p __default_vue_3__ utfail
cd utfail
# specific versions are based on what I was using originally
npm install vue#3.0.3
npm install -D #vue/test-utils#2.0.0-beta.11
npm install -D #vue/compiler-sfc#3.0.3
npm install -D vue-jest#5.0.0-alpha.7
npm install -D #vue/cli-plugin-unit-jest#4.5.9
npm install -D typescript#3.9.7
else
cd utfail
fi
# hack: replace the default lint script with test:unit
sed -i -e 's/^.*"lint":.*$/ "test:unit": "vue-cli-service test:unit"/' package.json
cat <<EOF > jest.config.js
module.exports = {
moduleFileExtensions: ["js", "json", "vue"],
preset: '#vue/cli-plugin-unit-jest',
transform: {
'^.+\\.js$': "babel-jest",
'^.+\\.vue$': 'vue-jest'
},
"automock": false,
"setupFiles": [
"./setupJest.js"
]
}
EOF
cat <<EOF > src/components/HelloWorld.vue
<!-- dummy comment 1 -->
<template>
<div class="hello">blah</div>
</template>
<script src="./helloworld.js"></script>
<style></style>
EOF
cat <<EOF > src/components/helloworld.js
export default {
name: 'HelloWorld',
created() {
console.log("hello created 1")
}
}
EOF
cat <<EOF > setupJest.js
// require('jest-fetch-mock').enableMocks()
EOF
mkdir -p __tests__
cat <<EOF > __tests__/app.spec.js
import { mount } from '#vue/test-utils'
import App from './../src/App.vue'
import HelloWorld from './../src/components/HelloWorld.vue'
describe('HelloWorld', () => {
beforeEach(() => {
})
it('exists', () => {
const wrapper = mount(HelloWorld)
expect(wrapper.exists()).toBe(true)
})
})
EOF
printf '\n*\n*\n*\n*** about to run tests (round 1)\n*\n*\n*\n'
grep 'hello created' src/components/helloworld.js
npm run test:unit
sed -i -e '/hello created/s/1/2/' src/components/helloworld.js
printf '\n*\n*\n*\n*** about to run tests (round 2)\n*\n*\n*\n'
grep 'hello created' src/components/helloworld.js
npm run test:unit
sed -i -e '/dummy comment/s/1/2/' src/components/HelloWorld.vue
printf '\n*\n*\n*\n*** about to run tests (round 3)\n*\n*\n*\n'
grep 'hello created' src/components/helloworld.js
npm run test:unit
Configure Jest to disable the test cache:
// jest.config.js
module.exports = {
cache: false,
}
Or add the --no-cache flag to the test:unit npm script:
// package.json
{
"scripts": {
"test:unit": "vue-cli-service test:unit --no-cache"
}
}
Or clear the cache with this command from the project's root directory:
npx jest --clearCache
I have some global packages such as serverless framework, ESLint and etc. I've implemented GitHub Actions cache for yarn. Below is my code.
- uses: actions/cache#v1
id: yarn-cache # use this to check for `cache-hit` (`steps.yarn-cache.outputs.cache-hit != 'true'`)
with:
path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-
- name: Adding serverless globally
run: yarn global add serverless
- name: Yarn Install
if: steps.yarn-cache.outputs.cache-hit != 'true'
run: |
echo "cache hit failed"
yarn install
env:
CI: false
But my global packages are not cached. Is there any way to cache Yarn globals?
I'm pasting the build file for the solution,
name: global-test
on:
push:
branches:
- dev
pull_request:
branches:
- dev
jobs:
aws-deployment:
runs-on: ubuntu-latest
steps:
- name: CHECKOUT ACTION
uses: actions/checkout#v2
- name: NODE SETUP ACTION
uses: actions/setup-node#v1
with:
node-version: '12.x'
- name: Get yarn cache directory path
id: yarn-cache-dir-path
run: |
echo "::set-output name=dir::$(yarn cache dir)"
- name: Set yarn global bin path
run: |
yarn config set prefix $(yarn cache dir)
- name: Add yarn bin path to system path
run: |
echo $(yarn global bin) >> $GITHUB_PATH
- name: Set yarn global installation path
run: |
yarn config set global-folder $(yarn cache dir)
- name: CACHE ACTION
uses: actions/cache#v2
env:
cache-version: v1
id: yarn-cache
with:
path: |
${{ steps.yarn-cache-dir-path.outputs.dir }}
**/node_modules
key: ${{ runner.os }}-yarn-${{ env.cache-version }}-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-${{ env.cache-version }}-
${{ runner.os }}-yarn-
${{ runner.os }}-
- name: Installing dependencies
if: steps.yarn-cache.outputs.cache-hit != 'true'
run: |
echo "YARN CACHE CHANGED"
yarn install
- name: Adding serverless globally
if: steps.yarn-cache.outputs.cache-hit != 'true'
run: |
echo "NO CACHE HIT"
yarn global add serverless
I named the steps, so they can be understood.
UPDATED the answer on 2020-12-06