Develop Reference

Github actions Connection error w/ Sequelize

I have a node js api that connects with sequelize to a mySQL database. I wrote a script to reset the database every time tests are ran. I am building a CI/CD pipeline and whenever the script is ran i get the following message:
throw new SequelizeErrors.ConnectionError(err);
^
ConnectionError [SequelizeConnectionError]: Connection lost: The server closed the connection.
...
parent: Error: Connection lost: The server closed the connection.
...
at TCP.<anonymous> (node:net:709:12) {
fatal: true,
code: 'PROTOCOL_CONNECTION_LOST'
},
original: Error: Connection lost: The server closed the connection.
This is the part of the pipeline that fails:
tests:
runs-on: ubuntu-20.04
env:
MYSQL_DEVELOPMENT_USER: root
MYSQL_DEVELOPMENT_PASSWORD: ${{ secrets.RootPassword }}
MYSQL_DEVELOPMENT_HOST: localhost
MYSQL_DEVELOPMENT_DATABASE: test
...
- uses: mirromutth/mysql-action#v1.1
with:
mysql root password: ${{ secrets.RootPassword }}
mysql database: test
- name: api testing
run: cd api && npm run test
...
This is the npm run test
"test": "npm run resetDevDb && cross-env NODE_ENV=test jest --verbose"
"resetDevDb" : "cross-env NODE_ENV=development node db/scripts/reset-development-db.js"
The reset database Script:
const resetDb = async () => {
await sequelize.authenticate();
await Book.sync({ force: true });
await Author.sync({ force: true });
await User.sync({ force: true });
await Promise.all(
users.map(async (user) => {
await User.create(user);
})
);
await Promise.all(
books.map(async (book) => {
await Book.create(book);
})
);
await Promise.all(
authors.map(async (author) => {
await Author.create(author);
})
);
console.log("Database reseted");
};
The sequelize config:
const developmentConfig = {
dialect: "mysql",
database: process.env.MYSQL_DEVELOPMENT_DATABASE,
username: process.env.MYSQL_DEVELOPMENT_USER,
password: process.env.MYSQL_DEVELOPMENT_PASSWORD,
host: process.env.MYSQL_DEVELOPMENT_HOST,
logging: console.log,
};
And initialization
const { database, username, password, ...options } = require("./config.js");
const { Sequelize } = require("sequelize");
const sequelize = new Sequelize(database, username, password, options);
module.exports = sequelize;
I have ran this exact pipeline with (I believe) the exact same config and was working fine. After changing the remote repository a few times (And adding the secret.RootPassword) to the new repository I started getting this connection error. From some by hand debugging I figure the error ocurrs during the Book.sync({ force: true }) but I'm not sure if it has something to do with the miromutth/mysql action. Thanks!
Note: When I run npm run test locally everything works great, all the tables drop and are created again and filled, and the tests run and pass normally.

Can't find module (Error: Cannot find module 'events/chat')

I am getting the following error:
internal/modules/cjs/loader.js:797
throw err;
^
Error: Cannot find module 'events/chat'
I have the following folder/file structure:
└─┬ chat
├─+─ events
| +── chat.js
| +── moderation.js
├─ index.js
But if I only call the chat.js or moderation.js file in my server.js file it works just fine.
index.js:
const chatEvent = require('events/chat');
const moderation = require('events/moderation');
module.exports = {
chatEvent,
moderation,
};

Can't connect Redis server to nodejs, Docker compose

I'm struggling to connect a redis deployment to my nodejs app. Of course locally without the use of docker, it works well, so I'm at odds as to whether this is an issue to do with my code, or the way I've set up my docker compose file
Dockerfile:
FROM node:8
WORKDIR /app
COPY package.json /app
COPY . /app
RUN npm install
CMD ["npm", "start"]
EXPOSE 3000
docker-compose.yml
version: "3"
services:
web:
container_name: web-container
restart: always
depends_on:
- redis
build: .
ports:
- "3000:3000"
links:
- redis
redis:
container_name: redis-container
image: "redis:latest"
ports:
- "6379:6379"
volumes:
- ./data:/data
Redis Connection File (RedisService.js)
const redis = require("redis");
const client = redis.createClient();
const DbUtils = require("../../db_utils");
const {promisify} = require("util");
const getAsync = promisify(client.get).bind(client);
const existsAsync = promisify(client.exists).bind(client);
class RedisCache {
constructor () {
var connected;
// * Initiliase the connection to redis server
client.on("connect", () => {console.log("📒 Redis cache is ready"); connected = true;})
client.on("error", (e) => {console.log("Redis cache error:\n" + e); connected = false;});
}
async setData (id, data) {
// * Stringify data if it's an object
data = data instanceof Object ? JSON.stringify(data) : data;
client.set(id, data);
return true;
}
async getData (key) {
return getAsync(key).then(data => {
data = JSON.parse(data) instanceof Object ? JSON.parse(data) : data;
return data;
})
}
async exists (key) {
return existsAsync(key).then(bool => {
return bool;
})
}
// Returns status of redis cache
async getStatus () {
return this.connected;
}
}
module.exports = new RedisCache();
ERROR
Error: Redis connection to 127.0.0.11:6379 failed - connect ECONNREFUSED 127.0.0.11:6379

When you run your containers via docker-compose they are all connected to a common network. Service name is a DNS name of given container so to access redis container from web you should create the client like :
const client = redis.createClient({
port : 6379,
host : 'redis'
});
You have not configured the host so it uses the default one - 127.0.0.1. But from the point of view of your web container the redis is not running on the localhost. Instead it runs in it's own container which DNS name is redis.

The beginning (docker part) of this tutorial worked for me :
https://medium.com/geekculture/using-redis-with-docker-and-nodejs-express-71dccd495fd3
docker run -d --name <CONTAINER_NAME> -p 127.0.0.1:6379:6379 redis
then in the node server (like in official redis website example) :
const redis = require('redis');
async function start() {
const client = redis.createClient(6379,'127.0.0.1');
await client.connect();
await client.set('mykey', 'Hello from node redis');
const myKeyValue = await client.get('mykey');
console.log(myKeyValue);
}
start();

What is Node.js cluster best practice?

Is it better to write our server logic before forking workers or after?
I'll give two examples below to make it clear.
example #1:
const express = require("express");
const cluster = require('cluster');
const app = express();
app.get("/path", somehandler);
if (cluster.Master)
// forking workers..
else
app.listen(8000);
or example #2:
const cluster = require('cluster');
if (cluster.Master)
// forking workers..
else {
const express = require("express");
const app = express();
app.get("/path", somehandler);
app.listen(8000);
}
What is the difference?

There is no difference. Since when You call cluster.fork() it calls child_process.fork on same entry file and keeps child process handler for interprocess communication.
Read following methods defined at lines following of cluster's master module: 167, 102, 51, 52
Let's get back to Your code:
In example #1 it assigns variables, creates app instance both for master and child processes, then checks for process master or not.
In example #2 it checks for process master or not and if not it assigns vars, creates app instance and binds listener on port for child workers.
In fact it will do the same operations in clild processes:
assigning vars
creating app instance
starting listener
My own best practices using cluster is has 2 steps:
Step 1 - having custom cluster wrapper in separate module and wrapping in application call:
Have cluster.js file:
'use strict';
module.exports = (callable) => {
const
cluster = require('cluster'),
numCpu = require('os').cpus().length;
const handleDeath = (deadWorker) {
console.log('worker ' + deadWorker.process.pid + ' dead');
const worker = cluster.fork();
console.log('re-spawning worker ' + worker.process.pid);
}
process.on('uncaughtException',
(err) => {
console.error('uncaughtException:', err.message);
console.error(err.stack);
});
cluster.on('exit', handleDeath);
// no need for clustering if there is just 1 cpu
if (numCpu === 1 || !cluster.isMaster) {
return callable();
}
// saving 1 cpu for master process (1 M + N instances)
// or create 2 instances since 1 M + 1 Instance
// is ineffective when respawning instance
// better to have 1 M + 2 instances if cpu count 2
const instances = numCpu > 2 ? numCpu - 1 : numCpu;
console.log('Starting', instances, 'instances');
for (let i = 0; i < instances; i++, cluster.fork());
};
Keep app.js simple like this for modularity and testability (read about supertest):
'use strict';
const express = require("express");
const app = express();
app.get("/path", somehandler);
module.exports = app;
Serving the app at some port must be handled by different module, so have server.js look like this:
'use strict';
const start = require('./cluster');
start(() => {
const http = require('http');
const app = require('./app');
const listenHost = process.env.HOST || '127.0.0.1';
const listenPort = process.env.PORT || 8080;
const httpServer = http.createServer(app);
httpServer.listen(listenPort, listenHost,
() => console.log('App listening at http://'+listenHost+':'+listenPort));
});
You may add in package.json such line in scripts section:
"scripts": {
"start": "node server.js",
"watch": "nodemon server.js",
...
}
Run the app using:
node server.js, nodemon server.js
or
npm start, npm run watch
Step 2 - when needed containerization:
Keep code structure like in Step 1 and use docker
Cluster module will get cpu resources which provided by container orkestrator
and as an extra You'll have ability to scale docker instances on demand using docker swarm, kubernetes, dc/os and etc.
Dockerfile :
FROM node:alpine
ENV PORT=8080
EXPOSE $PORT
ADD ./ /app
WORKDIR /app
RUN apk update && apk upgrade && \
apk add --no-cache bash git openssh
RUN npm i
CMD ["npm", "start"]

Mocha API Testing: getting 'TypeError: app.address is not a function'

My Issue
I've coded a very simple CRUD API and I've started recently coding also some tests using chai and chai-http but I'm having an issue when running my tests with $ mocha.
When I run the tests I get the following error on the shell:
TypeError: app.address is not a function
My Code
Here is a sample of one of my tests (/tests/server-test.js):
var chai = require('chai');
var mongoose = require('mongoose');
var chaiHttp = require('chai-http');
var server = require('../server/app'); // my express app
var should = chai.should();
var testUtils = require('./test-utils');
chai.use(chaiHttp);
describe('API Tests', function() {
before(function() {
mongoose.createConnection('mongodb://localhost/bot-test', myOptionsObj);
});
beforeEach(function(done) {
// I do stuff like populating db
});
afterEach(function(done) {
// I do stuff like deleting populated db
});
after(function() {
mongoose.connection.close();
});
describe('Boxes', function() {
it.only('should list ALL boxes on /boxes GET', function(done) {
chai.request(server)
.get('/api/boxes')
.end(function(err, res){
res.should.have.status(200);
done();
});
});
// the rest of the tests would continue here...
});
});
And my express app files (/server/app.js):
var mongoose = require('mongoose');
var express = require('express');
var api = require('./routes/api.js');
var app = express();
mongoose.connect('mongodb://localhost/db-dev', myOptionsObj);
// application configuration
require('./config/express')(app);
// routing set up
app.use('/api', api);
var server = app.listen(3000, function () {
var host = server.address().address;
var port = server.address().port;
console.log('App listening at http://%s:%s', host, port);
});
and (/server/routes/api.js):
var express = require('express');
var boxController = require('../modules/box/controller');
var thingController = require('../modules/thing/controller');
var router = express.Router();
// API routing
router.get('/boxes', boxController.getAll);
// etc.
module.exports = router;
Extra notes
I've tried logging out the server variable in the /tests/server-test.js file before running the tests:
...
var server = require('../server/app'); // my express app
...
console.log('server: ', server);
...
and I the result of that is an empty object: server: {}.

You don't export anything in your app module. Try adding this to your app.js file:
module.exports = server

It's important to export the http.Server object returned by app.listen(3000) instead of just the function app, otherwise you will get TypeError: app.address is not a function.
Example:
index.js
const koa = require('koa');
const app = new koa();
module.exports = app.listen(3000);
index.spec.js
const request = require('supertest');
const app = require('./index.js');
describe('User Registration', () => {
const agent = request.agent(app);
it('should ...', () => {

This may also help, and satisfies #dman point of changing application code to fit a test.
make your request to the localhost and port as needed
chai.request('http://localhost:5000')
instead of
chai.request(server)
this fixed the same error message I had using Koa JS (v2) and ava js.

The answers above correctly address the issue: supertest wants an http.Server to work on. However, calling app.listen() to get a server will also start a listening server, this is bad practice and unnecessary.
You can get around by this by using http.createServer():
import * as http from 'http';
import * as supertest from 'supertest';
import * as test from 'tape';
import * as Koa from 'koa';
const app = new Koa();
# add some routes here
const apptest = supertest(http.createServer(app.callback()));
test('GET /healthcheck', (t) => {
apptest.get('/healthcheck')
.expect(200)
.expect(res => {
t.equal(res.text, 'Ok');
})
.end(t.end.bind(t));
});

Just in case, if someone uses Hapijs the issue still occurs, because it does not use Express.js, thus address() function does not exist.
TypeError: app.address is not a function
at serverAddress (node_modules/chai-http/lib/request.js:282:18)
The workaround to make it work
// this makes the server to start up
let server = require('../../server')
// pass this instead of server to avoid error
const API = 'http://localhost:3000'
describe('/GET token ', () => {
it('JWT token', (done) => {
chai.request(API)
.get('/api/token?....')
.end((err, res) => {
res.should.have.status(200)
res.body.should.be.a('object')
res.body.should.have.property('token')
done()
})
})
})

Export app at the end of the main API file like index.js.
module.exports = app;

We had the same issue when we run mocha using ts-node in our node + typescript serverless project.
Our tsconfig.json had "sourceMap": true . So generated, .js and .js.map files cause some funny transpiling issues (similar to this). When we run mocha runner using ts-node. So, I will set to sourceMap flag to false and deleted all .js and .js.map file in our src directory. Then the issue is gone.
If you have already generated files in your src folder, commands below would be really helpful.
find src -name ".js.map" -exec rm {} \;
find src -name ".js" -exec rm {} \;

I am using Jest and Supertest, but was receiving the same error. It was because my server takes time to setup (it is async to setup db, read config, etc). I needed to use Jest's beforeAll helper to allow the async setup to run. I also needed to refactor my server to separate listening, and instead use #Whyhankee's suggestion to create the test's server.
index.js
export async function createServer() {
//setup db, server,config, middleware
return express();
}
async function startServer(){
let app = await createServer();
await app.listen({ port: 4000 });
console.log("Server has started!");
}
if(process.env.NODE_ENV ==="dev") startServer();
test.ts
import {createServer as createMyAppServer} from '#index';
import { test, expect, beforeAll } from '#jest/globals'
const supertest = require("supertest");
import * as http from 'http';
let request :any;
beforeAll(async ()=>{
request = supertest(http.createServer(await createMyAppServer()));
})
test("fetch users", async (done: any) => {
request
.post("/graphql")
.send({
query: "{ getQueryFromGqlServer (id:1) { id} }",
})
.set("Accept", "application/json")
.expect("Content-Type", /json/)
.expect(200)
.end(function (err: any, res: any) {
if (err) return done(err);
expect(res.body).toBeInstanceOf(Object);
let serverErrors = JSON.parse(res.text)['errors'];
expect(serverErrors.length).toEqual(0);
expect(res.body.data.id).toEqual(1);
done();
});
});
Edit:
I also had errors when using data.foreach(async()=>..., should have use for(let x of... in my tests