I'm developing a file importer, I decide to move with the stream approach, I have a series of step that I must follow to reach my goal, that can see below:
Download the file.
Parse to CSV
Validate the entire file.
"Translate" and save on db
Here is the snippet that pipe then all:
protected async pipe(readable: Readable, transform: Transform, validator: Transform, importer: T) {
const asyncPipeline = promisify(pipeline);
try {
await asyncPipeline(readable, transform, validator, importer)
logger.info("Import Finished")
} catch (error) {
const { message, stack } = error
logger.error(message, { stack })
}
}
Here is the snippet that calls the pipe method above
super.pipe(
response // response from http.get,
csv() // csv-parser library,
new Validator(),
new Importer()
)
The Validator class:
export class Validator extends Transform {
constructor() {
super({ objectMode: true })
}
_transform(chunk: any, encoding: string, done: any) {
this.push(chunk)
logger.info("Validating", { chunk })
done()
}
_flush(done: any) {
done()
}
}
and finally the importer class:
export class Importer extends Writable {
private buffer: Car[]
constructor() {
super({ objectMode: true })
this.buffer = new Array()
}
_write(row: object, enc: string, next: any) {
this.import(row)
.then(() => next())
.catch((error: Error) => next(error))
}
private async import(data: any): Promise<Car[]> {
this.buffer.push(data)
logger.info(this.buffer.length.toString());
return await db.save(data) // fake method;
}
}
When super.pipe is called the output is alternating from "Validation" and the total size of the buffer array.
{"level":30,"time":1588773096585,"pid":40537,"msg":"1"}
{"level":30,"time":1588773096586,"pid":40537,"msg":"Validating"}
{"level":30,"time":1588783063275,"pid":61571,"msg":"Importation finished"}
{"level":30,"time":1588773096633,"pid":40537,"msg":"2"}
There's a way, that first? I execute the validator stream, and after go to the importer stream?
Related
I am really having big trouble handling this issue. I have read Jest docs a lot, also other articles, but nothing has helped me yet, even chatGPT.
I am trying to test this class.
export class CookiesManager extends Auth {
static get(key: string) {
return this.cookies.get(key);
}
static set(key: string, value: any, config?: CookieSetOptions) {
this.cookies.set(key, value, config || this.config);
}
static remove(key: string) {
const { hostname } = new URL(`${process.env.WEB_HOST}`);
this.cookies.remove(key, { ...this.config, domain: hostname });
}
}
Here is the Auth class, but in my case, I haven't even reached here or I guess I will not need to handle that part in the scope of this one. Anyway, just for a reference.
import moment from 'moment';
import Cookies, { CookieSetOptions } from 'universal-cookie';
export class Auth {
static cookies = new Cookies();
static config: CookieSetOptions = {
path: '/',
maxAge: 1800,
expires: moment().add(30, 'm').toDate(),
secure: process.env.NODE_ENV !== 'development',
};
static setToken(token: string) {
token && this.cookies.set('auth_token', token, this.config);
}
static getToken() {
return this.cookies.get('auth_token');
}
static clear() {
this.cookies.remove('auth_token', this.config);
}
}
I have written the mock for the CookiesManager class.The module has also other named exports.
jest.mock('core/utils/storage-manager.ts', () => {
const originalClasses = jest.requireActual('core/utils/storage-manager.ts');
class CookiesManagerMock {
static config = {
path: '/',
maxAge: 1800,
}
static set = jest.fn().mockImplementation((key, value, config) => {
console.log('Mock set called');
mockCookies[key] = value;
})
static get = jest.fn().mockImplementation((key) => {
console.log('Mock get called');
return mockCookies[key];
})
static remove = jest.fn().mockImplementation((key, config) => {
console.log('Mock remove called');
delete mockCookies[key];
})
}
return {
...originalClasses,
CookiesManager: CookiesManagerMock,
}
})
This is the test block.
describe('CookiesManager', () => {
afterEach(() => {
jest.restoreAllMocks();
});
test('It should set a cookie', () => {
CookiesManager.set('test_key', 'test_value');
console.log(mockCookies.test_key, 'mockCookies')
expect(CookiesManager.set).toHaveBeenCalledWith('test_key', 'test_value');
});
test('It should get a cookie', () => {
CookiesManager.get.mockReturnValueOnce('test_value');
expect(CookiesManager.get('test_key')).toEqual('test_value');
expect(CookiesManager.get).toHaveBeenCalledWith('test_key');
});
test('It should remove a cookie', () => {
CookiesManager.remove('test_key');
expect(CookiesManager.remove).toHaveBeenCalledWith('test_key');
});
})
UPDATED
Even though the tests are passing, non of the console.log statements are being called in mocked class. And also mockCookies is always empty. I have tried the same with CommonJS modules and the strange thing is that console.log statements are being called.
In Jest docs, it's clearly stated.
mockFn.mockImplementation(fn)
Accepts a function that should be used as the implementation of the
mock. The mock itself will still record all calls that go into and
instances that come from itself – the only difference is that the
implementation will also be executed when the mock is called.
Maybe I am getting something wrong and don't understand the nature of the mocks.
You didn't mock the static methods correctly. The way you are using is trying to mock the instance methods of a class. Here is a solution, create a mock CookiesManager class with mock static methods.
storage-manager.ts:
type CookieSetOptions = any;
export class CookiesManager {
static get(key: string) {}
static set(key: string, value: any, config?: CookieSetOptions) {}
static remove(key: string) {}
}
export const a = () => 'a'
export const b = () => 'b'
storage-manager.test.ts:
import { CookiesManager, a, b } from './storage-manager';
jest.mock('./storage-manager', () => {
const originalModules = jest.requireActual('./storage-manager');
const mockCookies = {};
class CookiesManagerMock {
static set = jest.fn().mockImplementation((key, value, config) => {
console.log('Mock set called');
mockCookies[key] = value;
});
static get = jest.fn().mockImplementation((key) => {
console.log('Mock get called');
return mockCookies[key];
});
static remove = jest.fn().mockImplementation((key, config) => {
console.log('Mock remove called');
delete mockCookies[key];
});
}
return {
...originalModules,
CookiesManager: CookiesManagerMock,
};
});
describe('75323871', () => {
test('It should set a cookie', () => {
const config = { path: '/', maxAge: 1800 };
expect(jest.isMockFunction(CookiesManager.set)).toBeTruthy();
CookiesManager.set('test_key', 'test_value', config);
expect(CookiesManager.set).toHaveBeenCalledWith('test_key', 'test_value', config);
expect(CookiesManager.get('test_key')).toBe('test_value');
// other named exports
expect(a()).toBe('a');
expect(b()).toBe('b');
});
});
jest.config.js:
module.exports = {
preset: 'ts-jest/presets/js-with-ts'
};
Test result:
PASS stackoverflow/75323871/storage-manager.test.ts (8.614 s)
75323871
✓ It should set a cookie (17 ms)
console.log
Mock set called
at Function.<anonymous> (stackoverflow/75323871/storage-manager.test.ts:9:15)
console.log
Mock get called
at Function.<anonymous> (stackoverflow/75323871/storage-manager.test.ts:14:15)
Test Suites: 1 passed, 1 total
Tests: 1 passed, 1 total
Snapshots: 0 total
Time: 9.358 s, estimated 10 s
Note: Maybe jest.spyOn(CookiesManager, 'set').mockImplementation() is a better solution, you only need to mock the specific static method. see https://jestjs.io/docs/es6-class-mocks#static-getter-and-setter-methods
package version:
"jest": "^26.6.3",
"ts-jest": "^26.4.4"
The bounty expires in 2 days. Answers to this question are eligible for a +100 reputation bounty.
Ayoub k wants to draw more attention to this question.
As for now, I'm using mongoose middleware to handle Mongoose specific errors (validation, cast, ....).
I'm using the following code in all of my schemas:
schema.post('save', handleValidationError);
schema.post('findOneAndUpdate', handleValidationError);
schema.post(['findOne', 'deleteOne'], handleCastError);
Is there anyway to make this global in all schemas without repeating the code?
I tried to use plugins as the following but they don't get triggered if an error happens.
const errorsPlugin = (schema: any, _options: any): void => {
schema.post('save', handleValidationError);
schema.post('findOneAndUpdate', handleValidationError);
schema.post(['findOne', 'deleteOne'], handleCastError);
};
const connection = await mongoConnect(url);
plugin(errorsPlugin);
logger.info(`MongoDB connected: ${connection.connection.name}`);
Edit 1: error handler function
const handleValidationError = (error: NodeJS.ErrnoException, _res: Response, next: (err?: Error) => void): void => {
if (error.code?.toString() === '11000') {
const { keyPattern, keyValue } = error as Error & {
keyPattern: { [key: string]: number };
keyValue: { [key: string]: string };
};
const key = Object.keys(keyPattern)[0];
const value = Object.values(keyValue)[0];
throw new DuplicatedKeyError(key as string, value as string);
} else if (error instanceof mongooseError.ValidationError) {
throw new ValidationError(error.errors);
} else if (error instanceof mongooseError.CastError) {
throw new CastError(error.kind, error.value);
}
next();
};
I'm using TypeScript to create a match/player module in mongoose.
I create this insertOnMatch function to use the mongoose findOneAndUpdate function adding the player to my match schema, but I'm recieving a UnhandledPromiseRejectionWarning while trying to update that document in mongo
Full error code:
(node:508) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'insertOnMatch' of undefined
insertOnMatch function
private async insertOnMatch (matchId: String, playerId: String): Promise<void> {
await Match.findOneAndUpdate({ matchId: matchId }, { $push: { matchPlayers: { playerId: playerId, playerRefererMatchId: matchId } } })
}
create function
public async create (req: Request, res: Response): Promise<Response> {
const newPlayerConfig = {
playerId: generatedPlayerId,
playerMatchRefererId: req.params.matchId
}
const newPlayer = await Player.create(newPlayerConfig)
this.insertOnMatch(newPlayer.playerMatchRefererId, newPlayer.playerId).catch(err => { console.error(err) })
return res.json(newPlayer)
}
As I understand you try to call insertOnMatch on a functional component using this. And it is undefined as there's no class instance. accessing the 'this' keyword in a react functional component
I don't know how to reponse correctly my component.
I'm getting this error:
Type 'Promise<AnyTransaction>[]' is not assignable to type 'AnyTransaction[]'.
Type 'Promise<AnyTransaction>' is missing the following properties from type 'Transaction<any, any, any, any>': id, feeLookup, sender, receiver, and 10 more.
The component looks like this:
import { AnyTransaction, TransactionStatus } from '../types';
// more code...
export default async function getMyTransactions (
_parent: unknown,
_args: unknown,
// context: Context,
): Promise<Array<AnyTransaction>> {
// ): Promise<String> {
// some code
const { Items } = await DocumentClient.getInstance()
.query(some query
.promise();
const transactions = (Items || []) as Array<AnyTransaction>;
// transactions is an array of objects.
return transactions.map(parseDeprecatedStatuses).map(processTransactionStatus);
// parseDeprecatedStatuses: just parse some data
// processTransactionStatus code is below
}
// processTransactionStatus.ts:
import Factory from '../factory';
import { AnyTransaction } from '../types';
export default async function processTransactionStatus (
transaction: AnyTransaction
): Promise<AnyTransaction>{
const agent = Factory.buildAgentFromCode(transaction.destination.agent);
transaction.status = await agent.fetchTransactionStatus(transaction)
return transaction;
}
I'm really confused about how I'm returning from the component and what I got.
You are mapping to a list of promises, so you have to await all of those:
return await Promise.all(transactions
.map(parseDeprecatedStatuses)
.map(processTransactionStatus));
I'm having some trouble hitting a POST endpoint that triggers a typeorm repository.save() method to my postgres DB.
Here's my DTO object:
import { ApiProperty } from '#nestjs/swagger/';
import { IsString, IsUUID} from 'class-validator';
import { Client } from '../../../models';
import { User } from '../../../user.decorator';
export class ClientDTO implements Readonly<ClientDTO> {
#ApiProperty({ required: true })
#IsUUID()
id: string;
#ApiProperty({ required: true })
#IsString()
name: string;
public static from(dto: Partial<ClientDTO>) {
const cl = new ClientDTO();
cl.id = dto.id;
cl.name = dto.name;
return cl;
}
public static fromEntity(entity: Client) {
return this.from({
id: entity.id,
name: entity.name,
});
}
public toEntity = (user: User | null) => {
const cl = new Client();
cl.id = this.id;
cl.name = this.name;
cl.createDateTime = new Date();
cl.createdBy = user ? user.id : null;
cl.lastChangedBy = user ? user.id : null;
return cl;
}
}
My controller at POST - /client:
import {
Body,
Controller,
Get, Post
} from '#nestjs/common';
import { ClientDTO } from './dto/client.dto';
import { ClientService } from './client.service';
import { User } from 'src/user.decorator';
#Controller('client')
export class ClientController {
constructor(
private clientService: ClientService
) { }
#Get()
public async getAllClients(): Promise<ClientDTO[]> {
return this.clientService.getAllClients();
}
#Post()
public async createClient(#User() user: User, #Body() dto: ClientDTO): Promise<ClientDTO> {
return this.clientService.createClient(dto, user);
}
}
And my service:
import { Injectable } from '#nestjs/common';
import { InjectRepository } from '#nestjs/typeorm';
import { Repository } from 'typeorm';
import { Client } from '../../models';
import { ClientDTO } from './dto/client.dto';
import { User } from '../../user.decorator';
#Injectable()
export class ClientService {
constructor(
#InjectRepository(Client) private readonly clientRepository: Repository<Client>
) {}
public async getAllClients(): Promise<ClientDTO[]> {
return await this.clientRepository.find()
.then(clients => clients.map(e => ClientDTO.fromEntity(e)));
}
public async createClient(dto: ClientDTO, user: User): Promise<ClientDTO> {
return this.clientRepository.save(dto.toEntity(user))
.then(e => ClientDTO.fromEntity(e));
}
}
I get a 500 internal server error with log message stating that my ClientDTO.toEntity is not a function.
TypeError: dto.toEntity is not a function
at ClientService.createClient (C:\...\nest-backend\dist\features\client\client.service.js:29:47)
at ClientController.createClient (C:\...\nest-backend\dist\features\client\client.controller.js:27:35)
at C:\...\nest-backend\node_modules\#nestjs\core\router\router-execution-context.js:37:29
at process._tickCallback (internal/process/next_tick.js:68:7)
I'm confused because this only happens via http request. I have a script that seed my dev database after I launch it fresh in a docker container called seed.ts:
import * as _ from 'lodash';
import { Client } from '../models';
import { ClientDTO } from '../features/client/dto/client.dto';
import { ClientService } from '../features/client/client.service';
import { configService } from '../config/config.service';
import { createConnection, ConnectionOptions } from 'typeorm';
import { User } from '../user.decorator';
async function run() {
const seedUser: User = { id: 'seed-user' };
const seedId = Date.now()
.toString()
.split('')
.reverse()
.reduce((s, it, x) => (x > 3 ? s : (s += it)), '');
const opt = {
...configService.getTypeOrmConfig(),
debug: true
};
const connection = await createConnection(opt as ConnectionOptions);
const clientService = new ClientService(connection.getRepository(Client));
const work = _.range(1, 10).map(n => ClientDTO.from({
name: `seed${seedId}-${n}`,
}))
######################## my service calls ClientDTO.toEntity() without issue ###########################
.map(dto => clientService.createClient(dto, seedUser)
.then(r => (console.log('done ->', r.name), r)))
return await Promise.all(work);
}
run()
.then(_ => console.log('...wait for script to exit'))
.catch(error => console.error('seed error', error));
It makes me think I am missing something simple/obvious.
Thanks!
Looks like you are using ValidationPipe. The solution is mentioned here
https://github.com/nestjs/nest/issues/552
when setting your validation pipe you need to tell it to transform for example
app.useGlobalPipes(new ValidationPipe({
transform: true
}));
The fact that the dto is declared like this dto: ClientDTO in the controller is not enough to create instances of the class. This is just an indication for you and the other developers on the project, to prevent misuses of the dto object in the rest of the application.
In order to have instances of classes, and use the methods from the class, you have to explicitly set a mapping like this:
#Post()
public async createClient(#User() user: User, #Body() dto: ClientDTO): Promise<ClientDTO> {
const client = ClientDTO.from(dto);
return this.clientService.createClient(client, user);
}
Assuming ClientDTO.from is the right function to use for the data contained in dto. If not, adapt it, create a new one, or add a constructor.
Your dto was not a class-based object when coming in through your api call-- it's just a generic object. Therefore it can't have methods and so your toEntity method won't work. The error message you get is a red herring that doesn't tell you the true cause of the failure.
You can fix this by creating a new object based on your class and then calling a method on the new object to copy the properties in from your plain object dto, or by using the class-transformer library, or by whatever you want that achieves the same result.