Develop Reference

Async Function working in Express but not NestJs

I initially created a little express server to run a report and file write function.
var ssrs = require('mssql-ssrs');
var fs = require('fs');
const express = require('express')
const app = express()
const port = 3001
app.get('/', (req, res) => {
reportCreation();
res.send('File Created');
})
app.get('/api', (req, res) => {
reportCreation();
res.json({'File Created': true});
})
app.listen(port, () => {
console.log(`Report Api listening at http://localhost:${port}`)
})
The function reportCreation() is an async function which gets a report from a SSRS. This works fine
async function reportCreation() {
var serverUrl = 'http://reportServerName/ReportServer/ReportExecution2005.asmx';
ssrs.setServerUrl(serverUrl);
var reportPath = '/ApplicationPortalReports/TestReportNew';
var fileType = 'word';
var parameters = { ApplicationId: 3, TrainingCardId: 267, PortalPersonId: 52 }
var auth = {
username: 'USERNAME',
password: 'PASSWORD',
domain: 'dmz'
};
try {
var report = await ssrs.reportExecution.getReportByUrl(reportPath, fileType, parameters, auth)
} catch (error) {
console.log(error);
}
console.log(report);
try {
fs.writeFile('ReportApiTest.doc', report, (err) => {
if (!err) console.log('Data written');
});
} catch (error) {
console.log(error);
}
I have been working a lot with NestJs recently and wanted to use the same function but within a NestJs service.
#Injectable()
export class AppService {
async getReport(): Promise<string> {
const serverUrl = 'http://reportServerName/ReportServer/ReportExecution2005.asmx';
ssrs.setServerUrl(serverUrl);
const reportPath = '/ApplicationPortalReports/TestReportNew';
const fileType = 'word';
// var parameters = {appId: 3, ReportInstanceId: 1 }
const parameters = {ApplicationId: 3, TrainingCardId: 267, PortalPersonId: 52 };
const auth = {
username: 'USERNAME',
password: 'PASSWORD',
domain: 'dmz'
};
try {
var report = await ssrs.reportExecution.getReportByUrl(reportPath, fileType, parameters, auth)
} catch (error) {
console.log(error);
}
console.log(report);
// excel = xlsx
// word = doc
// pdf = pdf
try {
fs.writeFile('ReportApiTest.doc', report, (err) => {
if (!err) { console.log('Data written');
return 'File Written Succesfully'}
});
} catch (error) {
console.log(error);
return 'File Write Error'
}
}
}
As you can see the files are almost identical, but when I run it through NestJs I get an error which looks like a problem with the line
var report = await ssrs.reportExecution.getReportByUrl(reportPath, fileType, parameters, auth)
not awaiting. Why does this work with Express and not NestJS? Below is the error from NestJs
buffer.js:219
throw new ERR_INVALID_ARG_TYPE(
^
TypeError [ERR_INVALID_ARG_TYPE]: The first argument must be one of type string, Buffer, ArrayBuffer,
Array, or Array-like Object. Received type undefined
at Function.from (buffer.js:219:9)
at new Buffer (buffer.js:179:17)
at Object.createType3Message (C:\Projects\SSRS-report-api\ssrs-report-api\node_modules\httpntlm\ntlm.js:172:19)
at sendType3Message (C:\Projects\SSRS-report-api\ssrs-report-api\node_modules\httpntlm\httpntlm.js:77:23)
at Immediate._onImmediate (C:\Projects\SSRS-report-api\ssrs-report-api\node_modules\httpntlm\httpntlm.js:101:4)
within the mssql-ssrs node package the getReportByURL looks like this
async function getReportByUrl(reportPath, fileType, params, auth) {
try {
var config = {
binary: true, // very important
username: auth.userName,
password: auth.password,
workstation: auth.workstation,
domain: auth.domain,
url: soap.getServerUrl()
+ "?" + (testReportPath(reportPath).replace(/\s/g, '+'))
+ "&rs:Command=Render&rs:Format=" + reportFormat(fileType)
+ formatParamsToUrl(params)
};
} catch (err) { report.errorHandler(err) }
return new Promise((resolve, reject) => {
config.url = encodeURI(config.url);
httpntlm.post(config, function (err, res) {
if (res.statusCode === 500) { reject(res) }
if (err || res.statusCode !== 200) { reject(err) }
else { resolve(res.body) }
})
})
}
Here is the app.controller.ts
#Controller()
export class AppController {
constructor(private readonly appService: AppService) {}
#Get()
getHello(): Promise<string> {
return this.appService.getReport();
}
}

This is not an answer for the question. But after I see your code, I can see an error you will face in future if await ssrs.reportExecution.getReportByUrl(reportPath, fileType, parameters, auth) failed. Actually you see above error because of this.
The way you used the try catch is really bad.
Here's the way I code it.
#Injectable()
export class AppService {
async getReport(): Promise<string> {
const serverUrl = 'http://reportServerName/ReportServer/ReportExecution2005.asmx';
ssrs.setServerUrl(serverUrl);
const reportPath = '/ApplicationPortalReports/TestReportNew';
const fileType = 'word';
// var parameters = {appId: 3, ReportInstanceId: 1 }
const parameters = {ApplicationId: 3, TrainingCardId: 267, PortalPersonId: 52 };
const auth = {
username: 'USERNAME',
password: 'PASSWORD',
domain: 'dmz'
};
const report = await ssrs.reportExecution.getReportByUrl(reportPath, fileType, parameters, auth)
return new Promise(function(resolve, reject) {
fs.writeFile('ReportApiTest.doc', report, , function(err) {
if (err) reject(err);
resolve("File Created");
});
});
}
And in my controller
#POST
async writeFile() {
try {
const res = await this.appService.getReport();
return res;
} catch(err) {
// handle your error
}
}

I had fudged the code in the node_module changing the userName variable to username and had not done the same in the NestJS version. I forgot I had done that so now it is working.

Incomplete CSV to JSON via FTP

Got some code that connects to an FTP server, converts the data to CSV and stores it in an array that will be later be used to update a database.
I'm dealing with many large CSV files and when accessing from FTP, they return incomplete!
I output the array length, and it's different every time (but usually close to the full length)
Removing .end() does not change the result, so it's not disconnecting early...
Downloading the CSV files manually (not an option expt. for dev) does result in the full file being read.
Any ideas appreciated, cheers :)
Here is the code
var FTPClient = require('ftp');
var c = new FTPClient();
var csv = require("csvtojson");
c.connect({
host: '****',
user: '****',
password: '****',
debug: console.log,
});
dataArray = []
c.on('ready', function() {
c.get('/file.csv', function(err, stream) {
if (err) throw err;
stream.once('close', function() {
console.log(dataArray.length)
c.end();
});
stream.on('data', function(chunk) {
csv({
noheader: false,
headers: ["Header1", "Header2", ]
})
.fromString(chunk.toString())
.subscribe(function(jsonObj) {
return new Promise(function(resolve, reject) {
dataArray.push(jsonObj)
resolve()
})
})
.on("end", function() {
console.log("done");
});
});
});
});

I now use this for streaming and parsing
const PromiseFtp = require('promise-ftp'),
ftp = new PromiseFtp();
const csv = require("csvtojson");
var jsonArray = []
var credentails = {
host: '*****',
user: '*****',
password: '*****',
}
var csvFile = {
formatA: ['filesA.csv'],
formatB: ['filesB.csv'],
formatC: ['filesC.csv']
}
async function fptUpdate(file) {
await ftp.connect(credentails).then(function() {
return ftp.get(file);
}).then(function(stream) {
return new Promise(function(resolve, reject) {
stream.pipe(csv(headers))
.on('data', (data) => {
let string = data.toString('utf8')
let json = JSON.parse(string)
jsonArray.push(json);
})
.on('end', () => {
resolve()
});
});
}).then(function() {
return ftp.end();
});
}
async function start() {
for (const file of csvFiles) {
await fptUpdate(file)
console.log(jsonArray.length)
jsonArray = []
console.log('Processed file: ' + file)
}
}
start()

Here I am trying to return array of files but it returns undefined

Here in below function I am trying to return a file, but its returning undefined while searching for files, it returns undefined. Where I am performing search and getting S3 objects and preparing array of object for further sending email purpose.
const send_email = async(data) => {
try {
const data_attachment = await get_S3_files(batch_key);
console.log(data_attachment)
} catch (error) {
console.log(error)
}
}
const get_S3_files = (data) => {
return new Promise((resolve, reject) => {
var params = {
Bucket: S3_bucket,
Delimiter: '',
Prefix: `${data}/`
}
var files = []
s3.listObjects(params, function (err, data) {
if (err) throw err;
if (data.Contents.length) {
for (const row of data.Contents) {
await s3.getObject({ Bucket: 'name', Key: row.Key }, function (err, data) {
if (err) {
console.log('S3 Get Object Error', err)
reject(err)
} else {
console.log(data)
files.push({
filename: data.Body,
content: data.Metadata.file_name,
})
}
})
}
}
});
resolve(files);
})
}
EDIT-
REMOVED AWAIT
Kept return in function section
ISSUE: ITS RETURNING EMPTY ARRAY BEFORE THE FUNCTION GETS EXECUTED.
Can anyone help here? What am I doing wrong?

The issue is with a race condition. It is returning even before waiting for the array to be fill.
I think this solution might work.
const send_email = async (data) => {
try {
const data_attachment = await get_S3_files(batch_key);
console.log(data_attachment);
} catch (error) {
console.log(error);
}
};
const get_S3_files = (batch_key) => {
return new Promise((resolve, reject) => {
var params = {
Bucket: S3_bucket,
Delimiter: "",
Prefix: `${batch_key}/`,
};
s3.listObjects(params, async function (err, data) {
var files = [];
if (err) throw err;
if (data.Contents.length) {
await Promise.all(
data.Contents.map((row) => {
return new Promise((resolve, reject) => {
s3.getObject({ Bucket: S3_bucket, Key: row.Key }, (err, data) => {
if (err) return reject(err);
files.push({
filename: data.Body,
content: data.Metadata.file_name,
});
return resolve(data);
});
});
})
);
}
return resolve(files);
});
});
};

You are returning from the wrong place. Your return statement is inside s3.listObjects function. The return files; should be one line below just before closing the function. But, it is not gonna work anyway. You have to promisify the function.
const get_S3_files = (data) => {
return new Promise((resolve, reject) => {
var params = {
Bucket: 'name',
Delimiter: '',
Prefix: `${data}/`
}
var files = []
s3.listObjects(params, async function (err, data) {
if (err) throw err;
if (data.Contents.length) {
for (const row of data.Contents) {
await s3.getObject({ Bucket: 'name', Key: row.Key }, function (err, data) {
if (err) {
console.log('S3 Get Object Error', err)
reject(err)
} else {
console.log(data)
files.push({
filename: data.Body,
content: data.Metadata.file_name,
})
}
})
}
}
resolve(files);
});
})
}

The below code-snippet might not necessarily answer the original problem, but if OP is open to bring some drastic changes to his code, then I think this solution would work perfectly without affecting the original functionality.
const get_S3_files = async (data) => {
var params = {
Bucket: 'name',
Delimiter: '',
Prefix: `${data}/`
}
var files = []
await s3.listObjects(params, async function (err, data) {
if (err) throw err;
if (data.Contents.length) {
for (const row of data.Contents) {
try {
const dataContent = await s3.getObject({ Bucket: 'name', Key: row.Key }).promise();
files.push({
filename: dataContent.Body,
content: dataContent.Metadata.file_name,
})
} catch(err) {
console.log('S3 Get Object Error', err);
}
}
}
return files;
});
}
Using callback everywhere will make the code unreadable. I'd suggest you convert your response to promise using promise() with s3 sdk's methods.
Similarly, You can convert listObjects method to promise. It'll make code more readable.
Note: I've added few variables from my end into the code, please make sure to change them according to your code context.

Return values from AWS DynamoDB JavaScript SDK are undefined

I am currently using the JavaScript AWS SDK for DynamoDB and I am trying to parse the data I get from a DynamoDB call, but it always returns undefined. It does print the result successfully, but trying to store that data in a variable is unsuccessful. Below is how I am attempting to do it.
const AWS = require("aws-sdk");
AWS.config.update({ region: "us-east-1" });
const dynamoDb = new AWS.DynamoDB({ apiVersion: "2012-08-10" });
const promisify = foo =>
new Promise((resolve, reject) => {
foo((error, result) => {
if (error) {
reject(error);
} else {
resolve(result);
}
});
});
const params2 = {
TableName: "Users",
Key: {
userID: { S: "123456789" },
},
};
const test = params => {
dynamoDb.getItem(params, function(err, data) {
if (err) {
console.log("Error", err);
} else {
console.log("Success", data.Item);
return data.Item;
}
});
};
let user = test(params2);
console.log("User:", user);
I believe it has to do with getItem being asynchronous and I have researched how to implement a promise into this code, but I cannot get the correct syntax. Any and all help is greatly appreciated. Thank you.

The problem you have is that you are not returning anything from your test function. So it is expected that you get undefined outside.
Try to do it like this:
// used aws promise api https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/using-promises.html
function test(params) {
return dynamoDb.getItem(params).promise();
}
test(params2).then(
user => {
console.log('User:', user);
},
error => {
console.log('Error:', error);
},
);
Some further read:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function

How can I access a function within a router javascript file via node?

I'm handling with an webapplication that is not mine and now I've got to send a hundred e-mails.
Unfortunately, the code is not documented and not so well written, that means i have to go testing it to discover what I am able to do and what I'm not. but I don't know how to access this function that is on the code via node. Is it actually possible to do it? Here's the code:
router.post('/aprovadosemail', miPermiso("3"), (req, res) => {
var templatesDir = path.resolve(__dirname, '..', 'templates');
var emailTemplates = require('email-templates');
// Prepare nodemailer transport object
emailTemplates(templatesDir, function(err, template) {
if (err) {
console.log(err);
} else {
var users = [];
projetoSchema.find({"aprovado":true, "categoria":"Fundamental II (6º ao 9º anos)"}, function (err, docs) {
if (err) throw err;
//console.log(docs);
docs.forEach(function(usr) {
let url = "http://www.movaci.com.b/projetos/confirma/"+usr._id+"/2456";
let url2 = "http://www.movaci.com.br/projetos/confirma/"+usr._id+"/9877";
users.push({'email': usr.email, 'projeto': usr.nomeProjeto, 'url': url, 'url2': url2});
});
for (var i = 0; i < users.length; i++) {
console.log(users[i]);
}
const transporter = nodemailer.createTransport(smtpTransport({
host: 'smtp.zoho.com',
port: 587,
auth: {
user: "generic#mail.com",
pass: "genericpassword"
},
getSocket: true
}));
var Render = function(locals) {
this.locals = locals;
this.send = function(err, html, text) {
if (err) {
console.log(err);
} else {
transporter.sendMail({
from: 'no-reply4#movaci.com.br',
to: locals.email,
subject: 'MOVACI - Projeto aprovado!',
html: html,
text: text
}, function(err, responseStatus) {
if (err) {
console.log(err);
} else {
console.log(responseStatus.message);
}
});
}
};
this.batch = function(batch) {
batch(this.locals, templatesDir, this.send);
};
};
// Load the template and send the emails
template('rateada', true, function(err, batch) {
for(var user in users) {
var render = new Render(users[user]);
render.batch(batch);
};
});
res.send('ok');
});
};
});
});

Seems like previous dev did not knew email-templates package deeply (at least have not read how it works).
So in fact it has send method, You can create an email object from email-templates and pass necessary defaults, then You call .send method of it by passing dynamical parts - it just simply merge additional params passed in send arguments, sends mail using nodemailer inside of promise which it returns.
If it's interesting for You - read source code of it: https://github.com/niftylettuce/email-templates/blob/master/index.js
I tried to simplify it modular parts using promises.
I've not debugged it, but You may check my solution and fix it as You wish.
Have 2 files (to routing from handler separately, it may have variables that may conflict and etc):
1) methods/users/aprovadosEmail.js:
const
Email = require('email-templates'),
const
emailTemplatesDir = path.resolve(__dirname + '/../../templates'),
smtpTransportConfig = {
host: 'smtp.zoho.com',
port: 587,
secure: false,
auth: {
user: "no-reply4#movaci.com.br",
pass: "some-password-here"
}
},
createEmail = (template, subject) => {
return new Email({
views: {
root: emailTemplatesDir,
},
transport: smtpTransportConfig,
template,
message: {
from: 'no-reply4#movaci.com.br',
subject
}
});
},
getApprovedUsers = () => {
return new Promise((resolve, reject) => {
const criteria = {
aprovado: true,
categoria:"Fundamental II (6º ao 9º anos)"
};
projetoSchema.find(
criteria,
(error, docs) => {
if(error) return reject(error);
const users = docs.map(doc => {
return {
email: doc.email,
projeto: doc.nomeProjeto,
url: "http://www.movaci.com.b/projetos/confirma/"+doc._id+"/2456",
url2: "http://www.movaci.com.br/projetos/confirma/"+doc._id+"/9877"
};
});
resolve(users);
});
});
},
sendMailToUser = (mail, user) => {
return mail.send({
message: {
to: user.email
},
locals: user
});
},
broadcastMailToUsers = (mail, users) => {
return Promise
.all(
users
.map(user => sendMailToUser(mail, user))
);
};
module.exports = (req, res) => {
const mail = createEmail('rateada', 'MOVACI - Projeto aprovado!'); // mail object
getApprovedUsers()
.then(users => broadcastMailToUsers(mail, users))
.then(result => {
console.log('Result of broadcast:', result);
res.send('ok');
})
.catch(error => {
res.status(500).send(error);
});
};
2) current routes file where routing part that uses module file:
router.post(
'/aprovadosemail',
miPermiso("3"),
require(__dirname+'/../methods/users/aprovadosEmail')
);