Develop Reference

s3.getObject not working from dev environment

This is my code, which works fine if i run it from my local using local aws account , but it doesn't work from my dev environment. S3.getobject api doesnt get executed and code prints the next log skipping the getobject call :
const unzipFromS3 = (key) => {
return new Promise(async (resolve, reject) => {
log.info("inside unzipfroms3");
var zlib = require('zlib');
// let fileName = _.replace(key, 'Root/', '');
let options = {
'Bucket': config.bucketName,
'Key': "Root/" + key,
}
log.info("Key:", options);
await s3.getObject(options).on('error', error => {
log.error(error) }).promise().then((res) => {
yauzl.fromBuffer(res.body, { lazyEntries: true }, function (err, zipfile) {
log.info("Inside Yauzl")
if (err) throw err;
zipfile.readEntry();
zipfile.on("entry", function (entry) {
if (/\/$/.test(entry.fileName)) {
zipfile.readEntry();
} else {
zipfile.openReadStream(entry, function (err, readStream) {
if (err) throw err;
// readStream.pipe(fs.createWriteStream(`result/${entry.fileName}`));
readStream
.pipe(uploadFromStream(s3));
function uploadFromStream(s3) {
log.info("Inside uploadFromStream")
var pass = new Stream.PassThrough();
let options = {
'Bucket': config.bucketName,
'Key': entry.fileName,
}
var params = { ...options, Body: pass };
s3.upload(params, function (err, data) {
log.error(err, data);
});
return pass;
}
readStream.on("end", function () {
zipfile.readEntry();
});
});
}
});
});
});
});
};

In order to use await, i.e. the promised based version of S3.getObject(), you must add the promise() method to your method call as explained in the Using JavaScript Promises chapter of the AWS SDK developer guide. Moreover, there is also an Using async/await chapter that you can look into.
In your case, the code can be modified to something like:
await s3.getObject(options).promise()
.then((res) => {
yauzl.fromBuffer(/* more code */);
});

How can I use async/await in 'new Promise' block to fetch data from an api?

I'm unable to use async/await inside a 'new Promise' block which throws await is only valid in async function error. I don't know if it's even possible but I need to use it or a way to achieve what I want. Here's what I'm trying to do:
A user types in a query and it goes through a recursive function called doesExist. I have to use a recursive function because the API doesn't always provide data for a query. The recursive function will try 3 times, that is, it will send 3 api requests for the data before returning an error saying 'data couldn't be fetched'. In the function, I return a new promise as I have to make api requests from there.
To fetch data, I used request before but I want to use axios now with async/await. So how can I use axios with async/await inside a new Promise block?
This is the code with request:
router.get('/example', async (req, res) => {
try{
const query = 'Superman';
const data = await doesExist(query);
if(!data) {
console.log('No data');
}
res.render('../views/example', { data, query });
}catch(err) {
console.log(err);
}
});
const doesExist = (query, retries = 0) => {
const url = `http://api.example.com/json?fields=${query}`;
const maxRetries = 3;
return new Promise(( resolve, reject ) => {
const retry = () => {
if (retries < maxRetries) {
resolve(doesExist(query, retries + 1));
} else {
reject(`Could not get the data after ${retries} retries.`);
}
};
request(url, function (error, response, body) {
if (!error && response.statusCode === 200) {
const data = JSON.parse(body);
resolve(data);
} else {
retry();
}
});
});
};
And this is what I tried with async/await which throws the error:
const doesExist = async (query, retries = 0) => {
const url = `http://api.example.com/json?fields=${query}`;
const maxRetries = 3;
return new Promise(( resolve, reject ) => {
const retry = () => {
if (retries < maxRetries) {
resolve(doesExist(query, retries + 1));
} else {
reject(`Could not get the data after ${retries} retries.`);
}
};
const data = await axios.get(url);
if(data.statusCode === 200) {
resolve(data);
}else{
retry();
}
});
};
If it's not possible to use async/await inside promise block then please show me how I can achieve it. Thanks

Never pass an async function as the executor to new Promise! If you want to use async/await and axios which already does return a promise, you don't need - and shouldn't use - the Promise constructor to promisify a callback API. Just write
async function doesExist(query, retries = 0) {
const url = `http://api.example.com/json?fields=${query}`;
const maxRetries = 3;
const data = await axios.get(url);
if (data.statusCode === 200) {
return data;
} else if (retries < maxRetries) {
return doesExist(query, retries + 1);
} else {
throw new Error(`Could not get the data after ${retries} retries.`);
}
}

The function the promise calls needs to be an "async" function like so.
new Promise (async (Resolve) => {
await new Promise (async (_Resolve) => {
console.log ("A");
_Resolve ();
});
console.log ("B");
Resolve ();
});

NodeJS: Wait for all foreach with Promises to finish but never actually finishes

I am working with Nodejs. I have a forEach which is async as I have to wait for a result inside the forEach. As a result, I need to wait for the forEach to finish and then carry on with the result of the loop. I found several solutions for waiting for the forEach, one of them is using Promises. I did though, and these promises are created, however, the code after the forEach (and therefore the promises) are finished, is never actually executed (console.log is not printed). And the NodeJS function just ends without any errors.
Here is my Code:
var Client = require('ssh2').Client;
// eslint-disable-next-line no-undef
var csv = require("csvtojson");
// eslint-disable-next-line no-undef
var fs = require("fs");
// eslint-disable-next-line no-undef
const config = require('./config.json');
// eslint-disable-next-line no-undef
const os = require('os');
let headerRow = [];
let sumTxAmount = 0;
const filenameShortened = 'testFile';
let csvLists = [];
let csvFile;
const options = {
flags: 'r',
encoding: 'utf8',
handle: null,
mode: 0o664,
autoClose: true
}
var conn = new Client();
async function start() {
const list = await getCSVList();
let content = fs.readFileSync('./temp.json', 'utf8');
content = JSON.parse(content);
var promises = list.map(function(entry) {
return new Promise(async function (resolve, reject) {
if (!content['usedFiles'].includes(entry.filename)) {
const filename = entry.filename;
csvFile = await getCsv(filename);
csvLists.push(csvFile);
console.log('here');
resolve();
} else {
resolve();
}
})
});
console.log(promises)
Promise.all(promises)
.then(function() {
console.log(csvLists.length, 'length');
})
.catch(console.error);
}
start();
The "here" is printed once (not 8 times as the arrays length is 8), but there are 8 promises created. The lower part where I am printing the length of the array is not executed.
Can anyone tell me what I am doing wrong? Am I using Promises and forEach falsely as I have to do an await inside the forEach?
Note: getCSVList() and getCsv() are functions to get Csvs from an sftp server:
function getCSVList() {
return new Promise((resolve, reject) => {
conn.on('ready', function () {
conn.sftp(function (err, sftp) {
if (err) throw err;
sftp.readdir(config.development.pathToFile, function (err, list) {
if(err) {
console.log(err);
conn.end();
reject(err);
} else {
console.log('resolved');
conn.end();
resolve(list);
}
})
})
}).connect({
host: config.development.host,
port: config.development.port, // Normal is 22 port
username: config.development.username,
password: config.development.password
// You can use a key file too, read the ssh2 documentation
});
})
}
function getCsv(filename) {
return new Promise((resolve, reject) => {
conn.on('ready', function () {
conn.sftp(function (err, sftp) {
if (err) reject(err);
let csvFile = sftp.createReadStream(`${config.development.pathToFile}/${filename}`, options);
// console.log(csvFile);
conn.end();
resolve(csvFile);
})
}).connect({
host: config.development.host,
port: config.development.port, // Normal is 22 port
username: config.development.username,
password: config.development.password
// You can use a key file too, read the ssh2 documentation
});
});
}
The output in my console from all the console logs is:
`➜ node server.js
resolved
[ Promise { <pending> },
Promise { <pending> },
Promise { <pending> },
Promise { <pending> },
Promise { <pending> },
Promise { <pending> },
Promise { <pending> },
Promise { <pending> } ]
here`

Break up your problem into pieces, confirming they work along the way.
You are not using the stream correctly, among other things.
I made a working example with ssh2-sftp-client so you can maybe use it as a starting point.
Working example :
var fs = require('fs'); var _ = require('underscore');
var SFTPClient = require('ssh2-sftp-client');
const CONFIG = {
"SSH_CONN_OPTS":{"host":"XXXXXXXX","port":22,"username":"XXXXXXXX","password":"XXXXXXXX"},
"CSV_DIRECTORY":"/var/www/html"
}
//---------------
//.:The order-logic of the script is here
function StartScript(){
console.log("[i] SSH Connection")
LoadValidationFile(()=>{
InitializeSFTP(()=>{ console.log("[+] SSH Connection Established")
ListRemoteDirectory((list)=>{ console.log(`[i] Total Files # ${CONFIG.CSV_DIRECTORY} : ${list.length}`)
//console.log(list) //:now you have a 'list' of file_objects, you can iterate over to check the filename
var csvFileList = [] //store the names of the files you will request after
_.each(list,(list_entry)=>{ console.log(list_entry)
if(!CONFIG.USED_FILES.includes(list_entry.name)){ csvFileList.push(list_entry.name) }
})
//:now loop over the new final list of files you have just validated for future fetch
GenerateFinalOutput(csvFileList)
})
})
})
}
//.:Loads your validation file
function LoadValidationFile(cb){
fs.readFile(__dirname+'/temp.json','utf8',(err,data)=>{ if(err){throw err}else{
var content = JSON.parse(data)
CONFIG.USED_FILES = content.usedFiles
cb()
}})
}
//.:Connects to remote server using CONFIG.SSH_CONN_OPTS
function InitializeSFTP(cb){
global.SFTP = new SFTPClient();
SFTP.connect(CONFIG.SSH_CONN_OPTS)
.then(()=>{cb()})
.catch((err)=>{console.log("[!] InitializeSFTP :",err)})
}
//.:Get a list of files from a remote directory
function ListRemoteDirectory(cb){
SFTP.list(`${CONFIG.CSV_DIRECTORY}`)
.then((list)=>{cb(list)})
.catch((err)=>{console.log("[!] ListRemoteDirectory :",err)})
}
//.:Get target file from remote directory
function GetRemoteFile(filename,cb){
SFTP.get(`${CONFIG.CSV_DIRECTORY}/${filename}`)
.then((data)=>{cb(data.toString("utf8"))}) //convert it to a parsable string
.catch((err)=>{console.log("[!] ListRemoteDirectory :",err)})
}
//-------------------------------------------
var csvLists = []
function GenerateFinalOutput(csv_files,current_index){ if(!current_index){current_index=0}
if(current_index!=csv_files.length){ //:loop
var csv_file = csv_files[current_index]
console.log(`[i] Loop Step #${current_index+1}/${csv_files.length} : ${csv_file}`)
GetRemoteFile(csv_file,(csv_data)=>{
if(csv_data){csvLists.push(csv_data)}
current_index++
GenerateFinalOutput(csv_files,current_index)
})
}else{ //:completed
console.log("[i] Loop Completed")
console.log(csvLists)
}
}
//------------
StartScript()
Good luck!

Promise.all is a method that will return a promise object, but you are not waiting for your start method to execute.
function getCSVList() {
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve([1, 2, 3, 4]);
}, 1000);
});
}
function getCsv(params) {
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve(params);
}, 1000);
});
}
async function start() {
const list = await getCSVList();
const promises = list.map(item => {
return new Promise(async function (resolve, reject) {
const csvFile = await getCsv(item);
console.log('here');
resolve(csvFile);
});
});
return Promise.all(promises);
}
start().then(res => {
console.log(res);
});

await for indexdb event in async function

I'm trying to return a custom object from a async function that works as wrapper for a put using indexdb.
Using Promises this is easy.
However, using async/await became more challenging...
const set = async (storeName, key, value) => {
if (!db)
throw new Error("no db!");
try {
const result = {};
let tx = db.transaction(storeName, "readwrite");
let store = tx.objectStore(storeName);
let r = store.put({ data: key, value: value });
console.log(r);
r.onsuccess = async () => {
console.log('onsuccess');
result.something = true;
}
r.onerror = async () => {
console.log('onerror');
result.something = false;
}
await r.transaction.complete; // ok... this don't work
// how can I await until onsuccess or onerror runs?
return result;
} catch (error) {
console.log(error);
}
}
The ideia is to return a composed object... however all my attemps fails as onsuccess runs after returning the result.
I googled a lot and could't find a way to proper await for onsuccess/onerror events.
I know that returning a Promise is more easy as resolve(result) would end returning what I want... but i'm trying to learn to make same code using async/await.
Thank you so much,

Try this:
function set(db, storeName, key, value) {
return new Promise((resolve, reject) => {
let result;
const tx = db.transaction(storeName, 'readwrite');
tx.oncomplete = _ => resolve(result);
tx.onerror = event => reject(event.target.error);
const store = tx.objectStore(storeName);
const request = store.put({data: key, value: value});
request.onsuccess = _ => result = request.result;
});
}
async function callIt() {
const db = ...;
const result = await set(db, storeName, key, value);
console.log(result);
}
Edit, since you insist on using the async qualifier for the set function, you can do this instead. Please note I find this pretty silly:
async function set(db, storeName, key, value) {
// Wrap the code that uses indexedDB in a promise because that is
// the only way to use indexedDB together with promises and
// async/await syntax. Note this syntax is much less preferred than
// using the promise-returning function pattern I used in the previous
// section of this answer.
const promise = new Promise((resolve, reject) => {
let result;
const tx = db.transaction(storeName, 'readwrite');
tx.oncomplete = _ => resolve(result);
tx.onerror = event => reject(event.target.error);
const store = tx.objectStore(storeName);
const request = store.put({data: key, value: value});
request.onsuccess = _ => result = request.result;
});
// We have executed the promise, but have not awaited it yet. So now we
// await it. We can use try/catch here too, if we want, because the
// await will translate the promise rejection into an exception. Of course,
// this is also rather silly because we are doing the same thing as just
// allowing an uncaught exception to exit the function early.
let result;
try {
result = await promise;
} catch(error) {
console.log(error);
return;
}
// Now do something with the result
console.debug('The result is', result);
}

Ultimately you'll end up wrapping IDB in a promise-friend library, but for your specific need, you could use something like this:
function promiseForTransaction(tx) {
return new Promise((resolve, reject) => {
tx.oncomplete = e => resolve();
tx.onabort = e => reject(tx.error);
});
}
And then in your code you can write things such as:
await promiseForTransaction(r.tx);
... which will wait until the transaction completes, and throw an exception if it aborts. (Note that this requires calling the helper
before the transaction could possibly have completed/aborted, since
it won't ever resolve if the events have already fired)

I can't confirm it right now but I think it should be await tx.complete instead of await r.transaction.complete;.
But a general solution that would work even if the API would not support Promises directly would be to wrap a new Promise around the onsuccess and onerror and use await to wait for that Promise to resolve, and in your onsuccess and onerror you then call the resolve function:
const set = async (storeName, key, value) => {
if (!db)
throw new Error("no db!");
try {
const result = {};
let tx = db.transaction(storeName, "readwrite");
let store = tx.objectStore(storeName);
let r = store.put({
data: key,
value: value
});
console.log(r);
await new Promise((resolve, reject) => {
r.onsuccess = () => {
console.log('onsuccess');
result.something = true;
resolve()
}
r.onerror = () => {
console.log('onerror');
result.something = false;
// I assume you want to resolve the promise even if you get an error
resolve()
}
})
return result;
} catch (error) {
console.log(error);
}
}
I would furhter change it to:
try {
await new Promise((resolve, reject) => {
r.onsuccess = resolve
r.onerror = reject
})
console.log('success');
result.something = true;
} catch(err) {
console.log('error');
result.something = false;
}

Look for Promise bluebird code review for node.js

When and where need to use new Promise(Function<Function resolve, Function reject> resolver) -> Promise
My Sample code:
userInfo.js
var Promise = require('bluebird');
var winston = require('winston');
var _ = require('lodash');
var request = Promise.promisify(require("request"));
exports.getWeather = function (data) {
var cityName = data.userProfile.city;
return request("http://0.0.0.0:3003/api/Weather/byCity?city=" + cityName).spread(function (res, body) {
var result = JSON.parse(body).data;
return _.merge(data, result);
});
};
exports.getUserProfile = function (userId) {
return new Promise(function (resolve, reject) {
request("http://0.0.0.0:3003/api/UserProfile/getUserProfile?id=" + userId).spread(function (res, body) {
var result = JSON.parse(body).data;
resolve(result);
});
})
};
exports.getEvents = function (data) {
var cityName = data.userProfile.city;
return request("http://0.0.0.0:3003/api/Events/byCity?city=" + cityName).spread(function (res, body) {
var result = JSON.parse(body).data;
return _.merge(data, result);
});
};
exports.getFashion = function (data) {
var gender = data.userProfile.gender;
return request("http://0.0.0.0:3003/api/Fashion/byGender?gender=" + gender).spread(function (res, body) {
var result = JSON.parse(body).data;
return _.merge(data, result);
});
};
exports.displayDetail = function (data) {
console.log(data);
};
Above code I try call in 2 way in promise
getUserProfile.js
var userInfo = require('./userInfo');
module.exports = function(){
return userInfo.getUserProfile(3)
.then(userInfo.getFashion)
.then(userInfo.getEvents)
.then(userInfo.getWeather)
.then(userInfo.displayDetail)
.catch(function (e) {
console.log('Error:');
console.error(e.stack)
})
.finally(function () {
console.log('done');
});
}
2nd way:
getUserInformation.js
var userInfo = require('./userInfo');
module.exports = function () {
return new Promise(function (resolve, reject) {
resolve(3);
})
.then(userInfo.getUserProfile)
.then(userInfo.getFashion)
.then(userInfo.getEvents)
.then(userInfo.getWeather)
.then(userInfo.displayDetail)
.catch(function (e) {
console.log('Error:');
console.error(e.stack)
})
.finally(function () {
console.log('done');
});
};
getDetails.js
var userInfo = require('./getUserInformation');
userInfo()
.then(function(){
console.log('getDetails done')
})
.catch(function (e) {
console.log('Error:');
console.error(e.stack)
})
.finally(function () {
console.log('done');
});
please let me know what the difference and is there any issues by using these way?

exports.getUserProfile = function (userId) {
return new Promise(function (resolve, reject) {
request("http://0.0.0.0:3003/api/UserProfile/getUserProfile?id=" + userId).spread(function (res, body) {
var result = JSON.parse(body).data;
resolve(result);
});
})
};
Please don't do this. Just return from the callback, and return the promise created by then, like you have done it in your other three methods.
return userInfo.getUserProfile(3)
.then(…)
vs.
return new Promise(function (resolve, reject) {
resolve(3);
})
.then(userInfo.getUserProfile)
.then(…)
Well, the first one is much more readable and concise. They're pretty much equivalent except for the case that getUserProfile does throw synchronously, which it shouldn't anyway. Also in the first case getUserProfile is invoked as a method on userInfo, while in the second case it's just a callback function, the this in the calls will be different.
The second pattern can be tremendously simplified though by using Promise.resolve instead of the new Promise constructor:
return Promise.resolve(3)
.then(userInfo.getUserProfile)
.then(…)
This is totally fine, and aligns better with the rest of the chain. Speaking of which, …
.then(userInfo.getFashion)
.then(userInfo.getEvents)
.then(userInfo.getWeather)
where each of the functions returns a promise that resolves with
additional data merged into its argument
is not exactly the best way to solve this. Yes, it ensures that these three functions are called after each other, and is an acceptable pattern for that case. However, in your case you're mixing the request calls to the API with that argument-extraction and result-merging in the same function; which by the separation of concerns you shouldn't. Rather make the functions pure
exports.… = function (arg) {
return request("http://0.0.0.0:3003/api/…?…=" + arg).spread(function (res, body) {
return JSON.parse(body).data;
});
};
And now you can combine them separately - and not only in sequence, but also in parallel:
userInfo.getUserProfile(3)
.then(function(data) {
var p = data.userProfile;
return Promise.prop({
userProfile: 0,
fashion: userInfo.getFashion(p.gender), // `\
events: userInfo.getEvents(p.city), // }=> execute requests in parallel
weather: userInfo.getWeather(p.city) // ./
});
})
.then(userInfo.displayDetail)
.catch(function (e) {
console.error('Error:', e.stack)
});

The first way is much more readable, and there's no benefit to starting the chain with a promise that returns a constant as in your second way.
They both do effectively the same thing, with one caveat: In your second example (Starting the chain with a Promise), the getUserProfile call will be run on the next tick (Similar to if you'd thrown it in a setTimeout 0) rather than atomically.