Develop Reference

Socket hangup due to wrong handling of promises

I have script to move data from one platform to another. The source db allows only 100 records to be fetched in a single request. So I created a routine to fetch by batches of 100 which works fine I guess.
Now I try to process each records of 100 and do the necessary transformations (which involves axios call to get certain data) and create a record in firebase firestore.
Now when I run this migration in firebase express node, I get socket hang up ECONNRESET.
I know this is caused by wrong handling of promises.
Here is what my code looks like:
import { scrollByBatches } from "../helpers/migrations/apiScroll";
import { createServiceLocation } from "../helpers/locations";
const mapServiceLocationData = async (serviceLocation: any, env: string) => {
try {
const migratedServiceLocation: any = {
isMigrated: true,
id: serviceLocation._id,
};
if (serviceLocation.list?.length) {
await Promise.all(serviceLocation.ids.map(async (id: string) => {
const { data } = await dbEndPoint.priceMultiplier({ id }); // error says socket hangup on this call
let multiplierUnit;
let serviceType;
if (data.response._id) {
multiplierUnit = data.response;
const result = await dbEndPoint.serviceType({ id: multiplierUnit.service_custom_service_type }); // error says socket hangup on this call
if (result.data.response._id) {
serviceType = result.data.response.type_text;
migratedServiceLocation.logs = [...multiplierUnit.history_list_text, ...migratedServiceLocation.logs];
}
}
}));
}
await createServiceLocation(migratedServiceLocation); // create record in destination db
} catch (error) {
console.log("Error serviceLocation: ", serviceLocation._id, JSON.stringify(error));
}
return null; // is this even necessary?
};
export const up = async () => {
try {
// get 100 docs from source db => process it.. => fetch next 100 => so on...
await scrollByBatches(dbEndPoint.serviceLocation, async (serviceLocations: any) => {
await Promise.all(
serviceLocations.map(async (serviceLocation: any) => {
await mapServiceLocationData(serviceLocation);
})
);
}, 100);
} catch (error) {
console.log("Error", JSON.stringify(error));
}
return null; // is this even necessary?
};
The error I get in firebase functions console is:
For clarity on how the fetch by batches looks like:
const iterateInBatches = async (endPoint: any, limit: number, cursor: number, callback: any, resolve: any, reject: any) => {
try {
const result = await endPoint({ limit, cursor });
const { results, remaining }: any = result.data.response;
if (remaining >= 0) {
await callback(results);
}
if ((remaining)) {
setTimeout(() => {
iterateInBatches(endPoint, limit, (cursor + limit), callback, resolve, reject);
}, 1000); // wait a second
} else {
resolve();
}
} catch (err) {
reject(err);
}
};
export const scrollByBatches = async (endPoint: any, callback: any, limit: number, cursor: number = 0) => {
return new Promise((resolve, reject) => {
iterateInBatches(endPoint, limit, cursor, callback, resolve, reject);
});
};
What am I doing wrong? I have added comments in the code sections for readability.
Thanks.

There are two cases when socket hang up gets thrown:
When you are a client
When you, as a client, send a request to a remote server, and receive no timely response. Your socket is ended which throws this error. You should catch this error and decide how to handle it: whether to retry the request, queue it for later, etc.
When you are a server/proxy
When you, as a server, perhaps a proxy server, receive a request from a client, then start acting upon it (or relay the request to the upstream server), and before you have prepared the response, the client decides to cancel/abort the request.
I would suggest a number of possibilities for you to try and test that might help you solve your issue of ECONNRESET :
If you have access to the source database, you could try looking
there for some logs or metrics. Perhaps you are overloading the
service.
Quick and dirty solution for development: Use longjohn, you get long
stack traces that will contain the async operations. Clean and
correct solution: Technically, in node, whenever you emit an 'error'
event and no one listens to it, it will throw the error. To make it
not throw, put a listener on it and handle it yourself. That way you
can log the error with more information.
You can also set NODE_DEBUG=net or use strace. They both provide you
what the node is doing internally.
You could restart your server and run the connection again, maybe
your server crashed or refused the connection most likely blocked by
the User Agent.
You could also try running this code locally, instead of in cloud
functions to see if there is a different result. It's possible that
the RSG/google network is interfering somehow.
You can also have a look at this GitHub issue and stackoverflow
thread to see the common fixes for the ECONNRESET issue and see if
those help resolve the issue.

saving documents to mongoDB preventing duplicates

I'm trying to save multiple documents in mongodb using mongoose; and I'm also willing to prevent duplicates. my function looks sth like this:
const Stock = require('./models/stock')
let _symbol = 'symb'
const writeToDB = async (dataObj) => {
try {
let stock = await Stock.find({symbol : _symbol } , function (err) {
if(err) return null
})
if (!stock) {
stock = new Stock({
dataObj
})
await stock.save()
console.log(`${symbol} is successfully saved to database`)
} else {
stock = await Stock.updateMany(
dataObj, function (err) {
if (err) {
console.log(err)
} else {
console.log(`${symbol} successfully added`)
}
})
}
} catch (error) {
console.log(error)
}
}
but I keep getting timeout error. can someone pls inform me what's wrong.
update
with a well handled connection approach findOneAndUpdate()works fine

Using the upsert option, in findOneAndUpdate(). An upsert behaves like a normal findOneAndUpdate() if it finds a document that matches filter. But, if no document matches filter, MongoDB will insert one by combining filter and update as shown below
var query = {symbol : _symbol };
try{
let result = await Stock.findOneAndUpdate(query, dataObj, {upsert: true})
}
catch(err){
console.log();
}
if you have a big collection, for increase speed findOneAndUpdate(), you should indexed symbol field.
when you use async await, it's better don't use callback and use try catch

I think the best, simply and easy way to prevent duplicate values is use unique value in the schema.
So your Stock schema has to have something similar to this:
symbol:{
type: String, // or whatever
unique: true
}
If you try to insert two object with same value, mongoose will trhow an error like:
MongoError: E11000 duplicate key error dup key: { : "repeatedSymbol" }
Also you can check the documentation.

Unable to handle mysql error using async await

I am new to node js. I learned am learning to execute MySQL queries using async/await instead of callback functionality.
I am able to get data on success but If any error occurs, I couldn't catch that error. Please suggest me how to achieve so.
My MySQL code to create a row in the education table
Education.create = async (newEducation) => {
try {
const res = await sql.query("INSERT INTO education SET ?", newEducation);
return await {id: res.insertId, ...newEducation};
}catch(err) {
// I couldn't catch DUPLICATE ENTRY ERROR or any other MYSQL generated error
console.log(": ---------------------------")
console.log("Education.create -> err", err)
console.log(": ---------------------------")
return err;
}
}

ReactJS / Axios / Firebase - Passing response data from POST to use in a PUT

I am using Axios.post to create a new record. With this new record there is a unique key assigned to this record from Firebase that I need in my next PUT operation.
Currently I'm using nested .then and it seems to be working as it should. I'm just unsure how I can convert this to a Promise.all without knowing the unique key to pass into my second query. It seems it's susceptible to problems the way I have it structured. Is there a way I can use Promise.all to ensure both queries executed? Or maybe it's fine as is?
Fully functioning sandbox is here: https://codesandbox.io/s/react-sample-5nc1z?file=/index.js
And here is the part in question:
postColumn = (columnArr) => {
axios
.post(
"/workflow/boards/" + this.state.boardID + "/columns.json",
columnArr
)
.then((response) => {
// console.log(response);
const newColumnIds = [...this.state.columnIds, response.data.name];
const newColState = {
columns: {
...this.state.columns,
[response.data.name]: { columnvalue: columnArr.columnvalue },
},
};
this.setState(newColState);
axios
.put(
"/workflow/boards/" + this.state.boardID + "/columnIds.json",
newColumnIds
)
.then((response) => {
const newColIdState = {
columnIds: newColumnIds,
};
this.setState(newColIdState);
})
.catch((error) => {
console.log("Error in putColumnIds", error);
});
})
.catch((error) => {
console.log("Error in postColumn", error);
});
};

If your second request is dependent on some data from your first request, Promise.all will not work. The purpose of Promise.all is to run two Promises (in your case, network requests) simultaneously, so if you are unable to do that (because you rely on one completing first, you cannot use it). That being said, your code looks totally fine and as long as it works, there should be nothing wrong with your implementation.
See more: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all

Update all Rows and then send response sequelize

I'm Pretty new doing this kind of tasks in Node. I have a POST method (express) wich recive an array of objects like:
[{F1:'123',F2:'a',F3:'b',F4:'Z'},
{F1:'124',F2:'a',F3:'b',F4:'Z'},
{F1:'125',F2:'a',F3:'b',F4:'Z'},
{F1:'126',F2:'a',F3:'b',F4:'Z'},]
Then, i need do an Update for every object in the array.
Im using sequelize :
MODEL.update(
{
att4: art['F4'],
},
{
where:{
att1: {$eq: art['F1']}
}
}).then(function(result)
{
res.status(200).send();
}).catch(function(err)
{
res.status(500).send(err);
})
And this work for 1 object.
But i need the following: WHEN ALL THE UPDATES are processed, then send a response.
I try with
req.body.forEach(function(o)
{
updateO(o)
},this);
and in updateO(o) do the Model.Update , but i don't achieve the needed result.
Sorry for the bad english, hope can understand me and thank's for your help.

Read about promises, all Sequelize's functions return promises, you can handle multiple querying through Promise.all
var objects = req.body;
Promise.all(objects.map(object=>{
return MODEL.update({att4: object.F4},{where:{att1: object.F1});
})).then(function(result){
res.status(200).send();
}).catch(function(err){
res.status(500).send(err);
})