I cannot find a standard way to paginate my update in Cassandra using the Node connector. I cannot find any resource that would explain how to do it or whether it is possible at all.
My function looks like this:
async function whitelistCustomers(account, customerIds) {
let data = [account._id.toString(), customerIds]
let options = { prepare: true, fetchSize: 200 }
let query = "UPDATE customers SET reason_blacklist=null WHERE id_account = ? AND id IN ?"
await new Promise((resolve, reject) => {
function rowCallback(n, rows) {}
function callback(err, result) {
if (err) reject(err)
else resolve()
}
client.eachRow(query, data, options, rowCallback, callback)
})
}
But it does not seem to work. When there are lots of customerIds, the query fails with PayloadTooLargeError: too many parameters.
I could solve this by making a custom JS pagination. I would iterate over customerIds and make multiple queries to Cassandra with small portions of customerIds each, but if I could find a more standard way to do this it would be best.
do {
let paginatedCustomerIds = customerIds.splice(0, 200)
// make a request with paginatedCustomerIds
} while (customerIds.length > 0)
Is it possible to paginate the update request? If yes how do I do it?
Related
Currently I have a back-end endpoint which receives a page query parameter and returns 10 Brand entries based on that page. Now I need to be able to GET all brands too, without all the pagination functionality but I'm not sure what is the correct way to do it.
Option 1 - Create 2 endpoints, 'getBrands' which will return all brands without the pagination functionality, and 'getBrandsByPage' which will return 10 brands based on the page query parameter.
Option 2 - Have a single endpoint which will check if there is a query parameter page first. If there is, it will execute one piece of block, and if there isn't, another. This will make the function bigger and a bit more convoluted with all the ifs
This is what I have right now:
module.exports.getBrands = async (req, res, next) => {
let page = req.query.page
let limit = 10
let offset = ( page - 1 ) * limit
let brands
try {
brands = await Brand.findAndCountAll({
limit: limit,
offset: offset,
order: [
['createdAt', 'DESC']
]
})
} catch (e) {
console.log(e)
}
if (brands) {
let pages = Math.ceil( brands.count / limit )
res.status(200).json({
brands: brands.rows,
totalItems: brands.count,
totalPages: pages,
})
} else {
res.status(500)
}
}
The one endpoint alternative would look like this:
module.exports.getBrands = async (req, res, next) => {
let page = req.query.page
let brands
try {
if (page) {
let limit = 10
let offset = ( page - 1 ) * limit
brands = await Brand.findAndCountAll({
limit: limit,
offset: offset,
order: [
['createdAt', 'DESC']
]
})
if (brands) {
let pages = Math.ceil( brands.count / limit )
res.status(200).json({
brands: brands.rows,
totalItems: brands.count,
totalPages: pages,
})
} else {
res.status(500)
}
} else {
brands = await Brand.findAll({
order: [
['createdAt', 'DESC']
]
})
if (brands) {
res.status(200).json({
brands: brands
})
} else {
res.status(500)
}
}
} catch (e) {
console.log(e)
}
}
From a REST design point of view, I see no reason for two separate routes to get all brands vs. get all brands (paginated). This is exactly what a query parameter is for (to specify a modifier for the GET). Using only one route makes your API more compact and, if you write documentation for it, should make that documentation cleaner. I would personally design my REST API to optimize the REST design and not to optimize my implementation.
Besides, if you were going to have two completely separate blocks of code for two routes and you like that, you can also have one route with an if and then two functions where the if/else just decides which function to call. You can fit the implementation you like into the better REST design.
You should be able to share most of the code between the two branches. I don't know your database well enough to suggest code sharing improvements for the paged version, but it seems like you have a bit more copied code than should be required.
In addition, you need to change this:
res.status(500);
to:
res.sendStatus(500);
So, that you're actually sending the response, not just setting a state value for a future response.
In addition, you need to be sending a response in the catch() handler.
And, probably add a console.log() about why you're returning a 500. The last thing you want to happen is for your server to be returning 500 and have no idea that it's doing that or no idea why it's doing that or not be able to tell a customer why they're getting a 500.
P.S. Note that programming without semi-colons will bite you at some point as there are documented pieces of code where the interpreter will not do what you intended without a semi-colon at the end of the previous line. One place where this gets you is if you ever declare an IIFE inside a function body or copy in some code someone else wrote that declares an IIFE.
Here's one suggestion for the combined route:
module.exports.getBrands = async (req, res, next) => {
const query = {order: [['createdAt', 'DESC']]};
try {
if (req.query.page) {
const page = parseInt(req.query.page, 10);
if (isNaN(page) || page < 1) {
throw new Error(`Invalid page query "page=${req.query.page}"`);
}
// build query for generated paged results
const limit = 10;
query.limit = limit;
query.offset = (page - 1) * limit;
const brands = await Brand.findAndCountAll(query);
res.json({
brands: brands.rows,
totalItems: brands.count,
totalPages: Math.ceil(brands.count / limit),
});
} else {
const brands = await Brand.findAll(query);
res.json({ brands });
}
} catch (e) {
console.log(e);
res.sendStatus(500);
}
};
I'm guessing that you're using sequelize. If that's the case, then a successful database query will always return your brands object. There is no reason to check for that to be falsey. If the DB query fails, the promise will reject and you'll be in the catch handler. If no results are found, then the results will be a zero length array which is a legitimate result set to return.
I have a problem where I've got 20+k rows in a csv file and I'm trying to update them based on documents of a matching field in a Mongo DB that contains 350k docs.
The trick is that I need to perform some logic on the matches and then re-update the csv.
I'm using PapaParse to parse/unparse the csv file
Doing something like works to get all my matches
const file = fs.createReadStream('INFO.csv');
Papa.parse(file, {
header: true,
complete: function(row) {
getMatchesAndSave(row.data.map(el => { return el.fieldToMatchOn }));
}
});`
function getMatchesAndSave(fields) {
Order.find({fieldToMatchOn: { $in: fields}}, (err, results) => {
if (err) return console.error(err);
console.log(results);
});
}
That gets me matches fast. However, I can't really merge my data back into the csv bc the csv has a unique key column that Mongo has no idea about.
So all the data is really dependent of what's in the csv.
Therefore I thought of doing something like this
`
const jsonToCSV = [];
for (let row of csvRows) {
db.Collection.find({fieldToMatchOn: row.fieldToMatchOn}, (err, result) => {
//Add extra data to row based on result
row.foo = result.foo;
//push to final output
jsonToCSV.push(row);
}
}
papa.unparse(jsonToCSV);
//save csv to file
The issue with the above implementation (as terribly inefficient as it may seem) - is that the Find calls are asynchronous and nothing gets pushed to jsonToCSV.
Any tips? Solving this with $in would be ideal, are there any ways to access the current element in the $in (so looking for the iterator)..that way I could process on that.
You can try async/await to iterate csvRows array, like this:
const search = async () => {
const jsonToCSV = await Promise.all(csvRows.map(async row => {
/* find returns a promise, so we can use await, but to use await is
mandatory use it inside an async function. Map function not returns a
promise, so this can be solve using Promise.all. */
try {
const result = await db.Collection.find({ fieldToMatchOn: row.fieldToMatchOn });
row.foo = result.foo;
return row;
} catch (e) {
// do somenthing if error
}
}));
papa.unparse(jsonToCSV);
}
// call search function
search();
Check this https://flaviocopes.com/javascript-async-await-array-map to a better understanding.
I'm trying to get list of virtual communities on my Node.js app and then return it with callback function. When i call a getList() method with callback it returns a empty array.
const mysqli = require("../mysqli/connect");
class Communities{
getList(callback){
var list = [];
mysqli.query("SELECT * FROM communities", (err, communities) => {
for(let i = 0; i < communities.length; i++){
mysqli.query("SELECT name FROM users WHERE id='"+ communities[i].host +"'", (err, host) => {
list.push({
"id": communities[i].id,
"name": communities[i].name,
"hostID": communities[i].host,
"hostName": host[0].name,
"verified": communities[i].verified,
"people": communities[i].people
});
});
}
callback(list);
});
}
}
new Communities().getList((list) => {
console.log(list);
});
I need to make for loop to asynchronous and call callback when for loop ends. Please let me know how to do this. Thanks.
Callbacks get really ugly if you have to combine multiple of them, thats why Promises were invented to simplify that. To use Promises in your case you have to create a Promise first when querying the database¹:
const query = q => new Promise((resolve, reject) => mysqli.query(q, (err, result) => err ? reject(err) : resolve(result)));
Now doing multiple queries will return multiple promises, that can be combined using Promise.all to one single promise²:
async getList(){
const communities = await query("SELECT * FROM communities");
const result = await/*³*/ Promise.all(communities.map(async community => {
const host = await query(`SELECT name FROM users WHERE id='${community.host}'`);/*⁴*/
return {
...community,
hostName: host[0].name,
};
}));
return result;
}
Now you can easily get the result with:
new Communities().getList().then(list => {
console.log(list);
});
Read on:
Working with Promises - Google Developers
Understanding async / await - Ponyfoo
Notes:
¹: If you do that more often, you should probably rather use a mysql library that does support promises natively, that safes a lot of work.
²: Through that the requests are done in parallel, which means, that it is way faster than doing one after another (which could be done using a for loop & awaiting inside of it).
³: That await is superfluous, but I prefer to keep it to mark it as an asynchronous action.
⁴: I guess that could also be done using one SQL query, so if it is too slow for your usecase (which I doubt) you should optimize the query itself.
So, I am currently developing some functionality on one of my projects at work. The project is using JavaScript mainly - Node.JS for backend and React.JS for frontend, and I have to admit I am not experienced with either of them. I believe that the code I am writing could look much better and work more efficient if I utilised promises or async/await functionality (prior to asking the question here I read few articles about them, and I am still not sure how to use them in the project the way it actually makes sense, hence I decided to ask community here). I also had a glance at this article, but again I am not sure whether my implementation actually does anything StackOverflow.
At the end of this post I am going to paste some code from both front and backend and hopefully someone will be able to point me into a right direction. To make things clear - I am not asking for anybody to rewrite the code for me, but to explain what it is I'm doing wrong (or not doing at all).
Use case:
User writes a company name in the search bar on the website. Typed string is then sent to the backend via http-request and the database is checked for the entry (to get the company's logo) - here I am running an algorithm to check for spelling mistakes and propose similar names to the one typed, as a result the database may be queried more than 2 times before the result is sent back, but it's always working fine.
Once the response is received by the frontend few things should happen - to start with another request should be sent to the web in order to receive other results. If correct results are received, that should be the end of the function, otherwise it should send another request, to google this time, to get the results from there.
Backend Code:
.post('/logo', (req, res) => {
res.header("Access-Control-Allow-Origin", "*");
res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
if (req.body.key !== "" && req.body.key.trim().length > 0) {
let results = {};
let proposedNames = [];
var promise1 = new Promise((resolve, reject) => {
let getLogo = "SELECT title, img_dir FROM logo_approved WHERE LOWER(title) LIKE LOWER($1)";
let searchedCompanyName = ["%"+req.body.key+"%"];
db.queryDB(getLogo, searchedCompanyName, (data) => {
if (data.rows.length > 0){
results.databaseResults = data.rows;
}
resolve(data.rows);
});
});
// Returns the list of all companies' names from the database
var promise2 = new Promise((resolve, reject) => {
let returnAllNames = "SELECT title, img_dir as img FROM logo_approved";
db.queryDB(returnAllNames, [], (data) =>{
// Compare searched company's name with all names from the database
data.rows.forEach(function(element) {
// If name from the database is similar to the one searched for
// It's saved in propsedNames array and will be used later on for database query
if (jw.distance(req.body.key, element.title) > 0.7){
element.probability = parseFloat(jw.distance(req.body.key, element.title).toFixed(2));
proposedNames.push(element);
}
})
resolve(proposedNames);
});
proposedNames.sort(function(a,b){return a.distance-b.distance});
results.proposedNames = proposedNames;
});
var promiseAll = Promise.all([promise1, promise2]);
promiseAll.then(() => {
res.send(results);
});
}
else {
res.status(400);
res.send("Can't search for an empty name");
}
})
Frontend code:
engraveLogoInputHandler() {
let results = {};
let loadedFromWeb = false, loadedFromClearbit = false, loadedFromDatabase = false;
this.setState({
engraveLogo: this.engravingLogo.value.length
});
// charsElthis.engravingInput.value
if (inputLogoTimer) {
clearTimeout(inputLogoTimer);
// inputLogoTimer = null;
}
if (this.engravingLogo.value !== ''){
// Wait to see if there is any new input coming soon, only render once finished to prevent lag
inputLogoTimer = setTimeout(() => {
request.post({url: NODEENDPOINT+'/logo', form: {key: this.engravingLogo.value}}, (err, res, body) => {
if (err){
console.log(err);
}
else {
if (res.body && res.statusCode !== 400){
results.database = JSON.parse(res.body);
loadedFromDatabase = true;
}
}
});
request(link+(this.engravingLogo.value), (err, res, body) => {
if (err) {
console.log(err);
}
else {
let jsonBody = JSON.parse(body);
if (jsonBody && !jsonBody.error){
let sources = [];
let data = JSON.parse(body);
for (let item of data) {
sources.push({
domain: item.domain,
image: item.logo+'?size=512&grayscale=true',
title: item.name
});
}
loadedFromClearbit = true;
results.clearbit = sources;
}
}
});
if (!loadedFromClearbit && !loadedFromDatabase){
request('https://www.googleapis.com/customsearch/v1?prettyPrint=false&fields=items(title,displayLink)&key='+GOOGLE_CSE_API_KEY+'&cx='+GOOGLE_CSE_ID+'&q='+encodeURIComponent(this.engravingLogo.value), { json: true }, (err, res, body) => {
if (err) {
console.error(err);
}
else {
if (body && body.items) {
let sources = [];
for (let s of body.items) {
sources.push({
domain: s.displayLink,
image: 'https://logo.clearbit.com/'+s.displayLink+'?size=512&greyscale=true',
title: s.title
});
}
loadedFromWeb = true;
results.googleSearches = sources;
} else {
console.error(body);
}
}
});
}
console.log("Results: ", results);
if (loadedFromClearbit || loadedFromWeb){
console.log("Propose the logo to be saved in a local database");
}
}, 500);}
}
So, in regarding to the backend code, is my implementation of promises actually correct there, and is it usefull? Could I use something similar for the front end and put the first two requests in Promise, and run the third request only if those two fail? (and failing means that they return empty results).
I thought I could use logic like this (see below) to catch if the promise failed, but that didn't work and I got an error saying I didn't catch the rejection:
var promise1 = new Promise((resolve, reject) => {
// ... some logic there
else {
reject();
}
});
var promise2 = promise1.catch(() => {
new Promise((resolve, reject) => {
// some logic for 2nd promise
});
});
Any answer is appreciated. As mentioned, I'm not very familiar with JavaScript, and this is the first asynchronous project I am working on, so I want to make sure I utilise and adapt the correct behaviour and methods.
Thanks
this is my second Node project. I am using Node, Express, MySQL.
What I am doing is, I have an array of names of users that have posted something, I then loop over those names and for each of them I do a connection.query to get their posts and I store those into an array(after that I do some other data manipulation to it, but that's not the important part)
The problem is: my code tries to do that data manipulation before it even receives the data from the connection.query!
I google-d around and it seems async await is the thing I need, problem is, I couldn't fit it in my code properly.
// namesOfPeopleImFollowing is the array with the names
namesOfPeopleImFollowing.forEach(function(ele){
connection.query(`SELECT * FROM user${ele}posts`, function(error,resultOfThis){
if(error){
console.log("Error found" + error)
} else {
allPostsWithUsername.push([{username:ele},resultOfThis])
}
})
})
console.log(JSON.stringify(allPostsWithUsername)) // This is EMPTY, it mustn't be empty.
So, how do I convert that into something which will work properly?
(Incase you need the entire code, here it is: https://pastebin.com/dDEJbPfP though I forgot to uncomment the code)
Thank you for your time.
There are many ways to solve this. A simple one would be to wrap your function inside a promise and resolve when the callback is complete.
const allPromises = [];
namesOfPeopleImFollowing.forEach((ele) => {
const myPromise = new Promise((resolve, reject) => {
connection.query(`SELECT * FROM user${ele}posts`, (error, resultOfThis) => {
if (error) {
reject(error);
console.log(`Error found${error}`);
} else {
resolve({ username: ele });
}
});
});
allPromises.push(myPromise);
});
Promise.all(allPromises).then((result) => {
// your code here
})
You can read more about promise.all here