I am having trouble figuring out the problem to an issue where when I write data (create, update, delete) then write a query to get the data after, the data that I receive back is the data prior to the write.
For example:
Let's say I have two functions createApple() and getAppleById. I have a utility function called getConnection() that gets a connection from a pool to be used for transactional queries. I have an endpoint that creates an apple and I get back to the insertId from mysql then I use that to get the apple but when I return it as a response, I get an empty object.
const createApple = async ({ type }) => {
const connection = await getConnection();
await connection.beginTransaction();
return await connection.query(`INSERT INTO apple (type) VALUES (?)`, [type]);
}
const getAppleById = async (appleId) => {
const connection = await getConnection();
return await connection.query(`SELECT * FROM apple WHERE id = ?`, [appleId]);
}
router.post(`/api/apples`, async (req, res) => {
const { insertId: createdAppleId } = await createApple({ ...req.body });
const apple = await getAppleById(createdAppleId);
res.status(201).send(apple); // <-- this returns {}
});
I noticed that if I add a console.log() before sending the data back, it does get back the data, for example:
router.post(`/api/apples`, async (req, res) => {
const { insertId: createdAppleId } = await createApple({ ...req.body });
const apple = await getAppleById(createdAppleId);
console.log(apple);
res.status(201).send(apple); // <-- this now returns the newly created apple
});
Any ideas on why this may be happening? Also, is this considered a good way of getting a newly created/updated entity or would it be better to make two separate calls:
First call to create/edit the entity (a POST or PATCH call)
Second call to get the entity (a GET call)
Any help is appreciated!
Thanks!
const createApple = async ({ type }) => {
const connection = await getConnection();
await connection.beginTransaction();
await connection.query(`INSERT INTO apple (type) VALUES (?)`, [type]);
await connection.commit();
}
I think error this function when you use transaction, you should commit or rollback transaction after finish query
This is best practice for me, I hope it useful for you
const createApple = async ({ type }) => {
const connection = await getConnection();
await connection.beginTransaction();
try{
await connection.query(`INSERT INTO apple (type) VALUES (?)`, [type]);
await connection.commit();
}catch{
await connection.rollback()
}
}
Related
I have the following get request where I call a bunch of data and pass it through to my EJS view.
router.get('/currentrentals', ensureAuthenticated, async (req, res) => {
const companies = await Company.getCompanies();
const barges = await Barge.getBarges();
const parishes = await Parish.getParishes();
const states = await State.getStates();
const pickupdropoff = await PickupDropoff.getPickupDropoff();
var notifications = await Notification.getNotifications();
JSON.stringify(barges);
JSON.stringify(companies);
JSON.stringify(parishes);
JSON.stringify(states);
JSON.stringify(pickupdropoff);
JSON.stringify(notifications);
var notifications = await notifications.sort((a, b) => b.id - a.id).slice(0,3);
res.render('currentrentals', {
name: req.user.name, companies: companies, barges: barges, parishes: parishes, states: states, pickupdropoff : pickupdropoff, notifications : notifications
});
}
);
Two questions:
I have multiple get requests that requires the same information. Is there a way to make this data available across the entirety of my site, so I don't have to rewrite this for each get path?
Is there a more succinct way to write the existing code I have? Perhaps looping through them or something of the sort? Simply for learning purposes.
The code currently works as-is.
Thanks!
If the data is constant, you can try this:
let data = null;
async function getData() {
if (!data) {
data = {
companies: await Company.getCompanies(),
barges: await Barge.getBarges();
parishes: await Parish.getParishes(),
states: await State.getStates(),
pickupdropoff: await PickupDropoff.getPickupDropoff(),
notifications: (await Notification.getNotifications()).sort((a, b) => b.id - a.id).slice(0,3)
};
}
return data;
}
router.get('/currentrentals', ensureAuthenticated, async (req, res) => {
res.render('currentrentals', { name: req.user.name, ...(await getData()) });
}
);
// other routes
This is my first time using Cloud Functions. I'm trying to make a simple call to access all the businesses stored in my Firestore collection, but when I try to log the results, I always get an empty array.
All things w/ Firebase/store are set up properly, collection name is listed properly, and have confirmed access to the database by logging db. Is there something obviously wrong with my code here? Thanks!
// The Cloud Functions for Firebase SDK to create Cloud Functions and setup triggers.
const functions = require('firebase-functions');
// The Firebase Admin SDK to access Firestore.
const admin = require('firebase-admin');
admin.initializeApp();
exports.updateBusinessData = functions.https.onRequest((request, response) => {
const db = admin.firestore()
const businessesReference = db.collection('businesses')
var businesses = []
const getBusinesses = async () => {
const businessData = await businessesReference.get()
businesses = [...businessData.docs.map(doc => ({...doc.data()}))]
for (let business in businesses) {
console.log(business)
}
response.send("Businesses Updated")
}
getBusinesses()
});
I tweaked the way you were processing the docs and I'm getting proper data from firestore.
exports.updateBusinessData = functions.https.onRequest((request, response) => {
const db = admin.firestore();
const businessesReference = db.collection("businesses");
const businesses = [];
const getBusinesses = async () => {
const businessData = await businessesReference.get();
businessData.forEach((item)=> {
businesses.push({
id: item.id,
...item.data(),
});
});
// used for of instead of in
for (const business of businesses) {
console.log(business);
}
response.send(businesses);
};
getBusinesses();
});
Your getBusinesses function is async: you then need to call it with await. Then, since you use await in the Cloud Function you need to declare it async.
The following should do the trick (untested):
exports.updateBusinessData = functions.https.onRequest(async (request, response) => {
try {
const db = admin.firestore()
const businessesReference = db.collection('businesses')
var businesses = []
const getBusinesses = async () => {
const businessData = await businessesReference.get()
businesses = businessData.docs.map(doc => doc.data());
for (let business in businesses) {
console.log(business)
}
}
await getBusinesses();
// Send back the response only when all the asynchronous
// work is done => This is why we use await above
response.send("Businesses Updated")
} catch (error) {
response.status(500).send(error);
}
});
You are probably going to update docs in the for (let business in businesses) loop to use await. Change it to a for … of loop instead as follows:
for (const business of businesses) {
await db.collection('businesses').doc(...business...).update(...);
}
Update following the comments
Can you try with this one and share what you get from the console.logs?
exports.updateBusinessData = functions.https.onRequest(async (request, response) => {
try {
console.log("Function started");
const db = admin.firestore()
const businessesReference = db.collection('businesses');
const businessData = await businessesReference.get();
console.log("Snapshot size = " + businessData.size);
const businesses = businessData.docs.map(doc => doc.data());
for (const business of businesses) {
console.log(business);
}
response.send("Businesses Updated")
} catch (error) {
console.log(error);
response.status(500).send(error);
}
});
Im guessing this problem is because I don't know how to use async await effectively. I still dont get it and I've been trying to understand for ages. sigh.
Anyway, heres my function:
app.post("/declineTrades", async (request, response) => {
//---------------------------------------------
const batch = db.batch();
const listingID = request.body.listingID;
const tradeOfferQuery = db
//---------------------------------------------
//Get trade offers that contain the item that just sold
//(therefore it cannot be traded anymore, I need to cancel all existing trade offers that contain the item because this item isn't available anymore)
//---------------------------------------------
.collection("tradeOffers")
.where("status", "==", "pending")
.where("itemIds", "array-contains", listingID);
//---------------------------------------------
//Function that gets all trade offers that contain the ID of the item.
async function getIdsToDecline() {
let tempArray = [];
tradeOfferQuery.get().then((querySnapshot) => {
querySnapshot.forEach((doc) => {
//For each trade offer found
let offerRef = db.collection("tradeOffers").doc(doc.id);
//Change the status to declined
batch.update(offerRef, { status: "declined" });
//Get the data from the trade offer because I want to send an email
//to the who just got their trade offer declined.
const offerGet = offerRef.get().then((offer) => {
const offerData = offer.data();
//Check the items that the receiving person had in this trade offer
const receiverItemIds = Array.from(
offerData.receiversItems
.reduce((set, { itemID }) => set.add(itemID), new Set())
.values()
);
//if the receiver item id's array includes this item that just sold, I know that
//I can get the sender ID (users can be sender or receiver, so i need to check which person is which)
if (receiverItemIds.includes(listingID)) {
tempArray.push(offerData.senderID);
}
});
});
});
//With the ID's now pushed, return the tempArray
return tempArray;
}
//---------------------------------------------
//Call the above function to get the ID's of people that got declined
//due to the item no longer being available
const peopleToDeclineArray = await getIdsToDecline();
//Update the trade offer objects to declined
const result = await batch.commit();
//END
response.status(201).send({
success: true,
result: result,
idArray: peopleToDeclineArray,
});
});
Im guessing that my return tempArray is in the wrong place? But I have tried putting it in other places and it still returns an empty array. Is my logic correct here? I need to run the forEach loop and add to the array before the batch.commit happens and before the response is sent.
TIA Guys!
As #jabaa pointed out in their comment, there are problems with an incorrectly chained Promise in your getIdsToDecline function.
Currently the function initializes an array called tempArray, starts executing the trade offer query and then returns the array (which is currently still empty) because the query hasn't finished yet.
While you could throw in await before tradeOfferQuery.get(), this won't solve your problem as it will only wait for the tradeOfferQuery to execute and the batch to be filled with entries, while still not waiting for any of the offerRef.get() calls to be completed to fill the tempArray.
To fix this, we need to make sure that all of the offerRef.get() calls finish first. To get all of these documents, you would use the following code to fetch each document, wait for all of them to complete and then pull out the snapshots:
const itemsToFetch = [ /* ... */ ];
const getAllItemsPromise = Promise.all(
itemsToFetch.map(item => item.get())
);
const fetchedItemSnapshots = await getAllItemsPromise;
For documents based on a query, you'd tweak this to be:
const querySnapshot = /* ... */;
const getSenderDocPromises = [];
querySnapshot.forEach((doc) => {
const senderID = doc.get("senderID");
const senderRef = db.collection("users").doc(senderID);
getSenderDocPromises.push(senderRef.get());
}
const getAllSenderDocPromise = Promise.all(getSenderDocPromises);
const fetchedSenderDataSnapshots = await getAllSenderDocPromise;
However neither of these approaches are necessary, as the document you are requesting using these offerRef.get() calls are already returned in your query so we don't even need to use get() here!
(doc) => {
let offerRef = db.collection("tradeOffers").doc(doc.id);
//Change the status to declined
batch.update(offerRef, { status: "declined" });
//Get the data from the trade offer because I want to send an email
//to the who just got their trade offer declined.
const offerGet = offerRef.get().then((offer) => {
const offerData = offer.data();
//Check the items that the receiving person had in this trade offer
const receiverItemIds = Array.from(
offerData.receiversItems
.reduce((set, { itemID }) => set.add(itemID), new Set())
.values()
);
//if the receiver item id's array includes this item that just sold, I know that
//I can get the sender ID (users can be sender or receiver, so i need to check which person is which)
if (receiverItemIds.includes(listingID)) {
tempArray.push(offerData.senderID);
}
});
}
could be replaced with just
(doc) => {
// Change the status to declined
batch.update(doc.ref, { status: "declined" });
// Fetch the IDs of items that the receiving person had in this trade offer
const receiverItemIds = Array.from(
doc.get("receiversItems") // <-- this is the efficient form of doc.data().receiversItems
.reduce((set, { itemID }) => set.add(itemID), new Set())
.values()
);
// If the received item IDs includes the listed item, add the
// sender's ID to the array
if (receiverItemIds.includes(listingID)) {
tempArray.push(doc.get("senderID"));
}
}
which could be simplified to just
(doc) => {
//Change the status to declined
batch.update(doc.ref, { status: "declined" });
// Check if any items that the receiving person had in this trade offer
// include the listing ID.
const receiversItemsHasListingID = doc.get("receiversItems")
.some(item => item.itemID === listingID);
// If the listing ID was found, add the sender's ID to the array
if (receiversItemsHasListingID) {
tempArray.push(doc.get("senderID"));
}
}
Based on this, getIdsToDecline actually queues declining the invalid trades and returns the IDs of those senders affected. Instead of using the batch and tradeOfferQuery objects that are outside of the function that make this even more unclear, you should roll them into the function and pull it out of the express handler. I'll also rename it to declineInvalidTradesAndReturnAffectedSenders.
async function declineInvalidTradesAndReturnAffectedSenders(listingID) {
const tradeOfferQuery = db
.collection("tradeOffers")
.where("status", "==", "pending")
.where("itemIds", "array-contains", listingID);
const batch = db.batch();
const affectedSenderIDs = [];
const querySnapshot = await tradeOfferQuery.get();
querySnapshot.forEach((offerDoc) => {
batch.update(offerDoc.ref, { status: "declined" });
const receiversItemsHasListingID = offerDoc.get("receiversItems")
.some(item => item.itemID === listingID);
if (receiversItemsHasListingID) {
affectedSenderIDs.push(offerDoc.get("senderID"));
}
}
await batch.commit(); // generally, the return value of this isn't useful
return affectedSenderIDs;
}
This then would change your route handler to:
app.post("/declineTrades", async (request, response) => {
const listingID = request.body.listingID;
const peopleToDeclineArray = await declineInvalidTradesAndReturnAffectedSenders(listingID);
response.status(201).send({
success: true,
result: result,
idArray: peopleToDeclineArray,
});
});
Then adding the appropriate error handling, swapping out the incorrect use of HTTP 201 Created for HTTP 200 OK, and using json() instead of send(); you now get:
app.post("/declineTrades", async (request, response) => {
try {
const listingID = request.body.listingID;
const affectedSenderIDs = await declineInvalidTradesAndReturnAffectedSenders(listingID);
response.status(200).json({
success: true,
idArray: affectedSenderIDs, // consider renaming to affectedSenderIDs
});
} catch (error) {
console.error(`Failed to decline invalid trades for listing ${listingID}`, error);
if (!response.headersSent) {
response.status(500).json({
success: false,
errorCode: error.code || "unknown"
});
} else {
response.end(); // forcefully end corrupt response
}
}
});
Note: Even after all these changes, you are still missing any form of authentication. Consider swapping the HTTPS Event Function out for a Callable Function where this is handled for you but requires using a Firebase Client SDK.
I'm trying to scrape information from a webpage behind a login wall for two users. As it stands, I've managed to get the code to do what I want for the first user i.e. go to webpage, login, gather the links associated with properties in a saved list, use that list to gather more details and log them to console.
The challenge I have now is getting the code to loop this round the second user without having to dupe the code. How would you suggest I go about it?
Secondly I need to make the array for each user, declared as uniquePropertyLinks in the below, accessible outside of the function userProcess.
How can I produce a new array for each user?
How can I access the array outside the function?
Here is the code:
const puppeteer = require('puppeteer');
//Code to locate text and enable it to be clicked
const escapeXpathString = str => {
const splitedQuotes = str.replace(/'/g, `', "'", '`);
return `concat('${splitedQuotes}', '')`;
};
const clickByText = async (page, text) => {
const escapedText = escapeXpathString(text);
const linkHandlers = await page.$x(`//a[contains(text(), ${escapedText})]`);
if (linkHandlers.length > 0) {
await linkHandlers[0].click();
} else {
throw new Error(`Link not found: ${text}`);
}
};
//User credentials
const userAEmail = 'abc#hotmail.com';
const userAPassword = '123';
const userBEmail = 'def#hotmail.com';
const userBPassword = '456';
//Logout
const LogOut = async (page) => {
await page.goto('https://www.website.com');
await clickByText(page, 'Log out');
await page.waitForNavigation({waitUntil: 'load'});
console.log('Signed out');
};
///////////////////////////
//SCRAPE PROCESS
async function userProcess() {
try {
const browser = await puppeteer.launch({ headless : false });
const page = await browser.newPage();
page.setUserAgent('BLAHBLAHBLAH');
//Go to Website saved list
await page.goto('https://www.website.com/shortlist.html', {waitUntil: 'networkidle2'});
console.log('Page loaded');
//User A log in
await page.type('input[name=email]', userAEmail, {delay: 10});
await page.type('input[name=password]', userAPassword, {delay: 10});
await page.click('.mrm-button',{delay: 10});
await page.waitForNavigation({waitUntil: 'load'})
console.log('Signed in');
//Wait for website saved list to load
const propertyList = await page.$$('.title');
console.log(propertyList.length);
//Collecting links from saved list and de-duping into an array
const propertyLinks = await page.evaluate(() => Array.from(document.querySelectorAll('.sc-jbKcbu'), e => e.href));
let uniquePropertyLinks = [...new Set(propertyLinks)];
console.log(uniquePropertyLinks);
//Sign out
LogOut(page);
} catch (err) {
console.log('Our error - ', err.message);
}
};
userProcess();
Let's see some of the things you might need to complete your task. I think it's better to take time and develop the skills yourself, but I can perhaps point out a few key things.
You use:
const userAEmail = 'abc#hotmail.com';
const userAPassword = '123';
const userBEmail = 'def#hotmail.com';
const userBPassword = '456';
but then you're talking about looping. With such a data structure, it will be difficult to loop these two users. I recommend putting it into an object like so:
const users = {
a: {
email: 'abc#hotmail.com',
password: '123',
},
b: {
email: 'def#hotmail.com',
password: '456',
},
};
then you can easily look with for example for .. in:
for (const user in users) {
console.log(users[user]);
}
or with .forEach():
Object.values(users).forEach(user => {
console.log(user);
});
need to make the array for each user, declared as uniquePropertyLinks in the below, accessible outside of the function userProcess.
Then declare the array outside of the funtion:
let uniquePropertyLinks = [];
async function userProcess() {
// you can access uniquePropertyLinks here
}
// and you can access uniquePropertyLinks here as well
How can I produce a new array for each user? How can I access the array outside the function?
Again, it'd be better to choose a differen data structure, let's day an object with keys that would represent each user and values would be arrays. It'd look like so:
let uniquePropertyLinks = {};
uniquePropertyLinks.a = [];
uniquePropertyLinks.b = [];
which looks like this:
{ a: [], b: [] }
so you can save whatever values for user a into uniquePropertyLinks.a array and whatever values you need into uniquePropertyLinks.b array:
uniquePropertyLinks.a.push('new_value_for_a_user');
similarly for user b.
Now you should have all the bits you need in order to go back to your code and make the necessary changes.
For those looking for the results of pavelsaman's advice below is the updated code:
const puppeteer = require('puppeteer');
//Object storing user credentials
let userAEmail = 'abc';
let userAPassword = '123';
let userBEmail = 'def';
let userBPassword = '456';
const users = {
userA: {
email: userAEmail,
password: userAPassword,
},
userB: {
email: userBEmail,
password: userBPassword,
},
};
//Object storing users saved lists as arrays
const usersPropertyLinks = {};
usersPropertyLinks.userA = [];
usersPropertyLinks.userB = [];
//Function to retrieve users saved list of properties
async function retrieveUserSavedList(users, usersPropertyLinks) {
try {
//Load broswer
const browser = await puppeteer.launch({ headless : true });
const page = await browser.newPage();
page.setUserAgent('BLAHHBLAHHBLAHH');
for (const user in users) {
//Go to saved list
await page.goto('https://www.website.co.uk/user/shortlist.html', {waitUntil: 'networkidle2'});
await page.waitForSelector('.mrm-button');
//User log in
await page.type('input[name=email]', users[user].email, {delay: 10});
await page.type('input[name=password]', users[user].password, {delay: 10});
await page.click('.mrm-button',{delay: 10});
await page.waitForNavigation({waitUntil: 'load'})
console.log('Success: ' + users[user].email + ' logged in');
//Collecting saved property links and de-duping into an array
const propertyLinks = await page.evaluate(() => Array.from(document.querySelectorAll('.sc-jbKcbu'), e => e.href));
//Add saved property links to an array for each user
if (users[user].email === userAEmail ) {
usersPropertyLinks.userA.push(...new Set(propertyLinks));
} else if (users[user].email === userBEmail ) {
usersPropertyLinks.userB.push(...new Set(propertyLinks));
} else {
console.log('problem saving links to user array');
};
//Sign out
await page.click('.sc-kAzzGY',{delay: 10});
await page.waitForNavigation({waitUntil: 'load'});
console.log('Success: ' + users[user].email + ' logged out');
};
browser.close();
} catch (err) {
console.log('Error retrieve user saved list - ', err.message);
}
};
//Run the code
retrieveUserSavedList(users, usersPropertyLinks);
I'm trying to send a direct message using slack web api to a user but I think my getSlackUser method which gets all the available users does not complete in time for when I call slackId;
the console.log(slackId) gives undefined meaning it doesn't complete my api call with bolt
how do I ensure getSlackUser method finishes (make it blocking) before it moves on to the rest?
const { WebClient } = require('#slack/web-api');
const { App } = require('#slack/bolt')
const rtm = new RTMClient(process.env.SLACK_OAUTH_TOKEN);
const web = new WebClient(process.env.SLACK_OAUTH_TOKEN);
const app = new App({
token: process.env.SLACK_OAUTH_TOKEN,
signingSecret: process.env.SLACK_SIGNING_SECRET
});
exports.handler = async (event) => {
const slackId = await getSlackUser('example_real_name').id;
console.log(slackId);
await sendSlackMessage(slackId, 'Bot message');
}
sendSlackMessage = async (channel, message) => {
await web.chat.postMessage({
channel: channel,
text: message,
as_user: true
});
}
getSlackUser = async(real_name) => {
const result = await app.client.users.list({
token: process.env.SLACK_OAUTH_TOKEN
});
console.log(result);
return result.members.find((user) => user.real_name == real_name);
}
The problem is precedence on this line:
const slackId = await getSlackUser('example_real_name').id;
Since member access has a higher precedence (evaluated before) than await, it is effectively the same as:
const slackId = await (getSlackUser('example_real_name').id);
getSlackUser returns a Promise object, then its id member is undefined. Await waits for the undefined, which is undefined.
To fix this, make sure that the await is evaluated before the .id:
const slackId = (await getSlackUser('example_real_name')).id;