Unable to log user state information to transcript - javascript

I am using TranscriptLoggerMiddleware and CosmosDB to log my chatbot transcripts. We are trying to capture the user state information (user name, account number, account type, etc) as top level attributes in the transcript so that specific customers can easily be queried in the DB (if that information is just in the individual timestamp attributes of the document, they can't be queried).
Ideally I would just add the user state when I'm building the file, but I can't figure any way to access it since the logger is defined in index.js and TranscriptLoggerMiddleware only provides the activity to my function, not the full context. If anyone has a way to get the user state data via TranscriptLoggerMiddleware, let me know, that would solve this issue. Here is the customLogger code. Note that due to the function receiving both the user query and bot response, I couldn't get retrieving and resaving the transcript to work, so I'm overwriting the transcript from a local log object. Not trying to come up with a new approach here but if one would solve the overall issue I'd like to hear it.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.
const { CosmosDbPartitionedStorage } = require('botbuilder-azure');
const path = require('path');
/**
* CustomLogger, takes in an activity and saves it for the duration of the conversation, writing to an emulator compatible transcript file in the transcriptsPath folder.
*/
class CustomLogger {
/**
* Log an activity to the log file.
* #param activity Activity being logged.
*/
// Set up Cosmos Storage
constructor(appInsightsClient) {
this.transcriptStorage = new CosmosDbPartitionedStorage({
cosmosDbEndpoint: process.env.COSMOS_SERVICE_ENDPOINT,
authKey: process.env.COSMOS_AUTH_KEY,
databaseId: process.env.DATABASE,
containerId: 'bot-transcripts'
});
this.conversationLogger = {};
this.appInsightsClient = appInsightsClient;
this.msDelay = 250;
}
async logActivity(activity) {
if (!activity) {
throw new Error('Activity is required.');
}
// Log only if this is type message
if (activity.type === 'message') {
if (activity.attachments) {
try {
var logTextDb = `${activity.from.name}: ${activity.attachments[0].content.text}`;
} catch (err) {
var logTextDb = `${activity.from.name}: ${activity.text}`;
}
} else {
var logTextDb = `${activity.from.name}: ${activity.text}`;
}
if (activity.conversation) {
var id = activity.conversation.id;
if (id.indexOf('|') !== -1) {
id = activity.conversation.id.replace(/\|.*/, '');
}
// Get today's date for datestamp
var currentDate = new Date();
var day = currentDate.getDate();
var month = currentDate.getMonth()+1;
var year = currentDate.getFullYear();
var datestamp = year + '-' + month + '-' + day;
var fileName = `${datestamp}_${id}`;
var timestamp = Math.floor(Date.now()/1);
// CosmosDB logging (JK)
if (!(fileName in this.conversationLogger)) {
this.conversationLogger[fileName] = {};
this.conversationLogger[fileName]['userData'] = {};
this.conversationLogger[fileName]['botName'] = process.env.BOTNAME;
}
this.conversationLogger[fileName][timestamp] = logTextDb;
let updateObj = {
[fileName]:{
...this.conversationLogger[fileName]
}
}
// Add delay to ensure messages logged sequentially
await this.wait(this.msDelay);
try {
let result = await this.transcriptStorage.write(updateObj);
} catch(err) {
console.log(err);
this.appInsightsClient.trackTrace({message: `Logger Error ${err.code} - ${path.basename(__filename)}`,severity: 3,properties: {'botName': process.env.BOTNAME, 'error':err.body}});
}
}
}
}
async wait(milliseconds) {
var start = new Date().getTime();
for (var i = 0; i < 1e7; i++) {
if ((new Date().getTime() - start) > milliseconds) {
break;
}
}
}
}
exports.CustomLogger = CustomLogger;
Not being able to get user state in this function, I decided to try a few other approaches. The most promising was creating a separate "updateTranscript" function to grab the transcript, add user state, and save it back. But I think it was catching it only on user request and getting overidden again by local object on bot response. I added a delay to try to combat this, but it still didn't work. On my very first prompt of providing customer number user state data is getting stored on transcript, but at the next activity it is gone and never comes back (even though I can see it is supposedly getting written to DB). Here is that update function.
const { CosmosDbStorage } = require('botbuilder-azure');
var updateTranscript = async (context, userData, appInsightsClient) => {
const transcriptStorage = new CosmosDbStorage({
serviceEndpoint: process.env.COSMOS_SERVICE_ENDPOINT,
authKey: process.env.COSMOS_AUTH_KEY,
databaseId: process.env.DATABASE,
collectionId: 'bot-transcripts',
partitionKey: process.env.BOTNAME
});
var id = context.activity.conversation.id;
if (id.indexOf('|') !== -1) {
id = context.activity.conversation.id.replace(/\|.*/, '');
}
// Get today's date for datestamp
var currentDate = new Date();
var day = currentDate.getDate();
var month = currentDate.getMonth()+1;
var year = currentDate.getFullYear();
var datestamp = year + '-' + month + '-' + day;
var filename = `${datestamp}_${id}`;
var msDelay = 500;
await new Promise(resolve => setTimeout(resolve, msDelay));
var transcript = await transcriptStorage.read([filename]);
transcript[filename]['userData'] = userData
try {
await transcriptStorage.write(transcript);
console.log('User data added to transcript');
} catch(err) {
console.log(err);
appInsightsClient.trackTrace({message: `Log Updater Error ${err.code} - ${path.basename(__filename)}`,severity: 3,properties: {'botName': process.env.BOTNAME, 'error':err.body}});
}
return;
}
module.exports.updateTranscript = updateTranscript
I realize this approach is a bit of a cluster but I've been unable to find anything better. I know the Microsoft COVID-19 bot has a really nice transcript retrieval function, but I haven't been able to get any input from them on how that was accomplished. That aside, I'm quite happy to continue with this implementation if someone can help me figure out how to get that user state into the transcript without being overwritten or running into concurrency issues.
As to why I can't query an account number even via substring() function, here's an example of the documents data object. I have no idea which string to check for a substring, in this case 122809. I don't know what that timestamp could be. If this is stored at the top level (e.g. userData/accountNumber) I know exactly where to look for the value. For further context, I've displayed what I see after the first prompt for account number, where userData is populated. But it gets overidden on subsequent writes and I can't seem to get it back even with a delay in my updateTranscript function.
"document": {
"userData": {},
"botName": "AveryCreek_OEM_CSC_Bot_QA",
"1594745997562": "AveryCreek_OEM_CSC_Bot_QA: Hi! I'm the OEM CSC Support Bot! Before we get started, can you please provide me with your 6-digit Vista number? If you don't have one, just type \"Skip\".",
"1594746003973": "You: 122809",
"1594746004241": "AveryCreek_OEM_CSC_Bot_QA: Thank you. What can I help you with today? \r\nYou can say **Menu** for a list of common commands, **Help** for chatbot tips, or choose one of the frequent actions below. \r\n \r\n I'm still being tested, so please use our [Feedback Form](https://forms.office.com/Pages/ResponsePage.aspx?id=lVxS1ga5GkO5Jum1G6Q8xHnUJxcBMMdAqVUeyOmrhgBUNFI3VEhMU1laV1YwMUdFTkhYVzcwWk9DMiQlQCN0PWcu) to let us know how well I'm doing and how I can be improved!",
"1594746011384": "You: what is my account number?",
"1594746011652": "AveryCreek_OEM_CSC_Bot_QA: Here is the informaiton I have stored: \n \n**Account Number:** 122809 \n\n I will forget everything except your account number after the end of this conversation.",
"1594746011920": "AveryCreek_OEM_CSC_Bot_QA: I can clear your information if you don't want me to store it or if you want to reneter it. Would you like me to clear your information now?",
"1594746016034": "You: no",
"1594746016301": "AveryCreek_OEM_CSC_Bot_QA: OK, I won't clear your information. You can ask again at any time."
},
"document": {
"userData": {
"accountNumber": "122809"
},
"botName": "AveryCreek_OEM_CSC_Bot_QA",
"1594746019952": "AveryCreek_OEM_CSC_Bot_QA: Hi! I'm the OEM CSC Support Bot! What can I help you with today? \r\nYou can say **Menu** for a list of common commands, **Help** for chatbot tips, or choose one of the frequent actions below. \r\n \r\n I'm still being tested, so please use our [Feedback Form](https://forms.office.com/Pages/ResponsePage.aspx?id=lVxS1ga5GkO5Jum1G6Q8xHnUJxcBMMdAqVUeyOmrhgBUNFI3VEhMU1laV1YwMUdFTkhYVzcwWk9DMiQlQCN0PWcu) to let us know how well I'm doing and how I can be improved!"
},

You had said you were encountering concurrency issues even though JavaScript is single-threaded. As strange as that sounds, I think you're right on some level. TranscriptLoggerMiddleware does have its own buffer that it uses to store activities throughout the turn and then it tries to log all of them all at once. It could easily have provided a way to get that whole buffer in your own logger function, but instead it just loops through the buffer so that you still only get to log them each individually. Also, it allows logActivity to return a promise but it never awaits it, so each activity will get logged "simultaneously" (it's not really simultaneous but the code will likely jump between function calls before waiting for them to complete). This is a problem for any operation that isn't atomic, because you'll be modifying state without knowing about its latest modifications.
while (transcript.length > 0) {
try {
const activity: Activity = transcript.shift();
// If the implementation of this.logger.logActivity() is asynchronous, we don't
// await it as to not block processing of activities.
// Because TranscriptLogger.logActivity() returns void or Promise<void>, we capture
// the result and see if it is a Promise.
const logActivityResult = this.logger.logActivity(activity);
// If this.logger.logActivity() returns a Promise, a catch is added in case there
// is no innate error handling in the method. This catch prevents
// UnhandledPromiseRejectionWarnings from being thrown and prints the error to the
// console.
if (logActivityResult instanceof Promise) {
logActivityResult.catch(err => {
this.transcriptLoggerErrorHandler(err);
});
}
} catch (err) {
this.transcriptLoggerErrorHandler(err);
}
}
All in all, I don't think transcript logger middleware is the way to go here. While it may purport to serve your purposes, there are just too many problems with it. I would either write my own middleware or just put the middleware code directly in my bot logic like this:
async onTurn(turnContext) {
const activity = turnContext.activity;
await this.logActivity(turnContext, activity);
turnContext.onSendActivities(async (ctx, activities, next) => {
for (const activity of activities) {
await this.logActivity(ctx, activity);
}
return await next();
});
// Bot code here
// Save state changes
await this.userState.saveChanges(turnContext);
}
async logActivity(turnContext, activity) {
var transcript = await this.transcriptProperty.get(turnContext, []);
transcript.push(activity);
await this.transcriptProperty.set(turnContext, transcript);
console.log('Activities saved: ' + transcript.length);
}
Since your transcript would be stored in your user state, that user state would also have the account number you need and hopefully you'd be able to query for it.

Kyle's answer did help me solve the issue, and I think that will be the most reusable piece for anyone experiencing similar issues. The key takeaway is that, if you're using nodejs, you should not be using TranscriptLoggerMiddleware and instead use Kyle's function in your onTurn handler (repeated here for reference):
// Function provided by Kyle Delaney
async onTurn(turnContext) {
const activity = turnContext.activity;
await this.logActivity(turnContext, activity);
turnContext.onSendActivities(async (ctx, activities, next) => {
for (const activity of activities) {
await this.logActivity(ctx, activity);
}
return await next();
});
// Bot code here
// Save state changes
await this.userState.saveChanges(turnContext);
}
You need to note, though, that his logActivity function is just storing the raw activities to the user state using a custom transcriptProperty. As of yet I haven't found a good method to give business/admin users access to this data in a way that is easily readable and searchable, nor construct some sort of file out output to send to a customer requesting a transcript of their conversation. As such, I continued using my CustomLogger instead. Here is how I accomplished that.
First, you must create the transcriptLogger in the constructor. If you create it inside your turn handler, you will lose the cache/buffer and it will only have the latest activity instead of the full history. May be common sense but this tripped me up briefly. I do this in the constructor via this.transcriptLogger = new CustomerLogger(appInsightsClient);. I also modified my logActivity function to accept the userData (my state object) as a second, optional parameter. I have successfully been able to use that userData object to add the required customer information to the bot transcript. To modify Kyle's function above you just need to replace this.logActivity with your function call, in my case this.transcriptLogger.logActivity(context, userData);.
While there are still some other issues with this approach, it does solve the title question of how to get user state data into the transcript.

Related

Collect the full list of buttons to follow without having to scroll the page (DevTools Google Chrome)

DevTools Google Chrome:
On this site (https://booyah.live/users/41874362/followers), to load the complete list of followers it is necessary to keep scrolling down the page to reload more profiles, but there comes a time when the page weighs so much that the browser crashes and it ends up needing to be closed.
Is there any way to be able to collect the follow buttons without this happening?
The current script I use is:
setInterval(function(){
document.getElementById("layout-content").scrollTo(0, 50000000000000000000000000000000000000);
document.querySelectorAll('.components-button.components-button-size-mini.components-button-type-orange.desktop.components-button-inline').forEach(btn => btn.click());
}, 10)
I use setInterval to create a loop of:
1 - Scrolling the page
2 - Loading more profiles
3 - Clicking the follow buttons
My need:
For the study I'm doing for learning, the idea is that my profile follows all profiles followers of a single most famous profile in order to analyze how many people follow back on this social media.
Additional:
In this answer provided by Leftium, it is possible to follow only one profile:
https://stackoverflow.com/a/67882688/11462274
In this answer given by KCGD, it is possible to collect the entire list of followers but during this collection the profiles are not followed, it is possible to create a list and save the data, but not follow the profiles:
https://stackoverflow.com/a/67865968/11462274
I tried to contact them both, but they haven't returned yet. It was a good way but I couldn't combine the two answers so I can follow all the profiles, I thought about the possibility according to which I would collect the profiles of the KCGD response, I would follow the profiles too, but not only the first one but also the answer of the Leftium.
Would it be possible to take advantage of the loop created by the response from KCGD and from each response, already follow all profiles instead of just the first one as in Leftium's response?
I tried to create but was unsuccessful.
The browser crashes because too much memory is used. As you scroll down the page, the HTML DOM tree is extended and more avatar images are downloaded. These HTML and image resources are not necessary for your goal.
It is possible to avoid crashing by calling the (internal) Booyah API directly. This will be much faster and consume less resources since only the text is transferred. There are two API endpoints of interest:
GET /api/v3/users/[USERID]/followers?cursor=0&count=100
Gets list of followers following a certain user:
[USERID] is the ID of the user being studied (WEEDZAO's id).
cursor is where in the list of followers to start listing. When the page first loads, this is 0. As you scroll down, the following API calls increment this (101, 201, 301...)
count is how many results to return.
Since this is a GET call, you can open this URL in your browser.
POST /api/v3/users/[USERID]/followings
Follows a user (same as clicking their 'Follow' button).
Here [USERID] is ID of the user whose follower list will be updated (your own ID).
A payload must be sent that looks like this: {followee_uid: ID, source: 43}. I'm not sure what source is.
Also a CSRF header must be included.
Because this is a POST type call, it is not possible to open this type of URL directly in your browser.
DELETE /api/v3/users/[USERID]/followings
There is also an API to unfollow a user. (Just for reference).
If you call these API's from outside the browser, you probably need to send session cookies.
This script will list WEEDZAO's first 10 followers, then follow the first one from the list:
You must replace USERID and CSRF_TOKEN with your own values.
You can copy/paste this code into the browser dev console.
Alternatively, you can use this code from a web scraping framework like Puppeteer.
// Find these values in dev console "Network" tab:
var CSRF_TOKEN, USERID, USERID_TARGET, main;
USERID_TARGET = '41874362';
USERID = '12345678';
CSRF_TOKEN = 'MTYy...p8wg';
main = async function() {
var body, followers, json, options, payload, response, url;
// Get list of first 10 followers
console.log(url = `/api/v3/users/${USERID_TARGET}/followers?cursor=0&count=10`);
response = (await fetch(url));
json = (await response.json());
followers = json.follower_list;
console.table(followers);
// Follow first member from list above
console.log(url = `/api/v3/users/${USERID}/followings`);
payload = JSON.stringify({
followee_uid: followers[0].uid,
source: 43
});
response = (await fetch(url, options = {
method: 'POST',
body: payload,
headers: {
'X-CSRF-Token': CSRF_TOKEN
}
}));
body = (await response.text());
return console.log(body);
};
main();
It crashes because the interval is too fast
setInterval(function(){}, 10)
you are trying to call a scroll and click function every 10 milliseconds (that's 100 function call every 1 second). Which also interferes with the server as they fetch new users while scrolling.
Your script could work if you will adjust the interval to atleast 1000 milliseconds (1 second). Of course, it may take a while, but it will work. You should also expect that the page may become laggy specially when the page already loaded tons of users because Virtual Scrolling is not implemented in this page.
Even with slowing down the rate of the scrolling it still really bogs down the browser, the solution to this may be in the API the page contacts. To get the user's followers it contacts the site's V3 API
https://booyah.live/api/v3/users/41874362/followers?cursor=[LAST USER IN API RETURN]&count=100
to get all the users that would show up in the page. I wrote a script that can contact the api over and over again to get all the follower data, just run it in the page's console and use print() when you want to export the data
and copy/paste it into a .json file
//WARNING: THIS SCRIPT USES RECURSION, i have no clue how long the followers list goes so use at your own risk
var followers = []; //data collected from api
function getFollowers(cursor){
httpGet(`https://booyah.live/api/v3/users/41874362/followers?cursor=${cursor}&count=100`, function (data) { //returns data from API for given cursor (user at the end of last follower chunk)
console.log("got cursor: "+cursor);
var _followChunk = JSON.parse(String(data));
console.log(_followChunk)
followers.push(_followChunk.follower_list); //saves followers from chunk
var last_user = _followChunk.follower_list[_followChunk.follower_list.length - 1]; //gets last user of chunk (cursor for the next chunk)
setTimeout(function(){ //1 second timeout so that the API doesnt return "too many requests", not nessicary but you should probably leave this on
getFollowers(last_user.uid); //get next chunk
},1000)
})
}
var print = function(){console.log(JSON.stringify(followers))};
getFollowers(0); //get initial set of followers (cursor 0)
function httpGet(theUrl, callback) {
var xmlHttp = new XMLHttpRequest();
xmlHttp.open("GET", theUrl, false); // false for synchronous request
xmlHttp.setRequestHeader("Cache-Control", "no-store");
xmlHttp.send(null);
callback(xmlHttp.responseText);
};
if you really only need the button elements then the only way is to scroll all the way down for each time it loads new followers, as the page creates the elements as you scroll down
This is a fully working solution that I have tested in my own Chrome browser with a fresh account, successfully following all the follower accounts of the account you are targeting.
UPDATE (2021-06-18)
I've updated my solution to a drastically improved and faster function, rewritten with async/await. This new function reduces the estimated runtime from ~45min to ~10min. 10min is still a long while, but that's to be expected considering the large number of followers the user you are targeting has.
After a few iterations, the latest function not only improves speed, performance, and error reporting, but it also extends what is possible with the function. I provide several example below my solutions of how to use the function completely.
For the sake of de-cluttering my answer, I am removing my older function from this solution altogether, but you can still reference it in my solution's edit history if you like.
TL;DR
Here is the final, fastest, working solution. Make sure to replace PUT_YOUR_CSRF_TOKEN_HERE with your own CSRF token value. Detailed instructions on how to find your CSRF token are below.
You must run this in your console on the Booyah website in order to avoid CORS issues.
const csrf = 'PUT_YOUR_CSRF_TOKEN_HERE';
async function booyahGetAccounts(uid, type = 'followers', follow = 1) {
if (typeof uid !== 'undefined' && !isNaN(uid)) {
const loggedInUserID = window.localStorage?.loggedUID;
if (uid === 0) uid = loggedInUserID;
const unfollow = follow === -1;
if (unfollow) follow = 1;
if (loggedInUserID) {
if (csrf) {
async function getUserData(uid) {
const response = await fetch(`https://booyah.live/api/v3/users/${uid}`),
data = await response.json();
return data.user;
}
const loggedInUserData = await getUserData(loggedInUserID),
targetUserData = await getUserData(uid),
followUser = uid => fetch(`https://booyah.live/api/v3/users/${loggedInUserID}/followings`, { method: (unfollow ? 'DELETE' : 'POST'), headers: { 'X-CSRF-Token': csrf }, body: JSON.stringify({ followee_uid: uid, source: 43 }) }),
logSep = (data = '', usePad = 0) => typeof data === 'string' && usePad ? console.log((data ? data + ' ' : '').padEnd(50, '━')) : console.log('━'.repeat(50),data,'━'.repeat(50));
async function getList(uid, type, follow) {
const isLoggedInUser = uid === loggedInUserID;
if (isLoggedInUser && follow && !unfollow && type === 'followings') {
follow = 0;
console.warn('You alredy follow your followings. `follow` mode switched to `false`. Followings will be retrieved instead of followed.');
}
const userData = await getUserData(uid),
totalCount = userData[type.slice(0,-1)+'_count'] || 0,
totalCountStrLength = totalCount.toString().length;
if (totalCount) {
let userIDsLength = 0;
const userIDs = [],
nickname = userData.nickname,
nicknameStr = `${nickname ? ` of ${nickname}'s ${type}` : ''}`,
alreadyFollowedStr = uid => `User ID ${uid} already followed by ${loggedInUserData.nickname} (Account #${loggedInUserID})`;
async function followerFetch(cursor = 0) {
const fetched = [];
await fetch(`https://booyah.live/api/v3/users/${uid}/${type}?cursor=${cursor}&count=100`).then(res => res.json()).then(data => {
const list = data[type.slice(0,-1)+'_list'];
if (list?.length) fetched.push(...list.map(e => e.uid));
if (fetched.length) {
userIDs.push(...fetched);
userIDsLength += fetched.length;
if (follow) followUser(uid);
console.log(`${userIDsLength.toString().padStart(totalCountStrLength)} (${(userIDsLength / totalCount * 100).toFixed(4)}%)${nicknameStr} ${follow ? 'followed' : 'retrieved'}`);
if (fetched.length === 100) {
followerFetch(data.cursor);
} else {
console.log(`END REACHED. ${userIDsLength} accounts ${follow ? 'followed' : 'retrieved'}.`);
if (!follow) logSep(targetList);
}
}
});
}
await followerFetch();
return userIDs;
} else {
console.log(`This account has no ${type}.`);
}
}
logSep(`${follow ? 'Following' : 'Retrieving'} ${targetUserData.nickname}'s ${type}`, 1);
const targetList = await getList(uid, type, follow);
} else {
console.error('Missing CSRF token. Retrieve your CSRF token from the Network tab in your inspector by clicking into the Network tab item named "bug-report-claims" and then scrolling down in the associated details window to where you see "x-csrf-token". Copy its value and store it into a variable named "csrf" which this function will reference when you execute it.');
}
} else {
console.error('You do not appear to be logged in. Please log in and try again.');
}
} else {
console.error('UID not passed. Pass the UID of the profile you are targeting to this function.');
}
}
booyahGetAccounts(41874362);
Detailed explanation of the process
As the function runs, it logs the progress to the console, both how many users have been followed so far, and how much progress has been made percentage-wise, based on the total number of followers the profile you are targeting has.
Retrieving your CSRF token
The only manual portion of this process is retrieving your CSRF token. This is rather simple though. Once you log into Booyah, navigate to the Network tab of your Chrome console and click on the item named bug-report-claims, then scroll all the way down the details window which appears on the right. There should see x-csrf-token. Store this value as a string variable in your console as csrf, which my function will reference when it runs. This is necessary in order to use the POST method to follow users.
Here is what it will look like:
The solution
The function will loop through all users the account you are targeting follows in batches of 100 (the max amount allowed per GET request) and follow them all. When the end of each batch is met, the next batch is automatically triggered recursively.
🚀 Version 3 (Fastest and most flexible, using async/await and fetch())
My previous two solution versions (🐇 …🐢) can be referenced in this answer's edit history.
Make sure to replace PUT_YOUR_CSRF_TOKEN_HERE with your own CSRF token value. Detailed instructions on how to find your CSRF token are below.
You must run this in your console on the Booyah website in order to avoid CORS issues.
const csrf = 'PUT_YOUR_CSRF_TOKEN_HERE';
async function booyahGetAccounts(uid, type = 'followers', follow = 1) {
if (typeof uid !== 'undefined' && !isNaN(uid)) {
const loggedInUserID = window.localStorage?.loggedUID;
if (uid === 0) uid = loggedInUserID;
const unfollow = follow === -1;
if (unfollow) follow = 1;
if (loggedInUserID) {
if (csrf) {
async function getUserData(uid) {
const response = await fetch(`https://booyah.live/api/v3/users/${uid}`),
data = await response.json();
return data.user;
}
const loggedInUserData = await getUserData(loggedInUserID),
targetUserData = await getUserData(uid),
followUser = uid => fetch(`https://booyah.live/api/v3/users/${loggedInUserID}/followings`, { method: (unfollow ? 'DELETE' : 'POST'), headers: { 'X-CSRF-Token': csrf }, body: JSON.stringify({ followee_uid: uid, source: 43 }) }),
logSep = (data = '', usePad = 0) => typeof data === 'string' && usePad ? console.log((data ? data + ' ' : '').padEnd(50, '━')) : console.log('━'.repeat(50),data,'━'.repeat(50));
async function getList(uid, type, follow) {
const isLoggedInUser = uid === loggedInUserID;
if (isLoggedInUser && follow && !unfollow && type === 'followings') {
follow = 0;
console.warn('You alredy follow your followings. `follow` mode switched to `false`. Followings will be retrieved instead of followed.');
}
const userData = await getUserData(uid),
totalCount = userData[type.slice(0,-1)+'_count'] || 0,
totalCountStrLength = totalCount.toString().length;
if (totalCount) {
let userIDsLength = 0;
const userIDs = [],
nickname = userData.nickname,
nicknameStr = `${nickname ? ` of ${nickname}'s ${type}` : ''}`,
alreadyFollowedStr = uid => `User ID ${uid} already followed by ${loggedInUserData.nickname} (Account #${loggedInUserID})`;
async function followerFetch(cursor = 0) {
const fetched = [];
await fetch(`https://booyah.live/api/v3/users/${uid}/${type}?cursor=${cursor}&count=100`).then(res => res.json()).then(data => {
const list = data[type.slice(0,-1)+'_list'];
if (list?.length) fetched.push(...list.map(e => e.uid));
if (fetched.length) {
userIDs.push(...fetched);
userIDsLength += fetched.length;
if (follow) followUser(uid);
console.log(`${userIDsLength.toString().padStart(totalCountStrLength)} (${(userIDsLength / totalCount * 100).toFixed(4)}%)${nicknameStr} ${follow ? 'followed' : 'retrieved'}`);
if (fetched.length === 100) {
followerFetch(data.cursor);
} else {
console.log(`END REACHED. ${userIDsLength} accounts ${follow ? 'followed' : 'retrieved'}.`);
if (!follow) logSep(targetList);
}
}
});
}
await followerFetch();
return userIDs;
} else {
console.log(`This account has no ${type}.`);
}
}
logSep(`${follow ? 'Following' : 'Retrieving'} ${targetUserData.nickname}'s ${type}`, 1);
const targetList = await getList(uid, type, follow);
} else {
console.error('Missing CSRF token. Retrieve your CSRF token from the Network tab in your inspector by clicking into the Network tab item named "bug-report-claims" and then scrolling down in the associated details window to where you see "x-csrf-token". Copy its value and store it into a variable named "csrf" which this function will reference when you execute it.');
}
} else {
console.error('You do not appear to be logged in. Please log in and try again.');
}
} else {
console.error('UID not passed. Pass the UID of the profile you are targeting to this function.');
}
}
Usage
To run the function (for either of the above solutions), just call the function name with the desired User ID name as an argument, in your example case, 41874362. The function call would look like this:
booyahGetAccounts(41874362);
The function is quite flexible in its abilities though. booyahGetAccounts() accepts three parameters, but only the first is required.
booyahGetAccounts(
uid, // required, no default
type = 'followers', // optional, must be 'followers' or 'followings' -> default: 'followers'
follow = 1 // optional, must be 0, 1, or -1, -> default: 1 (boolean true)
)
The second parameter, type, allows you to choose whether you would like to process the targeted user's followers or followings (the users which that user follows).
The third parameter allows you to choose whether you would like to follow/unfollow the returned users or only retrieve their User IDs. This defaults to 1 (boolean true) which will follow the users returned, but if you only want to test the function and not actually follow the returned users, set this to a falsy value such as 0 or false. Using -1 will unfollow the users returned.
This function intelligently retrieves your own User ID for you from the window.localStorage object, so you don't need to retrieve that yourself. If you would like to process your own followers or followings, simply pass 0 as the main uid parameter value, and the function will default the uid to your own User ID.
Because you can't re-follow users you already follow, if you try to follow your followings, the function will produce the warning You already follow your followings. 'follow' mode switched to 'false'. Followings will be retrieved instead of followed. and instead return them as if you had set the follow parameter to false.
However, it can be very useful to process your own list. For example, if you want to follow all of your own followers back, you could do so like this:
booyahGetAccounts(0); // `type` and `follow` parameters already default to the correct values here
On the other hand, if you were strategically using a follow/unfollow technique in order to increase your number of followers and needed to unfollow all of your followers, you could do so like this:
booyahGetAccounts(0, 'followers', -1);
By setting the follow parameter value to -1, you instruct the function to run its followUser function on all returned User IDs using the DELETE method instead of the POST method, thereby unfollowing those users returned instead of following them.
Desired outcome
Function call
Follow all your own followers
booyahGetAccounts(0, 'followers');
Unfollow all your own followers
booyahGetAccounts(0, 'followers', -1);
Unfollow all your own followings
booyahGetAccounts(0, 'followings', -1);
Follow users that follow User ID #12345
booyahGetAccounts(12345, 'followers');
Follow users followed by User ID #12345
booyahGetAccounts(12345, 'followings');
Retrieve User IDs of accounts following User ID #12345
booyahGetAccounts(12345, 'followers', 0);
Retrieve User IDs of accounts followed by User ID #12345
booyahGetAccounts(12345, 'followings', 0);
Other notes
To improve the performance of this function, as it's very heavy, I've replaced all calls to userIDs.length with a dedicated userIDsLength variable which I add to using += with each iteration rather than calling length each time. Similarly, I store the length of the stringified followerCount in the variable followerCountStrLength rather than calling followerCount.toString().length with each iteration. Because this is a rather heavy function, it is possible for your browser window to crash. However, it should eventually complete.
If the page appears to crash by flickering and auto-closing the console, FIRST try to re-open the console without refreshing the page at all. In my case, the inspector occasionally closed on its own, likely due to the exhaustion from the function, but when I opened the inspector's console again, the function was still running.

Google Cloud function and reading data from database

I am trying to set up (Facebook) Webhook. My databse structure is like this:
{ "users": [
"1" : {"FacebookID" : "1234567"},
"2" : {"FacebookID" : "9876554"}
]}
I reduced the code for the webhook to the minimum to show where I have my problems. I go through all entries from the request and check what uid has the change. Before I want to change my data, I would like to see if the user with the id is available in my database. Facebook sends me the FacebookID of course, therefore I need to check, if there is an user account with that specific facebook id.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
exports.webhooks = functions.https.onRequest((req, res) => {
var i = 0;
for (i = 0; i < req.body.entry.length; i++)
{
//strCurrentUser = req.body.entry[i].uid;
strCurrentUser = "2357295025722342"; //For debug
if (checkIfUserAvailable(strCurrentUser))
{
console.log("User is available " + strCurrentUser);
}
else
{
console.log("User is not available " + strCurrentUser);
}
}
return res.send(200);
});
function checkIfUserAvailable(userFacebookID)
{
const rootRef = admin.database().ref();
return rootRef.child('users').orderByChild('FacebookID').equalTo(userFacebookID).once('value').then( snap =>
{
console.log("snap is : " + (snap.val() !== null));
return (snap.val() !== null);
});
}
Now the problem is, that the checkIfUserAvailable always returns true and does not wait till the read is done to return the correct boolean. How can I make sure, that it waits with the return until the read has been finished to get the entire code working?
What I get as result ist:
11:34:58.057 PM / webhooks / snap is : false
11:34:31.665 PM / webhooks / Function execution took 1880 ms, finished with status code: 200
11:34:31.569 PM / webhooks / User is available 2357295025722342
11:34:29.786 PM / webhooks / Function execution started
As you can see the res.send(200) is fired earlier and the snap is: false will be logged later. (I am also not sure why it takes 20 seconds to read data from a almost empty database)
Thanks for the help!
You're not using promises correctly. The call to once() is asynchronous and returns immediately before the query is finished. This means the response is sent before the query is done, and Cloud Functions shuts down that async work. Instead, you should be sending that response only after the query is complete.
You might want to watch my videos on using promises in Cloud Functions in order to learn how best to deal with situations like this. Also be sure you understand the documentation.

MongoDB - Mongoose - TypeError: save is not a function

I am attempting to perform an update to a MongoDB document (using mongoose) by first using .findById to get the document, then updating the fields in that document with new values. I am still a bit new to this so I used a tutorial to figure out how to get it working, then I have been updating my code for my needs. Here is the tutorial: MEAN App Tutorial with Angular 4. The original code had a schema defined, but my requirement is for a generic MongoDB interface that will simply take whatever payload is sent to it and send it along to MongoDB. The original tutorial had something like this:
exports.updateTodo = async function(todo){
var id = todo.id
try{
//Find the old Todo Object by the Id
var oldTodo = await ToDo.findById(id);
}catch(e){
throw Error("Error occured while Finding the Todo")
}
// If no old Todo Object exists return false
if(!oldTodo){
return false;
}
console.log(oldTodo)
//Edit the Todo Object
oldTodo.title = todo.title
oldTodo.description = todo.description
oldTodo.status = todo.status
console.log(oldTodo)
try{
var savedTodo = await oldTodo.save()
return savedTodo;
}catch(e){
throw Error("And Error occured while updating the Todo");
}
}
However, since I don't want a schema and want to allow anything through, I don't want to assign static values to specific field names like, title, description, status, etc. So, I came up with this:
exports.updateData = async function(update){
var id = update.id
// Check the existence of the query parameters, If they don't exist then assign a default value
var dbName = update.dbName ? update.dbName : 'test'
var collection = update.collection ? update.collection : 'testing';
const Test = mongoose.model(dbName, TestSchema, collection);
try{
//Find the existing Test object by the Id
var existingData = await Test.findById(id);
}catch(e){
throw Error("Error occurred while finding the Test document - " + e)
}
// If no existing Test object exists return false
if(!existingData){
return false;
}
console.log("Existing document is " + existingData)
//Edit the Test object
existingData = JSON.parse(JSON.stringify(update))
//This was another way to overwrite existing field values, but
//performs a "shallow copy" so it's not desireable
//existingData = Object.assign({}, existingData, update)
//existingData.title = update.title
//existingData.description = update.description
//existingData.status = update.status
console.log("New data is " + existingData)
try{
var savedOutput = await existingData.save()
return savedOutput;
}catch(e){
throw Error("An error occurred while updating the Test document - " + e);
}
}
My original problem with this was that I had a lot of issues getting the new values to overwrite the old ones. Now that that's been solved, I am getting the error of "TypeError: existingData.save is not a function". I am thinking the data type changed or something, and now it is not being accepted. When I uncomment the static values that were in the old tutorial code, it works. This is further supported by my console logging before and after I join the objects, because the first one prints the actual data and the second one prints [object Object]. However, I can't seem to figure out what it's expecting. Any help would be greatly appreciated.
EDIT: I figured it out. Apparently Mongoose has its own data type of "Model" which gets changed if you do anything crazy to the underlying data by using things like JSON.stringify. I used Object.prototype.constructor to figure out the actual object type like so:
console.log("THIS IS BEFORE: " + existingData.constructor);
existingData = JSON.parse(JSON.stringify(update));
console.log("THIS IS AFTER: " + existingData.constructor);
And I got this:
THIS IS BEFORE: function model(doc, fields, skipId) {
model.hooks.execPreSync('createModel', doc);
if (!(this instanceof model)) {
return new model(doc, fields, skipId);
}
Model.call(this, doc, fields, skipId);
}
THIS IS AFTER: function Object() { [native code] }
Which showed me what was actually going on. I added this to fix it:
existingData = new Test(JSON.parse(JSON.stringify(update)));
On a related note, I should probably just use the native MongoDB driver at this point, but it's working, so I'll just put it on my to do list for now.
You've now found a solution but I would suggest using the MongoDB driver which would make your code look something along the lines of this and would make the origional issue disappear:
// MongoDB Settings
const MongoClient = require(`mongodb`).MongoClient;
const mongodb_uri = `mongodb+srv://${REPLACE_mongodb_username}:${REPLACE_mongodb_password}#url-here.gcp.mongodb.net/test`;
const db_name = `test`;
let db; // allows us to reuse the database connection once it is opened
// Open MongoDB Connection
const open_database_connection = async () => {
try {
client = await MongoClient.connect(mongodb_uri);
} catch (err) { throw new Error(err); }
db = client.db(db_name);
};
exports.updateData = async update => {
// open database connection if it isn't already open
try {
if (!db) await open_database_connection();
} catch (err) { throw new Error(err); }
// update document
let savedOutput;
try {
savedOutput = await db.collection(`testing`).updateOne( // .save() is being depreciated
{ // filter
_id: update.id // the '_id' might need to be 'id' depending on how you have set your collection up, usually it is '_id'
},
$set: { // I've assumed that you are overwriting the fields you are updating hence the '$set' operator
update // update here - this is assuming that the update object only contains fields that should be updated
}
// If you want to add a new document if the id isn't found add the below line
// ,{ upsert: true }
);
} catch (err) { throw new Error(`An error occurred while updating the Test document - ${err}`); }
if (savedOutput.matchedCount !== 1) return false; // if you add in '{ upsert: true }' above, then remove this line as it will create a new document
return savedOutput;
}
The collection testing would need to be created before this code but this is only a one-time thing and is very easy - if you are using MongoDB Atlas then you can use MongoDB Compass / go in your online admin to create the collection without a single line of code...
As far as I can see you should need to duplicate the update object. The above reduces the database calls from 2 to one and allows you to reuse the database connection, potentially anywhere else in the application which would help to speed things up. Also don't store your MongoDB credentials directly in the code.

How can I log an Object in a Mongoose Schema?

I'm working on a project for a coding challenge and I don't know how to print an object in the Mongo schema below.
I need to know what the contents of the argument in finishQuest() are, but the script hardly gets called at all in the app. console.log gets blocked in testing, so that's a no-go and can't run node file.js, because it's using Node 6 and the file has imports. I've never worked in a file structure this large.
schema.methods._processBossQuest = async function processBossQuest
(options) {
let {
user,
progress,
} = options;
let group = this;
let quest = questScrolls[group.quest.key];
let down = progress.down * quest.boss.str; /* multiply by boss
strength */
// Everyone takes damage
let updates = {
$inc: {'stats.hp': down},
};
group.quest.progress.hp -= progress.up;
/* TODO Create a party preferred language option so emits like this
can be localized. Suggestion: Always display the English version
too. Or, if English is not displayed to the players, at least
include it in a new field in the chat object that's visible in the
database - essential for admins when troubleshooting quests!*/
let playerAttack = `${user.profile.name} attacks
${quest.boss.name('en')} for ${progress.up.toFixed(1)} damage.`;
let bossAttack = CRON_SAFE_MODE || CRON_SEMI_SAFE_MODE ?
`${quest.boss.name('en')} does not attack, because it respects the
fact that there are some bugs\` \`post-maintenance and it doesn't
want to hurt anyone unfairly. It will continue its rampage soon!` :
`${quest.boss.name('en')} attacks party for
${Math.abs(down).toFixed(1)} damage.`;
/*TODO Consider putting the safe mode boss attack message in an ENV
var*/
group.sendChat(`\`${playerAttack}\` \`${bossAttack}\``);
/* If boss has Rage, increment Rage as well */
if (quest.boss.rage) {
group.quest.progress.rage += Math.abs(down);
if (group.quest.progress.rage >= quest.boss.rage.value) {
group.sendChat(quest.boss.rage.effect('en'));
group.quest.progress.rage = 0;
/* TODO To make Rage effects more expandable, let's turn these
into functions in quest.boss.rage */
if (quest.boss.rage.healing) group.quest.progress.hp +=
group.quest.progress.hp * quest.boss.rage.healing;
if (group.quest.progress.hp > quest.boss.hp)
group.quest.progress.hp = quest.boss.hp;
if (quest.boss.rage.mpDrain) {
updates.$set = {'stats.mp': 0};
}
}
}
await User.update(
{_id:
{$in: this.getParticipatingQuestMembers()},
},
updates,
{multi: true}
).exec();
/* Apply changes the currently cronning user locally so we don't
have to reload it to get the updated state
// TODO how to mark not modified?
https://github.com/Automattic/mongoose/pull/1167
// must be notModified or otherwise could overwrite future changes:
if the user is saved it'll save
// the modified user.stats.hp but that must not happen as the hp
value has already been updated by the User.update above */
if (down) user.stats.hp += down;
// Boss slain, finish quest
if (group.quest.progress.hp <= 0) {
group.sendChat(`\`You defeated ${quest.boss.name('en')}! Questing
party members receive the rewards of victory.\``);
// Participants: Grant rewards & achievements, finish quest
await group.finishQuest(shared.content.quests[group.quest.key]);
}
return await group.save();
};

Parse.com master key doesn't let to write PFUser currentUser

UPDATE: In a nutshell, I would like to use the Master key, because I need to write an other user object with my current user, but I don't want to override all security, I just wanna use it in one function. The accepted answer in this question gave a very nice starting point, however I couldn't make it to work. It's the last code block in this question.
I have two separated functions. The first is pure objective-c, it deletes users from the currentUser's firstRelation. It worked well without any problems until i added a different CloudCode function into a different view controller. The CloudCode function uses the master key and adds currentUser to otherUser's sampleRelation & adds otherUser to currentUser's sampleRelation (firstRelation and sampleRelation is two different column inside the User class).
So the problem is when I delete a user from currentUser's firstRelation (with current user) my app crashes, because the user must be authenticated via logIn or signUp. Actually i don't understand this, because in this case I'm writing the currentUser with the currentUser instead of another user, so it must work without any problems (and worked before the CloudCode).
I'm almost sure that it's because I'm using the master key with the CloudCode, but have no idea how can I avoid it. Everything else is still working, for example I can upload images with currentUser.
Here is the code that I'm using for the CloudCode, JavaScript is totally unknown for me, maybe somebody will see what causes the problem.
Parse.Cloud.define('editUser', function(request, response) {
Parse.Cloud.useMasterKey();
var userQuery = new Parse.Query(Parse.User);
userQuery.get(request.params.userId)
.then(function (user) {
var relation = user.relation("sampleRelation");
relation.add(request.user);
// chain the promise
return user.save();
}).then(function (user) {
var currentUser = request.user;
var relation = currentUser.relation("sampleRelation");
relation.add(user);
// chain the new promise
return currentUser.save();
}).then(function () {
response.success();
}, function (error) {
response.error(error);
});
});
It crashes when i try to remove the object:
PFUser *user = [self.friends objectAtIndex:indexPath.row];
PFRelation *myFriendsRel = [self.currentUser relationForKey:#"simpleRelation"];
if ([self isFriend:user]) {
for (PFUser *friendName in self.friends) {
if ([friendName.objectId isEqualToString:user.objectId]){
[self.friends removeObject:friendName];
break; // to exit a loop
}
}
// remove from parse
[myFriendsRel removeObject:user];
NSLog(#"deleted: %#", user.username);
}
[self.currentUser saveInBackgroundWithBlock:^(BOOL succeeded, NSError *error) {
if (error){
NSLog(#"Error %# %#", error, [error userInfo]);
}
}];
This is the newest attempt, that based Fosco's answer from the other question. It works, but the same way as the earlier versions.
Parse.Cloud.define('editUser', function(request, response) {
var userId = request.params.userId;
var User = Parse.Object.extend('_User'),
user = new User({ objectId: userId });
var currentUser = request.user;
var relation = user.relation("friendsRelation");
relation.add(currentUser);
user.save(null, { useMasterKey:true}).then(function(user) {
response.success(user);
}, function(error) {
response.error(error)
});
});
At a quick glance it looks like its failing because you're trying to remove an object from an array whilst it is being iterated. I know this causes a crash in Objective C regardless of whether you're using Parse objects or not.
Try re-writing this segment:
for (PFUser *friendName in self.friends) {
if ([friendName.objectId isEqualToString:user.objectId]){
[self.friends removeObject:friendName];
break; // to exit a loop
}
}
To something like this:
NSMutableArray *tempArray = [[NSMutableArray alloc]init];
for (PFUser *friendName in self.friends) {
if (![friendName.objectId isEqualToString:user.objectId]) {
[tempArray addObject:friendName];
}
self.friends = [NSArray arrayWithArray:tempArray];
Again, only had a quick glance so not 100% if that is your problem but it looks like it, let me know if it helps

Categories

Resources