Develop Reference

loop not functioning correctly, partial data is being pushed to DB

My app uses a slack api that returns a gunzip file. The api is called for a particular date and the response is a gunzip file.
Then the gunzip file should be uncompressed into a .json file.
The json file will have around 1500 entries for that particular date.
Cumulative values of each field is calculated and the final result will be pushed to a database table. The final result that will be pushed to the DB is something like this:
{
"date" : "2022-09-01",
"number_of_users" : 1503,
"is_billable_seat" : 1202,
"is_active_ios" : 600,
"is_active_android" : 400,
"is_active_desktop" :503
}
The code for this is shown below.
An array with 4 dates is passed. In a loop the API is called and data is pushed to Database:
listDate = '2022-09-01', '2022-09-02', '2022-09-03', '2022-09-04']
for(let i=0; i<listDate.length; i++){
let arr = []; let data_to_db = [];
let date = listDate[i]
let URL = `${process.env.slackapi}?type=member&date=${date}&pretty=1`;
request(URL, { //fetch slack data from API
headers: {
Authorization: 'Bearer ' + process.env.accesstoken,
}})
.pipe(zlib.createGunzip())
.pipe(
concat(async (stringBuffer) => {
result = stringBuffer.toString()
fs.writeFileSync('gzip-file-' + date + '.json', result, (error) => {
if (error) throw error;
});
const allFileContents = fs.readFileSync('gzip-file-' + date + '.json', 'utf-8');
allFileContents.split(/\r?\n/).forEach(line => {
arr.push(line)
});
arr.map(obj => {
if(obj){
const myObj = JSON.parse(obj);
data_to_db["date"] = myObj.date
data_to_db["number_of_users"] = ++number_of_users
data_to_db["is_billable_seat"] = myObj.is_billable_seat == true ? is_billable_seat += 1 : is_billable_seat
data_to_db["is_active_ios"] = myObj.is_active_ios == true ? is_active_ios += 1 : is_active_ios
data_to_db["is_active_android"] = myObj.is_active_android == true ? is_active_android += 1 : is_active_android
data_to_db["is_active_desktop"] = myObj.is_active_desktop == true ? is_active_desktop += 1 : is_active_desktop
}
})
console.log(data_to_db)
connection.connect( async() => {
console.log("DB connected!!", date)
await pushSlackDataToDatabase(data_to_db, connection) //push slack data to DB
})
})
);
}
The issue is this doesn't push all the data to the database. It only pushes 2 days data.
I get all 4 days data from the API but only 2 days data gets pushed to the DB.
what is the error in the logic here?

Error when tryign to start the bot with node-fetch

I'm currently setting up a premium queue bot from previous code but I need to switch from snekfetch to node fetch
I've tried using the get function yet to no avail it comes out as nonexistent
run(msg, { user }) {
fetch.get('https://api.2b2t.dev/prioq').then(r => {
fetch.get('https://2b2t.io/api/queue').then(qwerty => {
let entry = r.body[1]
let response = qwerty.body[0][1]
client.user.setActivity("Queue: " + response + " PrioQueue: " + entry);
if(message.channel.id === 598643633398349854) {
message.delete().catch(O_o=>{});
message.author.send("The owner has disabled this command in this channel!")
return
}
const queueembed = new RichEmbed()
.setColor("#32CD32")
.addField('Regular Queue:', response, true)
.addField('Priority Queue:', entry, true)
.addField('Regular Queue Time:', Math.round(response * 0.7) + " minutes.", true)
.addField('Priority Queue Time:', Math.round(entry * 0.7) + " minutes.", true)
.setThumbnail('https://i.redd.it/jyvrickfyuly.jpg')
.setFooter("https://discordapp.com/invite/uGfHNVQ")
message.channel.send(queueembed).then(msg => {
var timerID = setInterval(function() {
const queueembed = new RichEmbed()
.setColor("#32CD32")
.addField('Regular Queue:', response, true)
.addField('Priority Queue:', entry, true)
.addField('Regular Queue Time:', Math.round(response * 0.7) + " minutes.", true)
.addField('Priority Queue Time:', Math.round(entry * 0.7) + " minutes.", true)
.setThumbnail('https://i.redd.it/jyvrickfyuly.jpg')
.setFooter("https://discordapp.com/invite/uGfHNVQ")
message.channel.edit(msg)
}, 5 * 1000);
})})
})
}
}
Usually the bot would start but this error pops up
I have tried switching around fetch and removing get but I am confused
on what do to next
"TypeError: Cannot read property 'get' of undefined"

Looks like there are two problems:
1) fetch is undefined (Do you need to install/require it?)
2) get is not part of the fetch API
For a GET request you can just do fetch('<url>').
Putting the two together:
const fetch = require('node-fetch')
fetch('https://api.2b2t.dev/prioq').then(r => {
fetch('https://2b2t.io/api/queue').then(qwerty => {
// ...rest
})
})
https://github.com/bitinn/node-fetch#api
EDIT
You also need to make the rest of your code fetch-compliant. As per the fetch spec, the body exposed by the response is a ReadableStream, which is probably causing your error. As you can see from its interface it also exposes text() and json() methods:
interface mixin Body {
readonly attribute ReadableStream? body;
readonly attribute boolean bodyUsed;
[NewObject] Promise<ArrayBuffer> arrayBuffer();
[NewObject] Promise<Blob> blob();
[NewObject] Promise<FormData> formData();
[NewObject] Promise<any> json();
[NewObject] Promise<USVString> text();
};
https://fetch.spec.whatwg.org/#body-mixin
I presume the response is JSON so you'll want to use response.json():
fetch('https://api.2b2t.dev/prioq').then(r => r.json()).then(r => {
fetch('https://2b2t.io/api/queue').then(qwerty => qwerty.json()).then(qwerty => {
// ...rest
})

NODE.JS App gets stuck getting 2.5 million of records through and api

I've got the code that retrieves couple of millions of rows from a db and a couple of millions through an api.
let closedOrdersStartDate;
preparedOrdersPromise = tickApiConnector.obtainToken().then(function () {
return ordersController.getAllTrades(0, [], 0, accountIdsToRun);
}).then(function (trades) {
closedOrdersStartDate = new Date();
return Promise.all([trades, fthsApiConnector.getData('closed_orders', '&sort=id', 10000, 0)]);
}).then(function (tradesAndClosedOrderIds) {
//stuck before getting there
console.log('now processing orders time spent from starting getting closed_orders till now is: ' +
((new Date().getTime() - closedOrdersStartDate.getTime())/ 1000) + ' seconds');
return ordersController.processOrders(tradesAndClosedOrderIds[0], tradesAndClosedOrderIds[1]);
});
The app gets stuck after calling that getData() function.
getData: async function (entityName, getParams = '', perRequest = 10000, skip = 0) {
if(getParams.indexOf('&take=') !== -1) {
return fthsApiRequest(entityName, getParams);
}
const totalCount = await fthsApiRequest(entityName, getParams + '&getCount');
let result = [];
let count = 0;
while(count < totalCount) {
result = result.concat(await fthsApiRequest(entityName, getParams + '&take=' + perRequest + '&skip=' + count));
count += perRequest;
}
return result;
}
The function executes till the last request(I see it in logs) and after that the script gets unresponsable. I thought that it could be a memory leak and I've rewritten that getData() function in different ways but still, there's enough memory on the server and the script doesn't consume even a bit of it. Still I get 100% of CPU load in a while after that last iteration of getData() is rant. After that the app gets stuck forever.
I've tried profiling it. And there are thousands of Code move event for unknown code: 0x2a5c24bfb4c0, I'm not sure what that means, but there could be a clue in that. Here's the V8.log

The possible problem maybe in block :
while(count < totalCount) { result = result.concat(await fthsApiRequest(entityName, getParams + '&take=' + perRequest + '&skip=' + count)); count += perRequest; }
Ensure that the api give response. And on the last statement :
return result;
In async function the better use :
return Promise.resolve(result);

Observable piped through `share()` invokes a single observer an unnecessary number of times

I'm trying to test latency of messages sent through a shared/hot observable. I noticed when I have multiple observers on a single shared observable, a single observer gets invoked n times from a single message (where n is the number of observers on the shared observable).
I ran the code below with 10 Observers and 1 Message Per Observer, and each observer gets invoked 10 times per message (implying 100 total observer.next() calls). From my understanding of observers/observable, each observer should only get invoked once per message. Am I just using the share() operator incorrectly here? Or is my understanding of it in general flawed?
const getMessageLatency = (observersCount, messagesPerObserver) => {
const completedMessages = [];
const source = new Subject();
const sharedObservable = source.pipe(
tap((message) => console.log(`Subject: Incoming for ${message.id}`)),
share()
);
// Setup observers
for (i = 0; i < observersCount; ++i) {
sharedObservable
.pipe(
tap((message) => console.log(`SharedObservable: Incoming for ${message.id}`)),
filter((message) => message.id === getObserverId(i)),
tap(() => console.log(`Filtered for ${getObserverId(i)}`))
)
.subscribe((message) => {
const date = new Date();
message.endTime = date.getMilliseconds();
completedMessages.push(message);
})
}
// send out messages
for (i = 0; i < observersCount; ++i) {
for (j = 0; j < messagesPerObserver; ++j) {
const date = new Date();
const message = {
id: getObserverId(i),
startTime: date.getMilliseconds()
}
// send message
source.next(message);
}
}
// process data (get average message latency)
const totalMessageLatency = completedMessages.reduce(
(accumulatedLatency, currentMessage) => {
const currentMessageLatency =
currentMessage.endTime - currentMessage.startTime;
return accumulatedLatency + currentMessageLatency;
}, 0);
const averageLatency = totalMessageLatency / completedMessages.length;
console.log("==============================================================================");
console.log(`Observers: ${observersCount}, MessagesPerObserver: ${messagesPerObserver}`);
console.log(`Total Messages Sent: ${observersCount * messagesPerObserver}`);
console.log(`Total Messages Received: ${completedMessages.length}`);
console.log(`Average Latency per Message: ${averageLatency}`);
console.log("==============================================================================");
return averageLatency;
}
Once this is done running, if "Total Messages Sent" is x, then "Total Message Received" will be x^2

Added let to the declarations of my for-loops. Y'all can tell I'm new to JavaScript too.
Thank you cartant

Create a cell range output dynamically

In my add-in I am making an HTTP request and receiving an output. I want to place that output into a binding and have it expand the binding if necessary because the user won't necessarily know how many rows x columns the output will be. How would I go about doing this? Currently I am binding to a range, but if that range does not match the size of the [[]] that I am providing, then the data is not displayed in the sheet. So, this ends up requiring the user to know the size of the output.
What I'm doing currently using Angular is as follows (the problem with this being that the output isn't always the same size as the Office.BindingType.Matrix that the user selected in the spreadsheet):
I create the binding to where the output should be placed as follows:
inputBindFromPrompt(parameterId: number): Promise<IOfficeResult> {
let bindType: Office.BindingType;
if(this.inputBindings[parameterId].type != 'data.frame' && this.inputBindings[parameterId].type != 'vector') {
bindType = Office.BindingType.Text;
} else {
bindType = Office.BindingType.Matrix;
}
return new Promise((resolve, reject) => {
this.workbook.bindings.addFromPromptAsync(bindType, { id: this.inputBindings[parameterId].name },
(addBindingResult: Office.AsyncResult) => {
if(addBindingResult.status === Office.AsyncResultStatus.Failed) {
reject({
error: 'Unable to bind to workbook. Error: ' + addBindingResult.error.message
});
} else {
this.inputBindings[parameterId].binding = addBindingResult.value;
resolve({
success: 'Created binding ' + addBindingResult.value.type + ' on ' + addBindingResult.value.id
});
}
})
})
}
Then when the user submits via a button, the inputs are passed to a HTTP request service which then receives an output that I process into an array of arrays so that it can go into an Office.BindingType.Matrix:
this.isBusy = true;
this.feedback = 'submitted';
// Grab the values from the form
// Send as a POST and receive an output
// Put the output in the Excel sheet
this.webServicesService.postWebServices(this.service, this.inputParameters)
.subscribe(
(data: any) => {
// Correctly received data
// Access the data by name while looping through output parameters
this.error = false;
this.feedback = 'received data';
let i = 0;
this.outputParameters.forEach(element => {
// temporary name to identify the parameter
let name = element.name;
// Set the data value in the parameter
if(element.type == 'data.frame') {
let parameter = data[name];
this.feedback = parameter;
let excelData = [];
for(var key in parameter) {
if(parameter.hasOwnProperty(key)) {
var val = parameter[key];
excelData.push(val);
}
}
element.value = excelData;
}
else {
element.value = data[name];
}
// Set value in the form
let param = (<FormArray>this.serviceForm.controls['outputParameters']).at(i);
param.patchValue({
value: element.value
});
// Set value in the spreadsheet
this.excelService.outputSetText(i, element.value)
.then((result: IOfficeResult) => {
this.onResult(result);
i++;
});
}, (result: IOfficeResult) => {
this.onResult(result);
});
},
(error) => {
if(error.status == 400 || error.status == 401) {
// Return user to authentication page
this.authService.logout();
this.router.navigate(['/']);
} else {
// Tell user to try again
this.error = true;
}
}
);
The line above that is setting the value to the Office.Matrix.Binding is this.excelService.outputSetText(i, element.value), which calls this method in the Excel Service:
outputSetText(parameterId: number, data: any): Promise<IOfficeResult> {
return new Promise((resolve, reject) => {
if(this.outputBindings[parameterId].binding) {
this.outputBindings[parameterId].binding.setDataAsync(data, function (result: Office.AsyncResult) {
if(result.status == Office.AsyncResultStatus.Failed) {
reject({ error: 'Failed to set value. Error: ' + result.error.message });
} else {
let test: Office.Binding;
resolve({
success: 'successfully set value'
});
}
})
} else {
reject({
error: 'binding has not been created. bindFromPrompt must be called'
});
}
})
}
It's essentially using addFromPromptAsync() to set an output spot for the HTTP request. Then the user submits which sends the request, receives the data back and processes it into an array of arrays [[]] so that it can be the correct data format for Office.BindingType.Matrix. However, unless this is the same number of rows and columns as the binding originally selected, it won't display in the sheet. So, is there a binding type that will dynamically grow based on the data I give it? Or would I just need to release the current binding and make a new binding according to the size of the HTTP response data?

So long as you're using the "shared" (Office 2013) APIs, you will have this issue.
However, in the host-specific (2016+) APIs, you can easily solve the problem by resizing the range to suit your needs. Or more precisely, getting the binding, then asking for its range, then getting just the first (top-left) cell, and then resizing it:
await Excel.run(async (context) => {
let values = [
["", "Price"],
["Apple", 0.99],
["Orange", 1.59],
];
let firstCell = context.workbook.bindings.getItem("TestBinding").getRange().getCell(0, 0);
let fullRange = firstCell.getResizedRange(
values.length - 1, values[0].length - 1);
fullRange.values = values;
await context.sync();
});
You can try this snippet live in literally five clicks in the new Script Lab (https://aka.ms/getscriptlab). Simply install the Script Lab add-in (free), then choose "Import" in the navigation menu, and use the following GIST URL: https://gist.github.com/Zlatkovsky/5a2fc743bc9c8556d3eb3234e287d7f3. See more info about importing snippets to Script Lab.