Develop Reference

Is this a proper async/await implementation when the nested promises are literally thousands?

Regarding: https://github.com/gvasquez95/binance-trader/issues/8
I'm concerned that the actual "inner" await (in every single invocation of getPriceChange) isn't blocking asynchronous invocation of the about 1.5k calls this method has on the outer loop (getPriceChanges). As if it was blocking, then the loop would be serial processing.
From the performance view and timing, I'd say it is working as expected, as it takes just some seconds to complete the whole process but, I'd like a programming ACK of the assumptions, if possible, please.
Block codes for those that don't want to get into GitHub and, prefer the code more visible here:
Outer loop:
async function getPriceChanges (since, breakpoint) {
const changes = []
for (const [i, ticker] of tickers.entries()) {
if (!ticker.includes('BIDR') && !ticker.includes('BIFI')) {
changes.push(getPriceChange(ticker, since, breakpoint))
}
}
const res = await Promise.all(changes)
let symbol
let maxChange = 0
let slope = 0
for (const ticker of res) {
if (ticker.percentChange > maxChange && ticker.slope > Math.pow(10, -1 * NUM_DECIMALS)) {
maxChange = ticker.percentChange
symbol = ticker.symbol
slope = ticker.slope
}
}
return { symbol, maxChange: Math.round(100 * maxChange) / 100, slope: round(slope, NUM_DECIMALS) }
}
Inner getPriceChange function:
async function getPriceChange (symbol, since, breakpoint) {
const params = {
TableName: DYNAMODB_TABLE,
ProjectionExpression: '#timestamp, price',
KeyConditionExpression: 'symbol = :symbol and #timestamp > :timestamp',
ExpressionAttributeNames: {
'#timestamp': 'timestamp'
},
ExpressionAttributeValues: {
':symbol': symbol,
':timestamp': since
}
}
const res = await documentClient.query(params).promise()
const prev = []
const recent = []
const trendData = []
for (const data of res.Items) {
if (data.timestamp < breakpoint) {
prev.push(data.price)
} else {
trendData.push({ x: trendData.length, y: data.price })
recent.push(data.price)
}
}
let sumPrev = 0
let sumRecent = 0
for (const price of prev) { sumPrev += price }
for (const price of recent) { sumRecent += price }
const avgPrev = sumPrev / prev.length
const avgRecent = sumRecent / recent.length
const trend = createTrend(trendData, 'x', 'y')
return { symbol, percentChange: (100 * (avgRecent - avgPrev)) / avgRecent, slope: trend.slope }
}
Update:
Function execution takes about 30 seconds in a Node.js 14.x AWS Lambda environment, configured with 624 MB RAM and, from a cost perspective the both the DynamoDB query (ReadRequests) costs and Lambda executions are 1/50th of the actual DynamoDB Batch Put operations (WriteRequest), so optimizing the query should be a needed goal.

Your loop is correctly asynchronous. And it seems you're concerned about how many queries will be in flight at the same time. In your case, you intended for many concurrent queries, and you have done as such.
Explanation
In the simple example of ordering 10 pizzas, consider these arrays:
concurrently
This is an array of pizza orders. We can await all 10 orders to get an array of 10 pizzas. Assuming there are at least 10 delivery people working that night, there's a good chance you'll get all your pizzas within 30 minutes.
for (let i = 0; i < 10; ++i) {
orders.push(orderPizza());
}
pizzas = await Promise.all(orders);
sequentially
This is an array of pizzas. Each order is awaited as it is issued. If each pizza is delivered in 30 minutes or it's free this is likely to result in about 5 hours (10 * ~30 minutes) of waiting for that last pizza because the next pizza isn't ordered until the previous one arrives. (No need for a final Promise.all in this case as each promise is already resolved before continuing with the loop.)
for (let i = 0; i < 10; ++i) {
pizzas.push(await orderPizza());
}

Not sure I understand what your question is here exactly. Your assumptions about your code seem to be correct – I (and many others I assume) would be highly surprised if 1500 promises changed anything in their execution model.
As a general note, if you need more control over promise execution, I can recommend checking out Sindre Sorhus' collection of promise helpers. You'll find tools for limiting concurrency, allowing for promise failures, async mapping … and much more.

How do I loop over a VERY LARGE 2D array without causing a major performace hit?

I am attempting to iterate over a very large 2D array in JavaScript within an ionic application, but it is majorly bogging down my app.
A little background, I created custom search component with StencilJS that provides suggestions upon keyup. You feed the component with an array of strings (search suggestions). Each individual string is tokenized word by word and split into an array and lowercase
For example, "Red-Winged Blackbird" becomes
['red','winged','blackbird']
So, tokenizing an array of strings looks like this:
[['red','winged','blackbird'],['bald','eagle'], ...]
Now, I have 10,000+ of these smaller arrays within one large array.
Then, I tokenize the search terms the user inputs upon each keyup.
Afterwards, I am comparing each tokenized search term array to each tokenized suggestion array within the larger array.
Therefore, I have 2 nested for-of loops.
In addition, I am using Levenshtein distance to compare each search term to each element of each suggestion array.

I had a couple drinks so please be patient while i stumble through this.
To start id do something like a reverse index (not very informative). Its pretty close to what you are already doing but with a couple extra steps.
First go through all your results and tokenize, stem, remove stops words, decap, coalesce, ects. It looks like you've already done this but i'm adding an example for completion.
const tokenize = (string) => {
const tokens = string
.split(' ') // just split on words, but maybe rep
.filter((v) => v.trim() !== '');
return new Set(tokens);
};
Next what we want to do is generate a map that takes a word as an key and returns us a list of document indexes the word appears in.
const documents = ['12312 taco', 'taco mmm'];
const index = {
'12312': [0],
'taco': [0, 1],
'mmm': [2]
};
I think you can see where this is taking us... We can tokenize our search term and find all documents each token belongs, to work some ranking magic, take top 5, blah blah blah, and have our results. This is typically the way google and other search giants do their searches. They spend a ton of time in precomputation so that their search engines can slice down candidates by orders of magnitude and work their magic.
Below is an example snippet. This needs a ton of work(please remember, ive been drinking) but its running through a million records in >.3ms. Im cheating a bit by generate 2 letter words and phrases, only so that i can demonstrate queries that sometimes achieve collision. This really doesn't matter since the query time is on average propionate to the number of records. Please be aware that this solution gives you back records that contain all search terms. It doesn't care about context or whatever. You will have to figure out the ranking (if your care at this point) to achieve the results you want.
const tokenize = (string) => {
const tokens = string.split(' ')
.filter((v) => v.trim() !== '');
return new Set(tokens);
};
const ri = (documents) => {
const index = new Map();
for (let i = 0; i < documents.length; i++) {
const document = documents[i];
const tokens = tokenize(document);
for (let token of tokens) {
if (!index.has(token)) {
index.set(token, new Set());
}
index.get(token).add(i);
}
}
return index;
};
const intersect = (sets) => {
const [head, ...rest] = sets;
return rest.reduce((r, set) => {
return new Set([...r].filter((n) => set.has(n)))
}, new Set(head));
};
const search = (index, query) => {
const tokens = tokenize(query);
const canidates = [];
for (let token of tokens) {
const keys = index.get(token);
if (keys != null) {
canidates.push(keys);
}
}
return intersect(canidates);
}
const word = () => Math.random().toString(36).substring(2, 4);
const terms = Array.from({ length: 255 }, () => word());
const documents = Array.from({ length: 1000000 }, () => {
const sb = [];
for (let i = 0; i < 2; i++) {
sb.push(word());
}
return sb.join(' ');
});
const index = ri(documents);
const st = performance.now();
const query = 'bb iz';
const results = search(index, query);
const et = performance.now();
console.log(query, Array.from(results).slice(0, 10).map((i) => documents[i]));
console.log(et - st);
There are some improvements you can make if you want. Like... ranking! The whole purpose of this example is to show how we can cut down 1M results to maybe a hundred or so canidates. The search function has some post filtering via intersection which probably isn't what you want you want but at this point it doesn't really matter what you do since the results are so small.

I want to get the average my firebase data

I want to average the related values ​​when the data in the FireBase is updated.
I am using Firebase functions and can not load data.
I can change the data I want when the event occurs, but I can not calculate the average of the data.
exports.taverage = functions.database.ref('/User/tsetUser/monthQuit/{pushId}')
.onCreate((snapshot, context) => {
const promiseRoomUserList = admin.database().ref('/User/tsetUser/monthQuit/{pushId}').once('value');
var sum=0;
const arrayTime = [];
snapshot.forEach(snapshot => {
arrayTime.push('/User/tsetUser/monthQuit/{pushId}'.val());
})
for(let i=0; i<arrayTime.length; i++){
sum+=arrayTime[i];
}
return admin.database().ref('/User/tsetUser/inform/standardQuit').set(sum);
});
//I Want 'standardQuit' value set average.

I'm not sure why you can't calculate the average, but a simpler version of your code would be:
exports.taverage = functions.database.ref('/User/tsetUser/monthQuit/{pushId}')
.onCreate((snapshot, context) => {
return admin.database().ref('/User/tsetUser/monthQuit/{pushId}').once('value')
.then(function(snapshot) {
let sum=0;
snapshot.forEach(child => {
sum = sum + child.val();
})
let avg = sum / snapshot.numChildren();
return admin.database().ref('/User/tsetUser/inform/standardQuit').set(avg);
});
});
The biggest differences:
This code returns promises from both the top-level, and the nested then(). This is needed so Cloud Functions knows when your code is done, and it can thus stop billing you (and potentially shut down the container).
We simply add the value of each child to the sum, since you weren't using the array in any other way. Note that the child.val() depends on your data structure, which you didn't share. So if it fails there, you'll need to update how you get the exact value (or share you data structure with us).
The code actually calculates the average by dividing the sum by the number of child nodes.
Consider using a moving average
One thing to keep in mind is that you're now reading all nodes every time one node gets added. This operation will get more and more expensive as nodes are added. Consider if you can use a moving average, which wouldn't require all child nodes, but merely the current average and the new child node. The value will be an approximate average where more recent value typically have more weight, and is much cheaper to calculate:
exports.taverage = functions.database.ref('/User/tsetUser/monthQuit/{pushId}')
.onCreate((snapshot, context) => {
return admin.database().ref('/User/tsetUser/inform/standardQuit').transaction(function(avg) {
if (!avg) avg = 0;
return (15.0 * avg + snapshot.val()) / 16.0;
});
});

JavaScript/React Native array(objects) sort

I'm starting with react-native building an app to track lap times from my RC Cars. I have an arduino with TCP connection (server) and for each lap, this arduino sends the current time/lap for all connected clients like this:
{"tx_id":33,"last_time":123456,"lap":612}
In my program (in react-native), I have one state called dados with this struct:
dados[tx_id] = {
tx_id: <tx_id>,
last_time:,
best_lap:0,
best_time:0,
diff:0,
laps:[]
};
This program connects to arduino and when receive some data, just push to this state. More specific in laps array of each transponder. Finally, I get something like this:
dados[33] = {
tx_id:33,
last_time: 456,
best_lap: 3455,
best_time: 32432,
diff: 32,
laps: [{lap:1,time:1234},{lap:2,time:32323},{lap:3,time:3242332}]
}
dados[34] = {
tx_id:34,
last_time: 123,
best_lap: 32234,
best_time: 335343,
diff: 10,
laps: [{lap:1,time:1234},{lap:2,time:32323},{lap:3,time:3242332}]
}
dados[35] = {
tx_id:35,
last_time: 789,
best_lap: 32234,
best_time: 335343,
diff: 8,
laps: [{lap:1,time:1234},{lap:2,time:32323},{lap:3,time:3242332},{lap:4,time:343232}]
}
This data in rendered to View's using map function (not a FlatList).
My problem now is that I need to order this before printing on screen.
Now, with this code, data are printed using tx_id as order, since it's the key for main array. Is there a way to order this array using number of elements in laps property and the second option to sort, use last_time property of element?
In this case, the last tx of my example (35) would be the first in the list because it has one lap more than other elements. The second item would be 34 (because of last_time). And the third would be tx 33.
Is there any way to to this in JavaScript, or I need to create a custom functions and check every item in recursive way?!

Tks #crackhead420
While waiting for reply to this question, I just found what you said.... :)
This is my final teste/solution that worked:
var t_teste = this.state.teste;
t_teste[33] = {tx_id: 33, last_time:998,best_lap:2,best_time:123,diff:0,laps:[{lap:1,time:123},{lap:2,time:456}]};
t_teste[34] = {tx_id: 34, last_time:123,best_lap:2,best_time:123,diff:0,laps:[{lap:1,time:123},{lap:2,time:456}]};
t_teste[35] = {tx_id: 35, last_time:456,best_lap:2,best_time:123,diff:0,laps:[{lap:1,time:123},{lap:2,time:456},{lap:3,time:423}]};
t_teste[36] = {tx_id: 36, last_time:789,best_lap:2,best_time:123,diff:0,laps:[{lap:1,time:123},{lap:2,time:456}]};
console.log('Teste original: ',JSON.stringify(t_teste));
var saida = t_teste.sort(function(a, b) {
if (a.laps.length > b.laps.length) {
return -1;
}
if (a.laps.length < b.laps.length) {
return 1;
}
// In this case, the laps are equal....so let's check last_time
if (a.last_time < b.last_time) {
return -1; // fastest lap (less time) first!
}
if (a.last_time > b.last_time) {
return 1;
}
// Return the same
return 0;
});
console.log('Teste novo: ',JSON.stringify(saida));

Using some simple helper functions, this is definitely possible:
const data = [{tx_id:33,last_time:456,best_lap:3455,best_time:32432,diff:32,laps:[{lap:1,time:1234},{lap:2,time:32323},{lap:3,time:3242332}]},{tx_id:34,last_time:123,best_lap:32234,best_time:335343,diff:10,laps:[{lap:1,time:1234},{lap:2,time:32323},{lap:3,time:3242332}]},{tx_id:35,last_time:789,best_lap:32234,best_time:335343,diff:8,laps:[{lap:1,time:1234},{lap:2,time:32323},{lap:3,time:3242332},{lap:4,time:343232}]}]
const sortBy = fn => (a, b) => -(fn(a) < fn(b)) || +(fn(a) > fn(b))
const sortByLapsLength = sortBy(o => o.laps.length)
const sortByLastTime = sortBy(o => o.last_time)
const sortFn = (a, b) => -sortByLapsLength(a, b) || sortByLastTime(a, b)
data.sort(sortFn)
// show new order of `tx_id`s
console.log(data.map(o => o.tx_id))
sortBy() (more explanation at the link) accepts a function that selects a value as the sorting criteria of a given object. This value must be a string or a number. sortBy() then returns a function that, given two objects, will sort them in ascending order when passed to Array.prototype.sort(). sortFn() uses two of these functions with a logical OR || operator to employ short-circuiting behavior and sort first by laps.length (in descending order, thus the negation -), and then by last_time if two objects' laps.length are equal.

Its possible to sort an object array by theire values:
dados.sort(function(a, b) {
return a.last_time - b.last_time;
});

Creating millions of Objects in Javascript

Let me be the first to say that this isn't something I normally do, but out of curiousity, I'll see if anyone has a good idea on how to handle a problem like this.
The application I am working on is a simulated example of the game Let's make a Deal featuring the Monty Hall problem.
I won't go into details about my implementation, but it more or less allows a user to enter a number of how many games they want to simulate, and then if an option is toggled off, the player of those x games won't switch their choice, while if it is toggled on, they will switch their choice every single instance of the game.
My object generator looks like this:
const game = function(){
this[0] = null;
this[1] = null;
this[2] = null;
this.pick = Math.floor(Math.random() * 3);
this.correctpick = Math.floor(Math.random() * 3);
this[this.correctpick] = 1;
for (let i=0; i<3; i++){
if ((this[i] !== 1) && (i !== this.pick)){
this.eliminated = i;
break;
}
}
}
const games = arg => {
let ret = [];
for(let i=0; i<arg; i++){
ret.push(new game);
}
return ret;
}
This structure generates an array which i stringify later that looks like this:
[
{
"0": 1,
"1": null,
"2": null,
"pick": 2,
"correctpick": 0,
"eliminated": 1
},
{
"0": null,
"1": null,
"2": 1,
"pick": 2,
"correctpick": 2,
"eliminated": 0
}
]
As sloppy as the constructor for game looks, the reason is because I have refactored it into having as few function calls as possible, where now I'm literally only calling Math functions at the current time (I removed any helper functions that made the code easier to read, in opt for performance).
This app can be ran both in the browser and in node (cross platform), but I have clamped the arg a user can pass into the games function to 5 million. Any longer than that and the process (or window) freezes for longer than a few seconds, or even potentially crashes.
Is there anything else I can do to increase performance if a huge number is given by a user? Also, if you need more information, I will be happy to supply it!
Thanks!

The obvious performance optimisation would be not to create and store 5 million objects at all, relieving memory pressure. Instead you'd create the objects on the fly only when you need them and throw them away immediately after. I'm not sure what your app does, but it sounds like you want to re-use the same game instances when evaluating results with the different options. In that case, you need to store them of course - but I'd advise to re-think the design and consider immediately evaluating each game with all possible options, accumulating only the results for each choice of options but not keeping all games in memory.
Apart from that, I'd recommend to simplify a bit:
You can drop that loop completely and use some clever arithmetic to find the eliminated option: this.eliminated = this.pick == this.correctpick ? +!this.pick : 3 - this.pick - this.correctpick;. Or use a simple lookup table this.eliminated = [1, 2, 1, 2, 0, 0, 1, 0, 0][this.pick * 3 + this.correctpick].
I'd avoid changing the type of the array elements from null (reference) to 1 (number). Just keep them as integers and initialise your elements with 0 instead.
Don't store 6 properties in your object that are completely redundant. You only need 2 of them: pick and correctpick - everything else can be computed on the fly from them when you need it. Precomputing and storing it would only be advantageous if the computation was heavy and the result was used often. Neither of this is the case, but keeping a low memory footprint is important (However, don't expect much from this).

Not sure about your implementation, but do you really need an Array?
How about only using results (see snippet)?
If it's blocking the browser that worries you, maybe delegating the work to a web worker is the solution for that: see this jsFiddle for a web worker version of this snippet.
(() => {
document.querySelector("#doit")
.addEventListener("click", playMontyHall().handleRequest);
function playMontyHall() {
const result = document.querySelector("#result");
const timing = document.querySelector("#timing");
const nOfGames = document.querySelector("#nGames");
const switchDoors = document.querySelector("#switchyn");
// Create a game
const game = (doSwitch) => {
const doors = [0, 1, 2];
const pick = Math.floor(Math.random() * 3);
const correctPick = Math.floor(Math.random() * 3);
const eliminated = doors.filter(v => v !== pick && v !== correctPick)[0];
return {
correctpick: correctPick,
pick: doSwitch ? doors.filter(v => v !== pick && v !== eliminated)[0] : pick,
eliminated: eliminated,
};
};
const getWinner = game => ~~(game.correctpick === game.pick);
// Sum wins using a generator function
const winningGenerator = function* (doSwitch, n) {
let wins = 0;
while (n--) {
wins += getWinner(game(doSwitch));
yield wins;
}
};
// calculate the number of succeeded games
const calculateGames = (nGames, switchAlways) => {
const funNGames = winningGenerator(switchAlways, nGames);
let numberOfWins = 0;
while (nGames--) {
numberOfWins = funNGames.next().value;
}
return numberOfWins;
}
const cleanUp = playOut => {
result.textContent =
"Playing ... (it may last a few seconds)";
timing.textContent = "";
setTimeout(playOut, 0);
};
const report = results => {
timing.textContent = `This took ${
(performance.now() - results.startTime).toFixed(3)} milliseconds`;
result.innerHTML =
`<b>${!results.switchAlways ? "Never s" : "Always s"}witching doors</b>:
${results.winners} winners out of ${results.nGames} games
(${((results.winners/+results.nGames)*100).toFixed(2)}%)`;
};
// (public) handle button click
function clickHandle() {
cleanUp(() => {
const nGames = nOfGames.value || 5000000;
const switchAlways = switchDoors.checked;
report({
switchAlways: switchAlways,
startTime: performance.now(),
winners: calculateGames(nGames, switchAlways),
nGames: nGames
});
});
}
return {
handleRequest: clickHandle
};
}
})();
body {
margin: 2em;
font: normal 12px/15px verdana, arial;
}
#timing {
color: red;
}
<p><input type="number" id="nGames" value="5000000"> N of games</p>
<p><input type="checkbox" id="switchyn"> Always switch doors</p>
<p><button id="doit">Play</button>
<p id="result"></p>
<p id="timing"></p>