React right way to retrieve multiple images from S3 Bucket - javascript

I have an array of objects which some of the items within it has a "logo" object containing a key of an image stored on a S3 Bucket.
This is an optional field, so sometimes it doesn't exist in the object. To retrieve the image, I am using the Storage api from AWS-Amplify. This Storage function returns me a promise and here is my first problem.
const fetchUsers = async () => {
setLoading(true);
const formatedData = [];
const { data } = await API.graphql(graphqlOperation(listUsers));
console.log({ data });
data.listUsers.items.map(async user => {
if (user.logo) {
console.log(Storage.get());
const image = await Storage.get(user.logo.key);
user.image = image;
formatedData.push(user);
} else {
formatedData.push(user);
}
})
}
While I am waiting the response for the response from the Storage.get function, the loop keep running, which is modifying the original sorting that came from the data array.
This is causing all the entries that has images are appearing at last on the array.
So given this, I assume that
There's a way to prevent the loop to run while I am still waiting for the response
There's a better/more efficient way to get these images from S3.
Any thoughts how to make it run more smoothly ?

Related

Should I be storing poke-api data locally?

I'm building a complete pokedex app using react-native/expo with all 900+ Pokémon.
I've tried what seems like countless ways of fetching the data from the API, but it's really slow.
Not sure if it's my code or the sheer amount of data:
export const getAllPokemon = async (offset: number) => {
const data = await fetch(
`https://pokeapi.co/api/v2/pokemon?limit=10&offset=${offset}`
);
const json = await data.json();
const pokemonURLS: PokemonURLS[] = json.results;
const monData: PokemonType[] = await Promise.all(
pokemonURLS.map(async (p, index: number) => {
const response = await fetch(p.url);
const data: PokemonDetails = await response.json();
const speciesResponse = await fetch(data.species.url);
const speciesData: SpeciesInfo = await speciesResponse.json();
return {
name: data.name,
id: data.id,
image: `https://raw.githubusercontent.com/PokeAPI/sprites/master/sprites/pokemon/other/official-artwork/${data.id}.png`,
description: speciesData.flavor_text_entries,
varieties: speciesData.varieties,
color: speciesData.color.name,
types: data.types,
abilities: data.abilities,
};
})
);
Then I'm using it with a useEffect that increases offset by 10 each time and concats the arrays, until offset > 900.
However like I said, it's really slow.
Should I be saving this data locally to speed things up a little?
And how would I go about it? Should I use local storage or save an actual file somewhere in my project folder with the data?
The biggest performance issue I can see is the multiple fetches you perform as you loop though each pokemon.
I'm guessing that the data returned by the two nested fetches (response and speciesResponse) is reference data and are potentially the same for multiple pokemon. If this is the case, and you can't change the api, then two options pop to mind:
Load the reference data only when needed ie. when a user clicks on a pokemon to view details.
or
Get ALL the reference data before the pokemon data and either combine it with your pokemon fetch results or store it locally and reference it as needed. The first way can be achieved using local state - just keep it long enough to merge the relevant data with the pokemon data. The second will need application state like redux or browser storage (see localStorage or indexeddb).

In getServerSideProps, how to execute the return statement after all fetching is complete?

I have access to the username of a person and I can access their corresponding data object using one fetch from firebase. This data object has a lot of properties, including one which is called "uploads" which is an array of documentIDs where each id is that of a post uploaded by the user. I want to fetch all these documents, make it an array and return the ENTIRE array back to the page.
I have written this code so far: https://i.stack.imgur.com/OC74t.png
What is happening is that the return on line 54 executes before all elements of postIDs is looped through because of which, an empty array is returned back to the component.
Let me know if further details are required. Please help me out. Thanks in advance!
Try to change your loop to a for...of:
for (const postID of currentUserData.uploads) {
const postReference = doc(db, `posts/${postID}`)
const snapshot = await getDoc(postReference)
const postData = snapshot.data()
postData.time = postData.time.toJSON()
uploadsArray.push(postData)
}
return { props: { uploadsArray }}
What about try to use Promise.all?
var promises = [];
promises.push();
// Wait for all promises to resolve
Promise.all(promises).then(function(res) {
// Do something...
});

How to use Promise.all with multiple Firestore queries

I know there are similar questions to this on stack overflow but thus far none have been able to help me get my code working.
I have a function that takes an id, and makes a call to firebase firestore to get all the documents in a "feedItems" collection. Each document contains two fields, a timestamp and a post ID. The function returns an array with each post object. This part of the code (getFeedItems below) works as expected.
The problem occurs in the next step. Once I have the array of post ID's, I then loop over the array and make a firestore query for each one, to get the actual post information. I know these queries are asynchronous, so I use Promise.all to wait for each promise to resolve before using the final array of post information.
However, I continue to receive "undefined" as a result of these looped queries. Why?
const useUpdateFeed = (uid) => {
const [feed, setFeed] = useState([]);
useEffect(() => {
// getFeedItems returns an array of postIDs, and works as expected
async function getFeedItems(uid) {
const docRef = firestore
.collection("feeds")
.doc(uid)
.collection("feedItems");
const doc = await docRef.get();
const feedItems = [];
doc.forEach((item) => {
feedItems.push({
...item.data(),
id: item.id,
});
});
return feedItems;
}
// getPosts is meant to take the array of post IDs, and return an array of the post objects
async function getPosts(items) {
console.log(items)
const promises = [];
items.forEach((item) => {
const promise = firestore.collection("posts").doc(item.id).get();
promises.push(promise);
});
const posts = [];
await Promise.all(promises).then((results) => {
results.forEach((result) => {
const post = result.data();
console.log(post); // this continues to log as "undefined". Why?
posts.push(post);
});
});
return posts;
}
(async () => {
if (uid) {
const feedItems = await getFeedItems(uid);
const posts = await getPosts(feedItems);
setFeed(posts);
}
})();
}, []);
return feed; // The final result is an array with a single "undefined" element
};
There are few things I have already verified on my own:
My firestore queries work as expected when done one at a time (so there are not any bugs with the query structures themselves).
This is a custom hook for React. I don't think my use of useState/useEffect is having any issue here, and I have tested the implementation of this hook with mock data.
EDIT: A console.log() of items was requested and has been added to the code snippet. I can confirm that the firestore documents that I am trying to access do exist, and have been successfully retrieved when called in individual queries (not in a loop).
Also, for simplicity the collection on Firestore currently only includes one post (with an ID of "ANkRFz2L7WQzA3ehcpDz", which can be seen in the console log output below.
EDIT TWO: To make the output clearer I have pasted it as an image below.
Turns out, this was human error. Looking at the console log output I realised there is a space in front of the document ID. Removing that on the backend made my code work.

How to use incoming Binance WebSocket data?

My question is how can I make use of WebSocket data in general. im using this package https://www.npmjs.com/package/binance to get binance live tickers. what i want to do is take that live data and compare it to some other exchanges data which im calling using axios. but all i get is continous stream of binance data and donno how to make use of it, i am trying to run function inside the stream but that does not seem to help as the data stream is continuous and axios request takes time to return. i can add more if my question is incomplete.
The binance package only supports websocket streams, but REST API (either directly or using a different package) will help you achieve your goal more easily.
You can have two functions - each for retrieving price form the exchange, and then compare the price from the different sources.
const axios = require('axios');
const getBinanceBtcPrice = async () => {
// binance allows retrieving a single symbol
// if you omit the param, it would return prices of all symbols
// and you'd have to loop through them like in the WazirX example
const response = await axios.get('https://api.binance.com/api/v3/avgPrice?symbol=BTCUSDT');
return response.data.price;
}
const getWazirxBtcPrice = async () => {
// WazirX only returns all symbol data
const response = await axios.get('https://api.wazirx.com/api/v2/market-status');
// you need to loop through all of them, until you find the desired pair
for (let market of response.data.markets) {
// in your case `btc` and `usdt` (mind the lowecase letters)
if (market.baseMarket === 'btc' && market.quoteMarket === 'usdt') {
// mind that WazirX only returns price in full integer,
// it does not return floating numbers
return market.last;
}
}
// did not find the combination
return null;
}
const run = async () => {
const binancePrice = await getBinanceBtcPrice();
const wazirxPrice = await getWazirxBtcPrice();
console.log(binancePrice);
console.log(wazirxPrice);
}
run();

How can I make a request to multiple URLs and parse the results from each page?

I'm using the popular npm package cheerio with request to retrieve some table data.
Whilst I can retrieve and parse the table from a single page easily, I'd like to loop over / process multiple pages.
I have tried wrapping inside loops / various utilities offers by the async package but can't figure this one out. In most cases, node runs out of memory.
current code:
const cheerio = require('cheerio');
const axios = require("axios");
var url = someUrl;
const getData = async url => {
try {
const response = await axios.get(url);
const data = response.data;
const $ = cheerio.load(data);
const announcement = $(`#someId`).each(function(i, elm) {
console.log($(this).text())
})
} catch (error) {
console.log(error);
}
};
getData(url); //<--- Would like to give an array here to fetch from multiple urls / pages
My current approach, after trying loops, is to wrap this inside another function with a callback param. However no success yet and is getting quite messy.
What is the best way to feed an array to this function?
Assuming you want to do them one at a time:
; (async() => {
for(let url of urls){
await getData(url)
}
})()
Have you tried using Promise.all (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all)?
For loops are usually a bad idea when dealing with asynchronous calls. It depends how many calls you want to make but I believe this could be enough. I would use an array of promises that fetch the data and map over the results to do the parsing.

Categories

Resources