Develop Reference

How to parse json if Key'name dynamicly changes each time node js

I receive JSON data from the service, but the keys change in the data with each request, below I will give an example in three versions.
Exmaple 1:
{
"trackingPayloads": {
"Rltyn4gLRIWRKj9kS0YpWXytG81GZwcPWjEE7f31ALlq": "{"title":"Red Shoes","index":3,"id":"17777","type":"category"}',
"ywtA6OyM0hzVZZvnUjxoxJDI1Er9ArfNr8XKyi1D5Zzk": "{"title":"White Shoes","index":3,"id":"17777","type":"category"}',
}
}
Example 2:
{
"trackingPayloads": {
"36tW7DqZ3H9KKBEAumZmowmUwmDRmVCjQgv5zi9GM3Kz": "{"title":"Red Shoes","index":3,"id":"17777","type":"category"}',
"OgtE51n3YtvrVXWLFjPmpnRt2k5DExF7ovxmBTZrZ6wV": "{"title":"White Shoes","index":3,"id":"17777","type":"category"}',
}
}
Example 3:
{
"trackingPayloads": {
"k2toY29glt2JEp9Wi1X5M7ocno0E0mS4JQVyDuGyQ2rM": "{"title":"Red Shoes","index":3,"id":"17777","type":"category"}'",
"5ef2ec3c3573eebecc9690b69619ec7b9c93b609": "{"title":"White Shoes","index":3,"id":"17777","type":"category"}',
}
}
As you can see, the data included in the keys does not change since I am requesting the same information, but the key will change with each request.
Please help, what are the options to get the data Title, Index and any other content in these keys using node js?
Only one option came to my mind - to rename the keys upon receipt in 1,2,3 ... and then get data from them, but this needs to be done dynamically, since about 120 requests per minute are made, you need to get this data quickly, there are no options to save it to a file (I didn’t understand how)
UPDATE, added my code.
I am attaching an example of my code, the idea is to eventually get the data I need from the right keys from trackingPayloads, please help with the code <3
const AwaitAPIResponse = await ProductAPI(product_sku);
const $ = cheerio.load(AwaitAPIResponse);
const JSONDATA = [];
$('pre').each(function() {
JSONDATA.push($(this).text());
});
const ProductJson = JSON.parse(JSONDATA[0]) // this is where I get all the data
const MainJson = ProductJson["trackingPayloads"] // here I go to the trackingPayloads you saw above
How can I get the data I need?

You can use Object.keys() to get all the different keys of an object and use a loop to go through them.
Therefore, you can rework this code in such a way that each of the values is stored as an element in an array, maybe makes the data easier to work with:
const convert = object => {
const ret = []
for (const key of Object.keys(object)) {
ret.push(object[key])
}
return ret
}
This will give you following result for your use case:
[{"title":"Red Shoes","index":3,"id":"17777","type":"category"},
{"title":"Red Shoes","index":3,"id":"17777","type":"category"}]
The way you'd call this is as follows:
const some_parsed_json = {
"k2toY29glt2JEp9Wi1X5M7ocno0E0mS4JQVyDuGyQ2rM": {
title:"Red Shoes",
index:3,
id:"17777",
type:"category"
},
"5ef2ec3c3573eebecc9690b69619ec7b9c93b609": {
title:"Red Shoes",
index:3,
id:"17777",
type:"category"
}
}
const json_object_values = convertor(some_parsed_json)

If you don't car about the key you could use Object.values on the received object to get the values
Object.values(payload)
// With your example it will return:
// [{"title":"Red Shoes","index":3,"id":"17777","type":"category"},
// {"title":"Red Shoes","index":3,"id":"17777","type":"category"}]
or in a more complete example
async function getParsedValues() {
const responseString = await service.getData(); // '{"trackingPayloads":{"Rltyn4gLRIWRKj9kS0YpWXytG81GZwcPWjEE7f31ALlq":{"title":"Red Shoes","index":3,"id":"17777","type":"category"},"ywtA6OyM0hzVZZvnUjxoxJDI1Er9ArfNr8XKyi1D5Zzk":{"title":"White Shoes","index":3,"id":"17777","type":"category"}}}'
const parsedResponse = JSON.parse(responseString); // { trackingPayloads: { Rltyn4gLRIWRKj9kS0YpWXytG81GZwcPWjEE7f31ALlq: { title:'RedShoes', index: 3, id: '17777', type: 'category' }, ywtA6OyM0hzVZZvnUjxoxJDI1Er9ArfNr8XKyi1D5Zzk:{title:'WhiteShoes', index: 3, id: '17777', type: 'category' } }}
const values = Object.values(parsedResponse); // [{"title":"Red Shoes","index":3,"id":"17777","type":"category"}, {title:'WhiteShoes', index: 3, id: '17777', type: 'category' }]
return values;
}

Fetch JSON API nested data with Async function

complete noob here just trying to extract nested data from a JSON API.
Let's say that I want to retrieve and store in a variable the field received "amount" of the item number 3 (highlighted in the screenshot), what's the correct syntax? (The item number 3 is just an example, when finally found out how to fetch the data, my number 3 would become a variable and I will insert it in a loop because I want to extract also number 4,5,6 etc... If you are so kind, could you please give me help providing instead of a fixed number a variable that I can change within a loop?)
When the correct data is retrieved I want the received amount to be shown in the Span with ID="lat".
I am using the below syntax but it's not working at all:
<span id="lat"></span>
<script>
const api_url = 'https://MYAPIURL.com';
async function getAmount() {
const response = await fetch(api_url);
const data1 = await response.json();
const { data.history_list[3].receives[0].amount } } = data1;
document.getElementById('lat').textContent = data1;
}
getAddress();
</script>
Many appreciate your help, sers! Thank you :)

Try creating another variable to store the amount. Your syntax right now is assigning the value of the whole json object you received to the path of the received data object amount. This means you're nesting the received object back into the received object at the amount position.
You're also declaring a const here so JavaScript will likely think you're trying to 'destructure' the provided object except the syntax isn't exactly right for that so I imagine that might be causing problems too.
<script>
const api_url = 'https://MYAPIURL.com';
async function getAmount() {
const response = await fetch(api_url);
const data1 = await response.json();
const dataAmount= data1.history_list[3].receives[0].amount //changed line
document.getElementById('lat').textContent = dataAmount; //changed line to assign the textcontent to new variable
}
getAddress();
</script>

You can achieve it by this simple way dynamically.
Working Demo :
const data = {
history_list: [{
receives: [{
amount: 10
}]
}, {
receives: [{
amount: 20
}]
}, {
receives: [{
amount: 30
}]
}, {
receives: [{
amount: 40
}]
}]
};
let amount = 0;
data.history_list.forEach((obj) => {
obj.receives.forEach((amountObj) => {
amount += amountObj.amount;
});
});
document.getElementById('lat').innerHTML = `Total Amount: ${amount}`
<div id="lat"></div>

Knexjs how to handle multiple updates

I´m quite unsure on how to handle multiple updates / inserts in knex and return whatever it was successfull on the end or not.
I´m passing an array through req.body loop through it and trigger actions based on informations inside the array.
Example:
const data = [...req.body]
for(let i = 0; i < data.length; i++) {
data[i].totals.length
for(let y = 0; y < data[i].totals.length; y++) {
if(data[i].totals[y].info === "Holiday") {
calcHoliday(data[i].totals[y].total, data[i].id)
} else if(data[i].totals[y].info === "ZA") {
calcZA(data[i].totals[y].total, data[i].id)
}
}
calcOvertime(data[i].totalSum, data[i].id)
if(i === data.length -1) {
res.json("Success")
}
}
The Array I´m passing in looks like this:
[
{
"id": 1,
"totals": [
{
"info": "Holiday",
"total": 4
}
]
},
{
"id": 1,
"totals": [
{
"info": "Holiday",
"total": 4
}
]
}
]
Function Example which gets called in for loop:
const calcHoliday = (hours, userid) => {
knex.transaction(trx => {
trx.insert({
created_at: convertedTime,
info: "Booking Holiday - Hours: " + hours,
statuscode: 200
}).into("logs")
.then(() => {
return trx("hours")
.decrement("holiday_hours", hours)
}).then(trx.commit)
.catch(trx.rollback)
}).then(() => console.log("WORKED"))
.catch(err => console.log(err))
}
This is working perfectly fine but I can´t figure out how to gather the results from each table update in order to respond if everything worked or an error appeared. If I call e.g. after one calcHoliday call .then(resp => res.json(resp) I receive only the response from the first operation.
In short I need a way on how to res.json if everything succeeded or an error appeared somewhere.
Thanks in advance!

TLDR;
Turning your insert calls into an array of promises and then using await and a Promise.all() / Promise.allSettled() structure might solve this problem, but there are some UX decisions to make on what to rollback and how to return errors.
Error Handling Choices:
Any error --> all insertions in all loop iterations should be rolled back
Do you want partial success? The way the code is written now, rollback only applies to items in one function call. If one of the hour-decrement calls fails, it will roll back one log insert, but not any that succeeded for previous data in the loop. If you want the whole dataset to rollback, you'd need to pass the txn through each function call or do a bulk insert of all of your rows in one function call, which might be nice for performance reasons anyway depending on the use case.
Partial success --> commits successes, rolls back single loop iterations that fail, sends detailed list of errors and successes
You'd want to use Promise.allSettled(), which aggregates the successes and errors as an array from all promises in the loop.
Partial success --> commits the successes, rolls back single loop iterations that fail, sends just one error
Opinion: This can be a misleading UX unless the error is "some of the insertions were unsuccessful" and the endpoint is idempotent
This looks closest to what you're describing you want. If this is the case, you'd want to use Promise.all(), which throws an error as soon as one promise in the array errors.
Example Implementation:
Since the original code is incomplete, this is a loose, incomplete example of what option 2/3 might look like. This could easily be transformed into option 1.
First, it might help to modify all of your functions with asynchronous calls to be fulfillable as promises. Async/await helps avoid .then() trees that are hard to reason about.
const calcHoliday = async (hours, userid) => {
try {
const result = await knex.transaction(async(trx) => {
await trx.insert({
created_at: convertedTime,
info: "Booking Holiday - Hours: " + hours,
statuscode: 200
}).into("logs")
return trx("hours").decrement("holiday_hours", hours)
}
return result
} catch(err) {
console.log("It didn't work.")
throw new Error(`Error: Failure to insert for user ${userid}:`, err)
}
}
Here are some utilities to get the data transformed, and to get the appropriate unfulfilled promise to supply to the map in Promise.all/allSettled.
/*
Here's an example of how you might transform the original data with maps in order to avoid nested for-loops:
[
{ id: 1, info: 'Holiday', total: 4 },
{ id: 1, info: 'Holiday', total: 4 }
]
*/
const flatData = data.map(item => {
return item.totals.map(total => ({
id: item.id,
...total
}))
}).flat()
// Returns the appropriate promise based on data
const getTotalsPromises = (row) => {
const {info, id, total} = row
if(info === "Holiday") {
return calcHoliday(total, id)
} else if(info === "ZA") {
return calcZA(total, id)
}
}
const getcalcOvertimePromises = (rowInData) = {
// work left to reader
return calcOvertime(rowInData.correctData, rowInData.otherData)
}
If you want option 2:
// Replaces the loop
// Fulfills *all* the promises, creating an array of errors and successes
const responses = await Promise.allSettled([
...flatData.map(getTotalsPromises),
...data.map(getCalcOvertimePromises)
])
// insert loop here to do something with errors if you want
res.send(responses)
OR Option 3
Create an array of all of the promises you want to run, run them, and process up to one error.
// Replaces the loop
// Runs the promises and waits for them all to finish or the first error.
try {
const responses = await Promise.all([
...flatData.map(getTotalsPromises),
...data.map(getCalcOvertimePromises)
])
res.send(responses)
} catch(err){
// Reached if one of the rows errors
res.send(err)
}
Docs:
Promise.allSettled
Promise.all

Problem with React Query's Infinite Query, using Edamam API

I currently have some issues trying to add the infinite query feature to a recipes app I'm working on using Edamam API.
All the examples I have looked for (even React Query's documentation) implement the infinite scroll using a page/cursor number system... I understand this is the ideal way, but... Edamam API doesn't work this way with paginated queries.
Instead, the API has the following structure for each recipe query we look for (let's assume we are searching for "chicken", this would be the JSON structure):
from: 1,
to: 20,
count: 10000,
_links: {
next: {
href: "https://api.edamam.com/api/recipes/v2?q=chicken&app_key=APIKEYc&_cont=CHcVQBtNNQphDmgVQntAEX4BYldtBAAGRmxGC2ERYVJ2BwoVX3cVBWQSY1EhBQcGEmNHVmMTYFEgDQQCFTNJBGQUMQZxVhFqX3cWQT1OcV9xBB8VADQWVhFCPwoxXVZEITQeVDcBaR4-SQ%3D%3D&type=public&app_id=APPID"
title: "Next Page"
}
},
hits: [{}] ... (This is where the actual recipes are)
As you can see, there is no numbering system for paginated queries, instead, it's a whole URL and it's giving me a hard time since I'm also new to React Query.
I tried the following, but it just fetches the same data over and over again as I reach the bottom of the page:
const getRecipes = async ({ pageParam }) => {
try {
const path = pageParam
? pageParam
: `https://api.edamam.com/api/recipes/v2?q=${query}&app_id=${process.env.REACT_APP_APP_ID}&app_key=${process.env.REACT_APP_API_KEY}&type=public`;
const response = await axios.get(path);
return response.data;
} catch (error) {
console.log(error);
}
const { ref, inView } = useInView();
useEffect(() => {
inView && fetchNextPage();
}, [inView]);
const {
data,
isFetching,
isFetchingNextPage,
error,
status,
hasNextPage,
fetchNextPage,
} = useInfiniteQuery(
["recipes", query],
({ pageParam = "" }) => getRecipes(pageParam),
{
getNextPageParam: (lastPage) => lastPage._links.next.href,
}
);
Since the next page param is a whole URL, I just say that IF there is a pageParam, then use that URL for the request, if not, then do a normal request using the query value the user is searching for.
Please help!

Since the next page param is a whole URL, I just say that IF there is a pageParam, then use that URL for the request, if not, then do a normal request using the query value the user is searching for.
I'd say that this is the correct approach. The only code issue I can see in your example is that you destruct page param, and then pass the page param string to getRecipes:
({ pageParam = "" }) => getRecipes(pageParam),
but in getRecipes, you expect an object to come in (which you again destructure):
const getRecipes = async ({ pageParam }) => {
You can fix that by either changing the call side, or the function syntax, and then it should work.

In rxjs, how do I chain mapping through arrays of data received from different API's?

I'm calling an API and receiving an array of results, I'm checking for pagination and if more pages exist I call the next page, repeat until no more pages.
For each array of results, I call another endpoint and do the exact same thing: I receive an array of results, check for another page and call endpoint again. Wash, rinse repeat.
For instance:
I want to grab a list of countries that might be a paginated response, then for each country I want to grab a list of cities, which might also be paginated. And for each city I execute a set of transformations and then store in a database.
I already tried this, but got stuck:
const grabCountries = Observable.create(async (observer) => {
const url = 'http://api.com/countries'
let cursor = url
do {
const results = fetch(cursor)
// results = {
// data: [ 'Canada', 'France', 'Spain' ],
// next: '47asd8f76358df8f4058898fd8fab'
// }
results.data.forEach(country => { observer.next(country) })
cursor = results.next ? `${url}/${results.next}` : undefined
} while(cursor)
})
const getCities = {
next: (country) => {
const url = 'http://api.com/cities'
let cursor = url
do {
const results = fetch(cursor)
// results = {
// data: [
// 'Montreal', 'Toronto',
// 'Paris', 'Marseilles',
// 'Barcelona', 'Madrid'
// ],
// next: '89ghjg98nd8g8sdfg98gs9h868hfoig'
// }
results.data.forEach(city => {
`**** What do I do here?? ****`
})
cursor = results.next ? `${url}/${results.next}` : undefined
} while(cursor)
}
}
I tried a few approaches:
Making a subject (sometimes I'll need to do parallel processed base on the results of 'grabCountries'. For example I may want to store the countries in a DB in parallel with grabbing the Cities.)
const intermediateSubject = new Subject()
intermediateSubject.subscribe(storeCountriesInDatabase)
intermediateSubject.subscribe(getCities)
I also tried piping and mapping, but it seems like it's basically the same thing.
As I was writing this I thought of this solution and it seems to be working fine, I would just like to know if I'm making this too complicated. There might be cases where I need to make more that just a few API calls in a row. (Imagine, Countries => States => Cities => Bakeries => Reviews => Comments => Replies) So this weird mapping over another observer callback pattern might get nasty.
So this is what I have now basically:
// grabCountries stays the same as above, but the rest is as follows:
const grabCities = (country) =>
Observable.create(async (observer) => {
const url = `http://api.com/${country}/cities`
let cursor = url
do {
const results = fetch(cursor)
// results = {
// data: [
// 'Montreal', 'Toronto',
// 'Paris', 'Marseilles',
// 'Barcelona', 'Madrid'
// ],
// next: '89ghjg98nd8g8sdfg98gs9h868hfoig'
// }
results.data.forEach(city => {
observer.next(city)
})
cursor = results.next ? `${url}/${results.next}` : undefined
} while (cursor)
})
const multiCaster = new Subject()
grabCountries.subscribe(multiCaster)
multiCaster.pipe(map((country) => {
grabCities(country).pipe(map(saveCityToDB)).subscribe()
})).subscribe()
multiCaster.pipe(map(saveCountryToDB)).subscribe()
tl;dr - I call an API that receives a paginated set of results in an array and I need to map through each item and call another api that receives another paginated set of results, each set also in an array.
Is nesting one observable inside another and mapping through the results via 'callApiForCountries.pipe(map(forEachCountryCallApiForCities))' the best method or do you have any other recommendations?

Here's the code that should work with sequential crawling of next url.
You start with a {next:url} until res.next is not available.
of({next:http://api.com/cities}).pipe(
expand(res=>results.next ? `${url}/${results.next}` : undefined
takeWhile(res=>res.next!==undefined)
).subscribe()

OK, so I have spent a lot of brain power on this and have come up with two solutions that seem to be working.
const nestedFlow = () => {
fetchAccountIDs.pipe(map(accountIds => {
getAccountPostIDs(accountIds) // Has the do loop for paging inside
.pipe(
map(fetchPostDetails),
map(mapToDBFormat),
map(storeInDB)
).subscribe()
})).subscribe()
}
const expandedflow = () => {
fetchAccountIDs.subscribe((accountId) => {
// accountId { accountId: '345367geg55sy'}
getAccountPostIDs(accountId).pipe(
expand((results) => {
/*
results : {
postIDs: [
131424234,
247345345,
],
cursor: '374fg8v0ggfgt94',
}
*/
const { postIDs, cursor } = results
if (cursor) return getAccountPostIDs({...accountId, cursor})
return { postIDs, cursor }
}),
takeWhile(hasCursor, true), // recurs until cursor is undefined
concatMap(data => data.postIDs),
map(data => ({ post_id: data })),
map(fetchPostDetails),
map(mapToDBFormat),
map(storeInDB)
).subscribe()
})
}
Both seem to be working with similar performance. I read some where that leaving the data flow is a bad practice and you should pipe everything, but I don't know how to eliminate the first exit in the 'expandedFlow' because the 'expand' needs to call back an observable, but maybe it can be done.
Now I just have to solve the race condition issues from the time the 'complete' is called in getAccountPostIDs the the last record is stored in the DB. Currently in my test, the observer.complete is finishing before 3 of the upsert actions.
Any comments are appreciated and I hope this helps someone out in the future.

What you need is the expand operator. It behaves recursively so it fits the idea of having paginated results.