I'm trying to create multiple PDF file using pdfkit, I have an array of users and I create a report for each one, the createTable()Function below returns a Buffer that I send to archiver to zip, once complete the zip file is sent for download to the front end.
My issue is that for some reason, Archiver will sometimes throw a QUEUECLOSED error, if I run the function too many time, sometimes I can run it 10 times and the 11th time I'll get an error and sometimes I get an error after the second time, each time i run it the data is the same and nothing else changed.
Any help is greatly appreciated.
users.forEach(async (worker, index) => {
createTable(date, worker.associateName, action, worker.email, worker.id, excahngeRate).then(resp => {
archive.append(resp, { name: worker.associateName + '.pdf' })
if (index === users.length - 1 === true) {//make sure its the last item in array
archive.pipe(output)
archive.finalize();
}
}).catch(err => {
console.log(err)
})
});
You finalize too soon. The createTable for the last user might not be the last to finish. You should add all to archive and once everything is done, finalize it.
// Use map to get an array of promises
const promises = users.map(async (worker, index) => {
return createTable(date, worker.associateName, action, worker.email, worker.id, excahngeRate).then(resp => {
archive.append(resp, { name: worker.associateName + '.pdf' })
}).catch(err => {
console.log(err)
})
});
// Wait for all promises to finish.
Promise.all(promises).then(()=>{
archive.pipe(output)
archive.finalize();
});
In your current code, you could console.log just before your IF statement, and log the index of the completed createTable, you'll see they do not finish in order.
Related
I am using react-native and I want to get specific data so I used a query, the problem is that it didn't work, but if I remove the where and do an if statement it works. How can I fix this ?
This is my implementation:
let query = firestore().collection('conversations');
query = query.where('users', 'array-contains', myId);
// query = query.where('postId', '==', postId);
query
.orderBy('timestamp', 'desc')
.limit(limit)
.get()
.then((snapshot) => {
const offersObjects = {};
if (snapshot.docs.length) {
snapshot.docs.forEach((doc) => {
if (postId === doc.data().postId) {
offersObjects[doc.id] = { ...doc.data(), pendingMessages: [] };
}
});
dispatch({
type: chatTypes.SET_OFFERS,
payload: { offers: offersObjects },
});
}
})
Where the code is commented is where this query doesn't work which is weird since the one above it works fine. How can I fix this ?
If you have more than one condition on a query, it may need a composite index that is not automatically created. If that is the case, executing the query (e.g. by calling get() on it) will raise an error, but you'r not hancling errors. I recommend adding a catch clause after your then and logging the error.
If the problem is caused by a missing index, the error message contains a URL that opens the Firebase console on the page to create the exact index that is needed, with all fields already filled in. Create the index, wait for it to be completely created, and run the query again.
I am connecting a REST api from React Native app. I have Json response with filename objects with different names but all the objects have same variables: filename, message, and display.
Number of objects changes with each request to API (REST), the names of objects in response are different depending on requests. But the variables in each object are same as above.
The information I need from this response is only filename text, but it will be acceptable if I get list of objects so I can read through the messages from errors.
The image shows how my objects look like.
This is my fetch request :
const getGists = async () => {
await axios
.get(`https://api.github.com/gists/public?per_page=30`)
.then((r) => {
let n;
for (n = 0; n < 30; n++) {
console.log(r.data[n].files.filename);
// console.log("____________________");
// console.log(r.data[n].owner.avatar_url);
// console.log("____________________");
// console.log(JSON.stringify(r.data[n].files));
}
})
.catch((e) => {
console.log("ERROR", e);
});
};
how is possible to get every filename from these requests even if object name is not the same in each iteration . Thanks for help
Working with the result of the API calls and some higher-order functions, this will work fine:
const getGists = async () => {
await axios
.get(`https://api.github.com/gists/public?per_page=30`)
.then((response) => {
const myDesireResult = response.data.reduce((acc, item) => {
const files = Object.values(item.files);
if (files.length > 1) {
files.forEach((file) => acc.push(file.filename));
} else {
acc.push(files[0].filename);
}
return acc;
}, []);
console.log(myDesireResult);
})
.catch((e) => {
console.log("ERROR", e);
});
};
Explanation:
in the then block, can get the API call result with result.data
with reduce function, looping through the data will start.
since the object in the files has different names, we can get the files with Object.values() easily.
Some of the files contain several items and most of them have just one item. so with checking the length of the file we can do proper action. if the files have more than one element, with another simple lop, we can traverse this file array easily.
Check the working example on codesandbox
My NestJS application has a simple purpose to:
Loop through an array of large files ( 29 files where each one have about 12k to 70k lines)
Read a file line by line and parse it
Insert (each line) into my MongoDB collection
The most important part of my code consist of:
for(let file of FILES){
result = await this.processFile(file);
resultInsert += result;
}
and the function processFile()
async processFile(fileName: string): Promise<number> {
count = 0;
return new Promise((resolve, reject) => {
let s = fs
.createReadStream(BASE_PATH + fileName, {encoding: 'latin1'})
.pipe(es.split())
.pipe(
es
.mapSync(async (line: string) => {
count++;
console.log(line);
let line_splited = line.split("#");
let user = {
name: line_splited[0],
age: line_splited[1],
address: line_splited[2],
job: line_splited[3],
country: line_splited[4]
}
await this.userModel.updateOne(
user,
user,
{ upsert: true }
);
})
.on('end', () => {
resolve(count);
})
.on('error', err => {
reject(err);
})
);
});
}
The main problem is by the interaction of the ~9th file, I have a memory failure: Allocation failed - JavaScript heap out of memory.
I saw that my problem is similar to Parsing huge logfiles in Node.js - read in line-by-line but the code still managed to fail.
I suspect the fact that I am opening a file, reading it and when I open another file, I am still inserting the previous one can cause the problem but I don't know how to handle it.
I could make it work by changing the updateOne() to insertMany().
Quick explanation: instead of inserting one by one, we would be inserting by 100k.
So I just created an array of user and when it reached 100k documents, we would insert with insertMany()
I created a function to communicate with the firestore database.
First, check if there is something in the relation. If not then add something.
If something already exists then use the data and then delete the entry in the queried relation. But you have to add that in the function(else section). And now the question arises what happens when two users simultaneously perform the function.
Is there a way to put the second user in a queue while the first user is done with the request?
let ref = db.collection('relation1').doc('test').collection('user');
var checkForAdd = ref.get().then(snapshot => {
if(snapshot.size < 1){
db.collection('relation1').doc('test').collection('user').add({
user: 'Test',
createdAt: Date.now()
}).catch(err =>{
console.log(err)
})
}
Cloud Firestore supports atomic operations for reading and writing data. In a set of atomic operations, either all of the operations succeed, or none of them are applied.
https://firebase.google.com/docs/firestore/manage-data/transactions
// Create a reference to the user doc you want to create if it doesn't exist.
const userCollectionRef = db.collection('relation1').doc('test').collection('user');
const userDocRef = userCollectionRef.doc('documentID');
return db.runTransaction(transaction => {
// This code may get re-run multiple times if there are conflicts.
return transaction.get(userDocRef).then(userDoc => {
if (userDoc.exists) {
// If something already exists then use the data and
// then delete the entry in the queried relation.
} else {
transaction.update(userDocRef, {
user: 'Test',
createdAt: Date.now()
});
}
});
}).then(() => {
console.log("Transaction successfully committed!");
}).catch(error => {
console.log("Transaction failed: ", error);
});
I'm trying to get the download url for multiple images, then trigger a change in my app. But... if one of those images doesn't exist for whatever reason, everything fails silently.
Here's the code:
const promises = [];
snapshot.forEach(childSnapshot => {
const child = childSnapshot.val();
const promise = firebase.storage()
.ref(child.songImagePath)
.getDownloadURL()
.catch(err => {
console.log('caught', err);
return "";
})
.then(imageURL => {
return imageURL;
});
promises.push(promise);
});
Promise.all(promises)
.catch(err => {
console.log('caught', err);
})
.then(urls => {
...do something with urls array
});
I'm using child.songImagePath in my database to store the image's location in storage. If ALL paths for ALL images have images, everything works perfectly.
BUT if an upload went awry or for some reason there's no image in the storage location, it fails silently. None of my catches fire. And Promise.all is never resolved.
What's going on here? Is there a way to check for a file's existence before calling getDownloadURL?
EDIT: As #mjr points out, in the documentation they've formatted their error callback slightly differently than I have. This also seems to never fire an error, though:
.then(
imageURL => {
return imageURL;
},
err => {
console.log('caught', err);
return "";
}
);
Firebase Storage JS dev here.
I ran your code with minor changes[1] in Chrome and React Native, and didn't see that behavior.
I see Promise.all always resolving (never failing), with an empty string in the array for invalid files. This is because your .catch handler for getDownloadURL returns an empty string.
For further troubleshooting, it would be useful to know:
version of the firebase JS library you are using
the browser/environment and version
network logs, for example from the network panel in Chrome's dev tools, or similar for other browsers
The firebase-talk Google Group tends to be a better place for open-ended troubleshooting with more back-and-forth.
[1] For reference, here's my code:
const promises = [];
// Swap out the array to test different scenarios
// None of the files exist.
//const arr = ['nofile1', 'nofile2', 'nofile3'];
// All of the files exist.
const arr = ['legitfile1', 'legitfile2', 'legitfile3'];
// Some, but not all, of the files exist.
//const arr = ['legitfile1', 'nofile2', 'nofile3'];
arr.forEach(val => {
const promise = firebase.storage()
.ref(val)
.getDownloadURL()
.catch(err => {
// This runs for nonexistent files
console.log('caught', err);
return "";
})
.then(imageURL => {
// This runs for existing files
return imageURL;
});
promises.push(promise);
});
Promise.all(promises)
.catch(err => {
// This never runs
console.log('caught', err);
})
.then(urls => {
// This always runs
console.log('urls', urls);
});