Develop Reference

How to use dataloader?

Im trying to figure this out.
I want to get all my users from my database, cache them
and then when making a new request I want to get those that Ive cached + new ones that have been created.
So far:
const batchUsers = async ({ user }) => {
const users = await user.findAll({});
return users;
};
const apolloServer = new ApolloServer({
schema,
playground: true,
context: {
userLoader: new DataLoader(() => batchUsers(db)),// not sending keys since Im after all users
},
});
my resolver:
users: async (obj, args, context, info) => {
return context.userLoader.load();
}
load method requiers a parameter but in this case I dont want to have a specific user I want all of them.
I dont understand how to implement this can someone please explain.

If you're trying to just load all records, then there's not much of a point in utilizing DataLoader to begin in. The purpose behind DataLoader is to batch multiple calls like load(7) and load(22) into a single call that's then executed against your data source. If you need to get all users, then you should just call user.findAll directly.
Also, if you do end up using DataLoader, make sure you pass in a function, not an object as your context. The function will be ran on each request, which will ensure you're using a fresh instance of DataLoader instead of one with a stale cache.
context: () => ({
userLoader: new DataLoader(async (ids) => {
const users = await User.findAll({
where: { id: ids }
})
// Note that we need to map over the original ids instead of
// just returning the results of User.findAll because the
// length of the returned array needs to match the length of the ids
return ids.map(id => users.find(user => user.id === id) || null)
}),
}),
Note that you could also return an instance of an error instead of null inside the array if you want load to reject.

Took me a while but I got this working:
const batchUsers = async (keys, { user }) => {
const users = await user.findAll({
raw: true,
where: {
Id: {
// #ts-ignore
// eslint-disable-next-line no-undef
[op.in]: keys,
},
},
});
const gs = _.groupBy(users, 'Id');
return keys.map(k => gs[k] || []);
};
const apolloServer = new ApolloServer({
schema,
playground: true,
context: () => ({
userLoader: new DataLoader(keys => batchUsers(keys, db)),
}),
});
resolver:
user: {
myUsers: ({ Id }, args, { userLoader }) => {
return userLoader.load(Id);
},
},
playground:
{users
{Id
myUsers
{Id}}
}
playground explained:
users basically fetches all users and then myusers does the same thing by inhereting the id from the first call.
I think I choose a horrible example here since I did not see any gains in performence by this. I did see however that the query turned into:
SELECT ... FROM User WhERE ID IN(...)

How to handle multiple promises at once

I am creating a program that...
1. Detects all of the drives on any given system.
2. Scans those drives for files of specific file types. For example, it may search all of the drives for any jpeg, png, and svg files.
3. The results are then stored in a JSON file in the following desired format.
{
"C:": {
"jpeg": [
...
{
"path": "C:\\Users\\John\\Pictures\\example.jpeg",
"name": "example",
"type": "jpeg",
"size": 86016
},
...
],
"png": [],
"svg": []
},
...
}
The code...
async function scan(path, exts) {
try {
const stats = await fsp.stat(path)
if (stats.isDirectory()) {
const
childPaths = await fsp.readdir(path),
promises = childPaths.map(
childPath => scan(join(path, childPath), exts)
),
results = await Promise.all(promises)
// Likely needs to change.
return [].concat(...results)
} else if (stats.isFile()) {
const fileExt = extname(path).replace('.', '')
if (exts.includes(fileExt)){
// Likely needs to change.
return {
"path": path,
"name": basename(path, fileExt).slice(0, -1),
"type": fileExt,
"size": stats.size
}
}
}
return []
}
catch (error) {
return []
}
}
const results = await Promise.all(
config.drives.map(drive => scan(drive, exts))
)
console.log(results) // [ Array(140), Array(0), ... ]
// And I would like to do something like the following...
for (const drive of results) {
const
root = parse(path).root,
fileExt = extname(path).replace('.', '')
data[root][fileExt] = []
}
await fsp.writeFile('./data.json', JSON.stringify(config, null, 2))
The global results is of course divided into individual arrays that correspond to each drive. But currently it combines all of the objects into one giant array despite their corresponding file types. There is also currently no way for me to know which array belongs to each drive, especially if the drive's array does not contain any items that I can parse to retrieve the root directory.
I can obviously map or loop thru the global results again, and then sort everything out, as illustrated below, but it would be a lot cleaner to have scan() handle everything from the get go.
// Initiate scan sequence.
async function initiateScan(exts) {
let
[config, data] = await Promise.all([
readJson('./config.json'),
readJson('./data.json')
]),
results = await Promise.all(
// config.drives.map(drive => scan(drive, exts))
['K:', 'D:'].map(drive => scan(drive, exts))
)
for (const drive of results) {
let root = false
for (const [i, file] of drive.entries()) {
if (!root) root = parse(file.path).root.slice(0,-1)
if (!data[root][file.type] || !i) data[root][file.type] = []
data[root][file.type].push(file)
}
}
await fsp.writeFile('./data.json', JSON.stringify(config, null, 2))
}
Due to my lack of experience with asynchronicity and objects in general, I am not quite sure how to best handle the data in map( ... )/scan. I am really not even sure how to best structure the output of scan() so that the structure of the global results is easily manipulable.
Any help would be greatly appreciated.

Mutating an outer object as asynchronously-derived results arrive is not particularly clean, however it can be done fairly simply and safely as follows:
(async function(exts, results) { // async IIFE wrapper
async function scan(path) { // lightly modified version of scan() from the question.
try {
const stats = await fsp.stat(path);
if (stats.isDirectory()) {
const childPaths = await fsp.readdir(path);
const promises = childPaths.map(childPath => scan(join(path, childPath)));
return Promise.all(promises);
} else if (stats.isFile()) {
const fileExt = extname(path).replace('.', '');
if (results[path] && results[path][fileExt]) {
results[path][fileExt].push({
'path': path,
'name': basename(path, fileExt).slice(0, -1),
'type': fileExt,
'size': stats.size
});
}
}
}
catch (error) {
console.log(error);
// swallow error by not rethrowing
}
}
await Promise.all(config.drives.map(path => {
// Synchronously seed the results object with the required data structure
results[path] = {};
for (fileExt of exts) {
results[path][fileExt] = []; // array will populated with data, or remain empty if no qualifying data is found.
}
// Asynchronously populate the results[path] object, and return Promise to the .map() callback
return scan(path);
}));
console.log(results);
// Here: whatever else you want to do with the results.
})(exts, {}); // pass `exts` and an empty results object to the IIFE function.
The results object is synchronously seeded with empty data structures, which are then populated asynchronously.
Everything is wrapped in an async Immediately Invoked Function Expression (IIFE), thus:
avoiding the global namespace (if not already avoided)
ensuring availabillty of await (if not already available)
making a safe closure for the results object.

This still needs some work, and it is iterating through the generated files collection a second time.
// This should get you an object with one property per drive
const results = Object.fromEntries(
(await Promise.all(
config.drives.map(async drive => [drive, await scan(drive, exts)])
)
)
.map(
([drive, files]) => [
drive,
// we reduce each drive's file array to an object with
// one property per file extension
files.reduce(
(acc, file) => {
acc[file.type].push(file)
return acc
},
Object.fromEntries(exts.map(ext => [ext, []]))
)
]
)
)
nodejs supports Object.fromEntries from version 12.0.0, so if you can guarantee your application will always be run in that version or a later one, Object.fromEntries should be fine here.

You can use the glob npm library to get all of the filenames and then just transform that array to your object like this:
import {basename, extname} from 'path';
import {stat} from 'fs/promises'; // Or whichever library you use to promisify fs
import * as glob from "glob";
function searchForFiles() {
return new Promise((resolve, reject) => glob(
"/**/*.{jpeg,jpg,png,svg}", // The files to search for and where
{ silent: true, strict: false}, // No error when eg. something cannot be accessed
(err, files) => err ? reject() : resolve(files)
));
}
async function getFileObject() {
const fileNames = await searchForFiles(); // An array containing all file names (eg. ['D:\\my\path\to\file.jpeg', 'C:\\otherfile.svg'])
// An array containing all objects describing your file
const fileObjects = await Promise.all(fileNames.map(async filename => ({
path: filename,
name: basename(path, fileExt).slice(0, -1),
type: extname(path).replace('.', ''),
size: stat(path).size,
drive: `${filename.split(':\\')[0]}:`
})));
// Create your actual object
return fileObjects.reduce((result, {path, name, type, size, drive}) => {
if (!result[drive]) { // create eg. { C: {} } if it does not already exist
result.drive = {};
}
if (!result[drive][type]) { // create eg. {C: { jpeg: [] }} if it does not already exist
result[drive][type] = [];
}
// Push the object to the correct array
result[drive][type].push({path, name, type, size});
return result;
}, {});
}

The function must traverse the file system recursively, looking for files that match your criteria. The recursion can be simplified by the fact that the result doesn't need to retain any hierarchy, so we can just carry a flat array (files) as a parameter.
let exts = [...]
async function scan(path, files) {
const stats = await fsp.stat(path)
if (stats.isDirectory()) {
childPaths = await fsp.readdir(path)
let promises = childPaths.map(childPath => {
return scan(join(path, childPath), files)
})
return Promise.all(promises)
} else if (stats.isFile()) {
const fileExt = extname(path).replace('.', '')
if (exts.includes(fileExt)) {
files.push({
path: path,
name: basename(path, fileExt).slice(0, -1),
type: fileExt,
size: stats.size
})
}
}
}
let files = []
await scan('/', files)
console.log(files)

Expand my image upload from 1 to 5 photos; map/foreach?

I am creating an app in which you can upload a photo, with some other data, to Firebase. The uploading part worked perfect with one picture. However I have now added a multiple-image picture (select 1 to 5 pictures) and would like my image upload function to upload the 5 pictures in stead of the 1.
The image upload works with 1 image provided, so how can I rearrange my code to upload the x-amount of photos in the array?
The pictures are added in the photos array with the following data (output shown below is a console.log from the images fetched);
Array [
Object {
"exists": true,
"file": "ph://8905951D-1D94-483A-8864-BBFDC4FAD202/L0/001",
"isDirectory": false,
"md5": "f9ebcab5aa0706847235887c1a7e4740",
"modificationTime": 1574493667.505371,
"size": 104533,
"uri": "ph://8905951D-1D94-483A-8864-BBFDC4FAD202/L0/001",
},
With this didFocus I check if the fethedImages param is set and set the photos array to the fetched images (So all the data that is shown above)
const didFocusSubscription = props.navigation.addListener(
'didFocus', () => {
let fetchedImages = props.navigation.getParam('fetchedImages')
console.log(fetchedImages)
setPhotos(fetchedImages)
setImageValid(true)
calculateImageDimensions()
}
);
When I save the page and start dispatching the data I run the following command the uploadImage function is ran and returns an uploadurl, this is then saved later on in the dispatch function to the Firebase Database to be fetched later;
uploadurl = await uploadImageAsync(photos)
SO the uploadImageAsync starts with the photos array forwarded. How can I make sure the function below is started for every photo.uri in the array? Can I use .map of for each for this, and in what context should I be using this?
Also I am not quite sure how I can send back an array of URLs to be saved together with the rest of the information.
async function uploadImageAsync(photos) {
console.log('uploadImage is gestart')
// Why are we using XMLHttpRequest? See:
// https://github.com/expo/expo/issues/2402#issuecomment-443726662
const blob = await new Promise((resolve, reject) => {
const xhr = new XMLHttpRequest();
xhr.onload = function () {
resolve(xhr.response);
};
xhr.onerror = function (e) {
console.log(e);
reject(new TypeError('Network request failed'));
};
xhr.responseType = 'blob';
xhr.open('GET', photos, true);
xhr.send(null);
});
const ref = firebase
.storage()
.ref()
.child(uuid.v4());
const snapshot = await ref.put(blob);
// We're done with the blob, close and release it
blob.close();
return await snapshot.ref.getDownloadURL();
}
==============edited because of progress with uploading====================
Once again I am a little bit further. However the image upload function is now running, and because of is being multiple images I would like to await the response of all the images before continuing.
try {
uploadurl = await uploadImageAsync()
address = await getAddress(selectedLocation)
console.log(uploadurl)
if (!uploadurl.lenght) {
Alert.alert('Upload error', 'Something went wrong uploading the photo, plase try again', [
{ text: 'Okay' }
]);
return;
}
dispatch(
At this moment when I start the uploadImageAsync function. With he help of console.log I see it uploading the images, they also show up online. But while the pictures are uploading the upload url already returns with 0 and shows the Alert and stops the function.
uploadImageAsync = async () => {
const provider = firebase.database().ref(`providers/${uid}`);
let imagesArray = [];
try {
await photos.map(img => {
let file = img.data;
const path = "Img_" + uuid.v4();
const ref = firebase
.storage()
.ref(`/${uid}/${path}`);
ref.putString(file).then(() => {
ref
.getDownloadURL()
.then(images => {
imagesArray.push({
uri: images
});
console.log("Out-imgArray", imagesArray);
})
return imagesArray <== this return imagesArray is fired to early and starts the rest of my upload function.
} catch (e) {
console.error(e);
}
};

So a Discord chat pointed me in the way of a promise.all function for this to work. I tried that, but opened another stack overflow topic for getting this to work.
await response of image upload before continue function
The solution for my image upload function is in the topic above;
uploadImages = () => {
const provider = firebase.database().ref(`providers/${uid}`);
// CHANGED: removed 'let imagesArray = [];', no longer needed
return Promise.all(photos) // CHANGED: return the promise chain
.then(photoarray => {
console.log('all responses are resolved successfully');
// take each photo, upload it and then return it's download URL
return Promise.all(photoarray.map((photo) => { // CHANGED: used Promise.all(someArray.map(...)) idiom
let file = photo.data;
const path = "Img_" + uuid.v4();
const storageRef = firebase // CHANGED: renamed 'ref' to 'storageRef'
.storage()
.ref(`/${uid}/${path}`);
let metadata = {
contentType: 'image/jpeg',
};
// upload current photo and get it's download URL
return storageRef.putString(file, 'base64', metadata) // CHANGED: return the promise chain
.then(() => {
console.log(`${path} was uploaded successfully.`);
return storageRef.getDownloadURL() // CHANGED: return the promise chain
.then(fileUrl => ({uri: fileUrl}));
});
}));
})
.then((imagesArray) => { // These lines can
console.log("Out-imgArray: ", imagesArray) // safely be removed.
return imagesArray; // They are just
}) // for logging.
.catch((err) => {
console.error(err);
});
};

How to properly map data from multiple nested queries into desired object. React/Firebase

I am trying to map out my schema from Firebase Firestore. I used Flamelink as a headless CMS to make things easier, however I am facing a limitation.
The image storage location is separate from the data that i'm pulling in from my schema. To remedy this I created some nested Queries to find the correct path of the storage location and generate the URL for the image. I need that image URL to be placed back into the respected object somewhere.
Currently it's just adding it to the overall array not the objects already within it. The first console log is the file reference in the correct order, the second console log is the correct URL's I need into the original object. However, these seem to not be in the correct order. I am fairly new to react and ES6 so I could be over complicating things. Essentially I just want the output of the second Console.log into my ThumbnailURL position for each object in the respective place.
I have tried to just create a new Array of objects and map them in my front end, however that causes issues because they are not in the same object.
const storage = firebase.storage().ref('flamelink/media/sized/240/')
class ContentUpdates extends React.Component {
constructor(){
super();
this.ref = firebase.firestore().collection('fl_content').where('_fl_meta_.schema', '==', 'collateral');
this.unsubscribe = null;
this.state = {
collateral: [],
}
}
onCollectionUpdate = (querySnapshot) => {
const collateral = [];
querySnapshot.forEach((doc) => {
const { name, image, thumbnail, thumbnailURL, img} = doc.data();
collateral.push({
key: doc.id,
doc, // DocumentSnapshot
name,
image,
img,
thumbnail: image[0].id,
thumbnailURL: firebase.firestore().collection(`fl_files`).where(`id`, '==', `${image[0].id}`).get().then((querySnapshot) => {
querySnapshot.forEach((doc) => {
const { file } = doc.data();
console.log('this is in the correct order', `${file}`);
storage.child(`${file}`).getDownloadURL().then((url) => {
const fileurl = url
console.log('this is the value I need mapped to thumbnailURL', fileurl)
collateral.push({thumbnailURL: fileurl})
})
})
})
});
});
this.setState({
collateral
});
}
componentDidMount() {
this.unsubscribe = this.ref.onSnapshot(this.onCollectionUpdate);
}
I want each objected in collateral to go through the necessary queries to return the fileurl back to it's respected collateral object.
UPDATE: I cleaned up the function a bit. Issue now is that it's going through two loops and not returning the values but only a promise. Very close to solving it however. Will update soon.
onCollectionUpdate = (querySnapshot) => {
const collateral = [];
querySnapshot.forEach((doc) => {
const { name, image, thumbnail, thumbnailURL, img} = doc.data();
const thumbnailImg = this.getThumbnailUrl(image[0].id);
let obj = {
key: doc.id,
doc, // DocumentSnapshot
name,
image,
img,
thumbnail: image[0].id,
thumbnailURL: thumbnailImg
}
collateral.push(obj);
});
this.setState({
collateral
});
}
async getThumbnailUrl(imgRef) {
console.log('your in', imgRef)
let snapshot = await firebase.firestore().collection(`fl_files`).where(`id`, '==', imgRef).get();
console.log(snapshot)
let snapVal = snapshot.forEach( async(doc) => {
const { file } = doc.data();
console.log('this is in the correct order', `${file}`);
let downloadURL = await storage.child(`${file}`).getDownloadURL()
console.log('this is the value I need mapped to thumbnailURL', downloadURL)
return downloadURL;
})
return snapVal;
}
componentDidMount() {
this.unsubscribe = this.ref.onSnapshot(this.onCollectionUpdate);
}

firebase storage, retrieving an image to a vue app

I am trying to download an image from my firebase storage to render it in my Vue app, the upload from the application to the firebase storage is successful, however upon retrieval it gives me an error, i am using the firebase SDK in a Vue CLI 3 setup and vuex to manage my state. Here is the function setting in my actions in the main store.js file
createMeetUp({commit, getters}, payload) {
//here my payload is an object contains the following props
const meetup = {
title: payload.title,
location: payload.location,
date: payload.date.toISOString(),
description: payload.description,
creatorId: getters.user.id
}
let imageUrl
let key
//now i am reaching out to the firebase database to store the above object
firebase.database().ref('meetup').push(meetup)
.then(data => {
key = data.key
return key
})
.then(key => {
//also in my payload object i stored an image file
//so here i am uploading the image to the firebase storage
const fileName = payload.image.name
const extension = fileName.slice(fileName.lastIndexOf('.'))
return firebase.storage().ref('meetup/' + key + '.' + extension).put(payload.image)
})
.then(imageInfo => {
//the issue is here in this then() block as i am stuck on how to retrieve the image from the storage to render it in the app
imageUrl = imageInfo.getDownloadURL()
return firebase.database().ref('meetups').child(key).update({
imageUrl: imageUrl
})
})
.then(() => {
//here i am simply commiting my mutation..
commit('createMeetUp', {
...meetup,
imageUrl: imageUrl,
id : key
})
})
.catch(err => console.log(err))
}
the error I am getting is:
TypeError: imageInfo.getDownloadURL is not a function
Again I believe the issue is in the then() block where I retrieve the image from the firebase storage.
thanks in advance

Following the comments above, the following should work if I am not mistaking (not tested...).
Note that getDownloadURL() returns a promise (see here), therefore you have to chain the promises.
....
.then(key => {
//also in my payload object i stored an image file
//so here i am uploading the image to the firebase storage
const fileName = payload.image.name
const extension = fileName.slice(fileName.lastIndexOf('.'))
return firebase.storage().ref('meetup/' + key + '.' + extension).put(payload.image)
})
.then(uploadTaskSnapshot => {
return uploadTaskSnapshot.ref.getDownloadURL()
})
.then(imageUrl => {
return firebase.database().ref('meetups').child(key).update({
imageUrl: imageUrl
})
})
....