Develop Reference

How can I read a CSV file from a URL in a Next.js application?

I have a Next.js application here which needs to read a CSV file from a URL in the same repo in multiple places, but I cannot seem to be able to retrieve this data. You can find the relevant file in my repo here.
Note, the URL I'm trying to pull data from is this: https://raw.githubusercontent.com/ivan-rivera/balderdash-next/main/public/test_rare_words.csv
Here is what I've tried so far:
Approach 1: importing the data
let vocab = {};
...
async function buildVocab() {
const words = await import(VOCAB_URL); // this works when I point to a folder in my directory, but it does not work when I deploy this app. If I point at the URL address, I get an error saying that it cannot find the module
for (let i = 0; i < words.length; i++) {
vocab[words[i].word] = words[i].definition;
}
}
Approach 2: papaparse
const papa = require("papaparse");
let vocab = {};
...
export async function buildVocab() {
await papa.parse(
VOCAB_URL,
{
header: true,
download: true,
delimiter: ",",
step: function (row) {
console.log("Row:", row.data); // this prints data correctly
},
complete: function (results) {
console.log(results); // this returns an object with several attributes among which is "data" and "errors" and both are empty
},
}
);
// this does not work because `complete` does not return anything
vocab = Object.assign({}, ...raw.map((e) => ({ [e.word]: e.definition })));
console.log(vocab);
}
Approach 3: needle
const csvParser = require("csv-parser");
const needle = require("needle");
let vocab = {};
...
let result = [];
needle
.get(VOCAB_URL)
.pipe(csvParser())
.on("data", (data) => {
result.push(data);
});
vocab = Object.assign({}, ...result.map((e) => ({ [e.word]: e.definition })));
// This approach also returns nothing, however, I noticed that if I force it to sleep, then I do get the results I want:
setTimeout(() => {
console.log(result);
}, 1000); // now this prints the data I'm looking for
What I cannot figure out is how to force this function to wait for needle to retrieve the data. I've declared it as an async function and I'm calling it with await buildVocab() but it doesn't help.
Any ideas how I can fix this? Sorry, I'm a JS beginner, so it's probably something fundamental that I'm missing :(

After spending hours on this, I think I finally found a solution:
let vocab = {};
export async function buildVocab() {
await fetch(VOCAB_URL)
.then((resp) => resp.text())
.then((text) => {
papa.parse(text, { header: true }).data.forEach((row) => {
vocab[row.word] = row.definition;
});
});
}
The only oddity that I still can't work out is this: I'm calling my buildVocab function inside another async function and I noticed that if I do not include a console.log statement in that function, then the vocab still does not get populated in time. Here is the function:
export async function sampleWord() {
await buildVocab();
const keys = Object.keys(vocab);
const index = Math.floor(Math.random() * keys.length);
console.log(`selected word: ${keys[index]}`); // this is important!
return keys[index];
}

Cache manegment in recoil js

I decided to try using recoil instead of redux and ran into the problem of caching mutable collections. To be more precise, the problem is not in the response caching itself, but in the further data management, without re-requesting. For example, I requested an array of free time slots that will be displayed in the calendar, the user can change them (unblock or block). After the request (unblock or block) I can use useRecoilRefresher_UNSTABLE to reset the selector cache and request updated slots again, this works as expected, but why do this if you can remember the result after the first request and then just update it. My code is:
export const _cacheFreeSlotsA = atom<SlotsArray | null>({
key: '_cacheFreeSlotsA',
default: null,
});
export const freeSlotsS = selector<SlotsArray>({
key: 'freeSlotsS',
get: async ({ get }) => {
const cache = get(_cacheFreeSlotsA);
if (cache) {
return cache;
}
const res = await slotsAPI.getFreeSlots();
return res.slots;
},
});
export const useUpdateFreeSlotsS = () => {
const setFreeSlots = useSetRecoilState(_cacheFreeSlotsA);
const currentSlots = useRecoilValue(freeSlotsS);
return async (req: ChangeSlotsRequest) => {
await slotsAPI.changeSlots(req);
switch (req.operation) {
case 'open':
setFreeSlots([...currentSlots, ...req.slots]);
break;
case 'close':
setFreeSlots(difference(currentSlots, req.slots));
break;
}
};
};
// export const useUpdateFreeSlotsS = () => {
// const refresh = useRecoilRefresher_UNSTABLE(freeSlotsS);
// return async (req: ChangeSlotsRequest) => {
// await slotsAPI.changeSlots(req);
// refresh();
// };
// };
It works but look like workaround. Is there a more elegant and clear way to implement this behavior? (it would be just perfect if inside the freeSlotsS get method I could get access to the set method, but unfortunately it is not in the arguments)

How can I connect twice to db in Cypress tests?

I need to connect to my db (Postgres) twice during autotest - at the beginning to truncate and at the end to select a new note from the table.
I tried to do it using pg-promise, but in the second connection it returns null or undefined (the following error shows after asserting expected and real results - "Target cannot be null or undefined.").
If I don't execute the first connecting (for truncating), the second runs normally and returns new note from table.
Also, if I make Truncate after Select, it gives two different result, depending of whether the table empty or not.
If it is empty, the result is the same error. But if there is some record initially, all becomes ok and test finishes without error.
This is how I make connecting and disconnecting to db:
const pgp = require('pg-promise')();
const postgresConfig = require(require('path').resolve('cypress.json'));
function dbConnection(query, userDefineConnection) {
const db = pgp(userDefineConnection || postgresConfig.db);
return db.any(query).finally(db.$pool.end)
// return db.any(query).finally(pgp.end)
}
/**#type {Cypress.PluginConfig} */
module.exports = (on, config) => {
on("task", {
dbQuery: (query) => dbConnection(query.query, query.connection)
});
}
And this is how I make requests to DB in test:
describe('example to-do app', () => {
beforeEach(() => {
cy.task('dbQuery', {'query': 'TRUNCATE TABLE auto_db.public.requests RESTART IDENTITY CASCADE'})
})
it('tablist displays class', () => {
cy.contains('button > span', 'Save').click()
cy.task('dbQuery', {'query': 'SELECT * FROM auto_db.public.requests'})
.then(queryResponse => {
expect(queryResponse[0]).to.deep.contain({
id: 1,
author_id: 4,
type_id: 1,
})
})
})
})

If you look at the implementation for cypress-postgres it's similar but they break up the response/return to fit the call to pgp.end() in between.
Since it sounds like the 1st connection isn't closing, I'd suspect the .finally() call isn't working.
const pgp = require('pg-promise')();
const postgresConfig = require(require('path').resolve('cypress.json'));
function dbConnection(query, userDefineConnection) {
const db = pgp(userDefineConnection || postgresConfig.db);
let response = db.any(query)
pgp.end()
return response
}
/**#type {Cypress.PluginConfig} */
module.exports = (on, config) => {
on("task", {
dbQuery: (query) => dbConnection(query.query, query.connection)
});
}
You should be able to get rid of the postgresConfig line since (on, config) is the same (in fact it's better because you might want to overwrite some config on the command line).
const pgp = require('pg-promise')();
/**#type {Cypress.PluginConfig} */
module.exports = (on, config) => {
function dbConnection(query, userDefineConnection) {
const db = pgp(userDefineConnection || config.db);
let response = db.any(query)
pgp.end()
return response
}
on("task", {
dbQuery: (query) => dbConnection(query.query, query.connection)
});
}

The problem was that new record in database has been creating not right away and cypress just was looking in db earlier that creating.
Adding of waiting before db checking helped.

How to use dataloader?

Im trying to figure this out.
I want to get all my users from my database, cache them
and then when making a new request I want to get those that Ive cached + new ones that have been created.
So far:
const batchUsers = async ({ user }) => {
const users = await user.findAll({});
return users;
};
const apolloServer = new ApolloServer({
schema,
playground: true,
context: {
userLoader: new DataLoader(() => batchUsers(db)),// not sending keys since Im after all users
},
});
my resolver:
users: async (obj, args, context, info) => {
return context.userLoader.load();
}
load method requiers a parameter but in this case I dont want to have a specific user I want all of them.
I dont understand how to implement this can someone please explain.

If you're trying to just load all records, then there's not much of a point in utilizing DataLoader to begin in. The purpose behind DataLoader is to batch multiple calls like load(7) and load(22) into a single call that's then executed against your data source. If you need to get all users, then you should just call user.findAll directly.
Also, if you do end up using DataLoader, make sure you pass in a function, not an object as your context. The function will be ran on each request, which will ensure you're using a fresh instance of DataLoader instead of one with a stale cache.
context: () => ({
userLoader: new DataLoader(async (ids) => {
const users = await User.findAll({
where: { id: ids }
})
// Note that we need to map over the original ids instead of
// just returning the results of User.findAll because the
// length of the returned array needs to match the length of the ids
return ids.map(id => users.find(user => user.id === id) || null)
}),
}),
Note that you could also return an instance of an error instead of null inside the array if you want load to reject.

Took me a while but I got this working:
const batchUsers = async (keys, { user }) => {
const users = await user.findAll({
raw: true,
where: {
Id: {
// #ts-ignore
// eslint-disable-next-line no-undef
[op.in]: keys,
},
},
});
const gs = _.groupBy(users, 'Id');
return keys.map(k => gs[k] || []);
};
const apolloServer = new ApolloServer({
schema,
playground: true,
context: () => ({
userLoader: new DataLoader(keys => batchUsers(keys, db)),
}),
});
resolver:
user: {
myUsers: ({ Id }, args, { userLoader }) => {
return userLoader.load(Id);
},
},
playground:
{users
{Id
myUsers
{Id}}
}
playground explained:
users basically fetches all users and then myusers does the same thing by inhereting the id from the first call.
I think I choose a horrible example here since I did not see any gains in performence by this. I did see however that the query turned into:
SELECT ... FROM User WhERE ID IN(...)

How to handle Promise in custom react useEffect

I have two components which i am working with. In the first component, i made a custom useEffect hook that retrieves data from my server. Please see the code below:
Code snippet One
import {useState, useCallback} from 'react';
import {stageQuizApi} from '../api/quiz';
import {QuestionService} from "../services/IdDbServices/question_service";
const usePostData = ({url, payload, config}) => {
const [res, setRes] = useState({data: null, error: null, isLoading: false});
const callAPI = useCallback(() => {
setRes(prevState => ({...prevState, isLoading: true}));
stageQuizApi.patch(url, payload, config).then( res => {
setRes({data: res.data, isLoading: false, error: null});
const questionInDb = {};
const {NoTimePerQuestion,anwser, question, playerDetails, option} = res.data.data;
const {playerid,anwserRatio, name} = playerDetails
questionInDb.timePerQuestion = NoTimePerQuestion;
questionInDb.anwserRatio = anwserRatio;
questionInDb.options = option;
questionInDb.answer = anwser;
questionInDb.playerId = playerid;
questionInDb.name = name;
questionInDb.question = question;
const Service = new QuestionService();
Service.addStudent(questionInDb).
then(response=>console.log(response))
.catch(err=>console.log(err));
}).catch((error) => {
console.log(error)
if (error.response) {
const errorJson = error.response.data
setRes({data: null, isLoading: false, error: errorJson.message});
} else if (error.request) {
setRes({data: null, isLoading: false, eror: error.request});
} else {
setRes({data: null, isLoading: false, error: error.message});
}
})
}, [url, config, payload])
return [res, callAPI];
}
export default usePostData;
The above module has two purpose. It first makes an axios request to my endpoint and secondly makes a database insertion to browser IndexDb (similar to localstorage but with sql approach) ( like inserting data into the database using the response that was gotten from the first request. so typically i have a promise in the outer .then block. This part:
Code snippet Two
const questionInDb = {};
const {NoTimePerQuestion,anwser, question, playerDetails, option} = res.data.data;
const {playerid,anwserRatio, name} = playerDetails
questionInDb.timePerQuestion = NoTimePerQuestion;
questionInDb.anwserRatio = anwserRatio;
questionInDb.options = option;
questionInDb.answer = anwser;
questionInDb.playerId = playerid;
questionInDb.name = name;
questionInDb.question = question;
const Service = new QuestionService();
Service.addStudent(questionInDb).
then(response=>console.log(response))
.catch(err=>console.log(err));
Here is the problem, I am trying to maintain state as i want the result of this module to be shared in another route and i don't want to hit the server again hence i inserted the result into indexDb browser storage. Here is the code that executes the above module:
Code snippet Three
const displaySingleQuestion = ()=>{
OnUserGetQuestion();
history.push('/player/question');
}
The above method is called from my first route /question and it is expected to redirect user to the /player/question when the displaySingleQuestion is called.
On the new route /player/question i then want to fetch the data from IndexDb and update the state of that component using the useEffect code below:
Code snippet Four
useEffect(()=>{
const getAllUserFromIndexDb = async()=>{
try{
const result = await new Promise((resolve, reject) => {
service.getStudents().then(res=>resolve(res)).catch(err=>reject(err))
});
console.log('it did not get to the point i was expecting',result)
if(result[0]){
console.log('it got to the point i was expecting')
const singleQuestion = result[0];
const questionPage = playerQuestionToDisplay;
questionPage.name = singleQuestion.name;
questionPage.anwserRatio = singleQuestion.anwserRatio;
questionPage.answer = singleQuestion.answer;
questionPage.options = singleQuestion.options;
questionPage.playerId = singleQuestion.playerId;
questionPage.question = singleQuestion.question;
questionPage.timePerQuestion = singleQuestion.timePerQuestion;
return setplayerQuestionToDisplay({playerQuestionToDisplay:questionPage})
}
}
catch(error){
console.log(error)
}
}
getAllUserFromIndexDb();
return function cleanup() {
setplayerQuestionToDisplay({playerQuestionToDisplay:{}})
}
},[history.location.pathname]);
The problem is that only one Button click (Code snippet three)(displaySingleQuestion()) triggers the whole functionality and redirect to the /player/question page but in this new route the state is not been set until a page reload as occurred, i tried debugging the problem and i found out that when the button is clicked i found out that Code snippet two is executed last hence when Code snippet Four ran it was in promise and until a page reloads occurs the state of the component is undefined
Thanks for reading, Please i would appreciate any help in resolving this issue.