I have followed the instructions here to use a Google Sheet as a JSON endpoint. I am then using that data in Eleventy. This code is currently working:
module.exports = async function() {
let url = `https://docs.google.com/spreadsheets/d/${process.env.GOOGLE_SHEETS_ID}/gviz/tq?tqx=out:json`;
console.log("Fetching from Google Sheets...");
return await fetch(url)
.then(res => res.text()) // node-fetch option to transform to json
.then(text => {
let json = JSON.parse(text.substr(47).slice(0, -2));
return {
items: json.table.rows
};
});
}
...however, this is making for slow build times, so I am trying to tie this in with the eleventy-cache-assets plugin as described in the 11ty docs.
Here is what I've tried:
module.exports = async function() {
let url = `https://docs.google.com/spreadsheets/d/${process.env.GOOGLE_SHEETS_ID}/gviz/tq?tqx=out:json`;
var text = await Cache(url, {
duration: "1s",
type: "text"
})
.then(text => {
var json = JSON.parse(text.substr(47).slice(0, -2));
console.log(json);
return {
items: json.table.rows
};
});
};
In the console, it does return the JSON, but then when I try to make a collection out of the data in .eleventy.js like this:
eleventyConfig.addCollection("myGSheets", (collection) => {
return collection.getAll()[0].data.myGSheets.items;
});
I get an error: Cannot read property 'items' of undefined
I'm not sure what is happening in between the JSON data appearing in the console, and then being undefined.
I am guessing maybe I need to do the string manipulation of the response before calling Cache, perhaps? I'm just not sure how to put it all together...
It looks like you're not actually returning anything from your data function, since your return statement is inside the callback function, not the top-level data function.
module.exports = async function() { // <= top level data function
let url = `https://docs.google.com/spreadsheets/d/${process.env.GOOGLE_SHEETS_ID}/gviz/tq?tqx=out:json`;
var text = await Cache(url, {
duration: "1s",
type: "text"
})
.then(text => { // <= inner callback function
var json = JSON.parse(text.substr(47).slice(0, -2));
console.log(json);
return { // <= This return statement returns from the inner callback
items: json.table.rows
};
});
// <= This function doesn't return anything!
};
Since you're using async/await, there isn't a need for using Promise.then. You can just await the promises, which prevents needing all the callbacks.
Try:
module.exports = async function() {
let url = `https://docs.google.com/spreadsheets/d/${process.env.GOOGLE_SHEETS_ID}/gviz/tq?tqx=out:json`;
var text = await Cache(url, {
duration: "1s",
type: "text"
})
var json = JSON.parse(text.substr(47).slice(0, -2));
console.log(json);
return {
items: json.table.rows
};
};
Related
I am trying to make 10 API calls to the reddit API to get a users most recent comments.
Basically, Reddit uses a parameter 'after' in the URI to get the next page. So basically, if there is some dummy example comment data below, I need the LAST ID.
Example.
API Call: https://www.reddit.com/user/USERNAME/comments.json?limit=3
returns:
{
id: 'd4gf125',
comment: 'blah blah blah'
},
{
id: 'dag42ra',
comment: 'blah blah blah'
},
{
id: 'hq6ir3j', // I need this ID right here, the LAST in the list for the next API call
comment: 'blah blah blah'
},
Then, another API call needs to be made directly AFTER the previous, using the ID as a parameter.
API Call:
`https://www.reddit.com/user/USERNAME/comments.json?limit=3&after=tl_hq6ir3j`
This will run 10 times, which will get 10 pages, so in total, 30 results.
I have tried using this for loop but as for loops are not asynchronous, I cannot change the variable 'lastID' in time.
First attempt:
const handleSubmit = async () => {
setLoading(true);
axios.get(`${apiUrl}user/${username}/about.json`).then((res) => {
setUserData({
about: res.data.data,
});
axios
.get(`${apiUrl}user/${username}/comments.json?limit=100`)
.then((res) => {
let data = res.data.data.children;
let lastItem = data[data.length - 1];
for (i = 0; i < 10; i++) {
axios
.get(
`${apiUrl}comments/${lastItem.data.id}/comments.json?limit=100&after=tl_${lastItem}`
)
.then((res) => {
lastItem =
res.data.data.children[res.data.data.children.length - 1];
});
}
});
});
};
If you need some example REAL data, see the link below:
https://www.reddit.com/user/bulkorcut99/comments.json?limit=1000&after=t1_hq6ir3j
To do async things in sequence, chain promises together with then(). The generic form looks like ths...
let promise = /* starting promise */
for (/* loop stuff */) {
promise = promise.then(/*another promise*/)
}
Refactoring your code to make this more clear...
function getAbout(username) {
return axios.get(`${apiUrl}user/${username}/about.json`).then(res => {
return res.data.data
});
}
function getComment(username, afterId) {
const url = `${apiUrl}comments/${lastItem.data.id}/comments.jsonlimit=100`;
if (afterId) url += `&after=tl_${afterId}`;
return axios.get(url).then(res => res.data.data);
}
// create a chain of promises to get comments, pushing results as we go
function getComments(username, results) {
let promise = getComment(username);
for (let i=0; i<10; i++) { // not sure why 10 here... from the OP
promise = promise.then(data => {
results.push(data);
const lastChild = data.children[data.children.length-1];
return getComment(username, lastChild.id);
});
}
return promise;
}
const handleSubmit = async () => {
setLoading(true);
return getAbout(username).then(data => {
setUserData({ about: data });
}).then(() => {
let comments = [];
return getComments(username, comments).then(() => comments)
}).then(comments => {
// comments is the result from all of the get comments calls
})
}
The real API might stop finding comments in fewer than 10 calls. If so, rather than looping to 10, you'd loop until the results were empty or whatever condition indicates there are no more.
You can simply await each of your axios.get calls in your for loop. To do so you can do something like this:
const handleSubmit = async () => {
setLoading(true);
axios.get(`${apiUrl}user/${username}/about.json`).then((res) => {
setUserData({
about: res.data.data,
});
axios
.get(`${apiUrl}user/${username}/comments.json?limit=100`)
.then(async (res) => {
let data = res.data.data.children;
let lastItem = data[data.length - 1];
for (i = 0; i < 10; i++) {
var {data} = await axios
.get(`${apiUrl}comments/${lastItem.data.id}/comments.json?limit=100&after=tl_${lastItem}`)
lastItem = data.data.children[data.data.children.length - 1];
}
});
});
};
I have node.js code that fetches an XML feed, pipes it into a sax parser and extracts the data I need into a JS Object.
The code, at its simplest level looks like this.
const fetch = require('node-fetch')
const sax = require('sax')
function fetchAndParse(url)
{
const saxStream = require("sax").createStream(strict, {normalize: true, trim: true})
const desiredData = []
saxStream.on("error", function (err) {
...
}).on("opentag", function (node) {
...
}).on("cdata", function(t) {
...
}).on("text", function(t) {
...
}).on("closetag", function (nodeName) {
...
}).on("end", function() {
console.log("END", desiredData)
})
fetch(url)
.then(
res => { res.body.pipe(saxStream) }
)
}
Ideally, I'd like to turn this into an async function and just use await when calling it, but at the moment I'm not seeing how to get the data out of here when it completes because the only place I've been able to access the finished data is in the on("end") function. I think I'm missing something really basic here.
I haven't worked with the sax library before so I'm not entirely sure what its API looks like, but you should be able to just return a promise that resolves with the data you want.
Something like this (note: I didn't test this):
const fetch = require("node-fetch");
const sax = require("sax");
function fetchAndParse(url) {
const saxStream = require("sax").createStream(strict, {
normalize: true,
trim: true,
});
const desiredData = [];
return new Promise((resolve, reject) => {
saxStream
.on("error", function (err) {
reject(err);
})
.on("opentag", function (node) {
// ...
})
.on("cdata", function (t) {
// ...
})
.on("text", function (t) {
// ...
})
.on("closetag", function (nodeName) {
// ...
})
.on("end", function () {
resolve(desiredData);
});
fetch(url).then((res) => {
res.body.pipe(saxStream);
});
});
}
The sax parser has 2 forms: one that takes in a pipe and another that takes in a string. The piped version can only emit the pipe stream, in case you want to pass it along to something else, and there appears to be no way to override this behavior.
As a solution, use the parser as written, and invoke it as shown below. This requires fetching the entire document, unfortunately. Otherwise consider using another parser.
const fetch = require('node-fetch')
const sax = require('sax')
function fetchAndParse(url)
{
const saxStream = require("sax").createStream(strict, {normalize: true, trim: true})
const desiredData = []
saxStream.on("error", function (err) {
// Rest of code from question goes here
}
const res = await fetch(url)
const rss = await res.text()
await saxStream.write(rss)
return desiredData;
}
I'm working with a Grafana plugin API and I'm trying to create a function that returns an object literal var literals that will contain new data that I fetched from the server.
The result of this function return literals; will go, as a parameter, in to another Grafana API function later.
I'm very confused by how promises work and I don't really know how to tackle this problem.
Right now in this code I'm having a problem with a console error
TypeError: data.dbs is undefined'
at the line data.dbs.forEach(element => {
I thought that after calling the .then() function the data should already be returned and yet it's still unresolved?
I would also like to know how exactly can I create this whole getDropdown() function so that it will be able to send the resolved data forward?
export function getDropdown() {
var literals = {
path: 'displayMode',
name: 'Cell display mode',
description: 'Color text, background, show as gauge, etc',
settings: {
options: [{
value: 1,
label: 'Placeholder'
}],
},
};
var data = {
dbs: [{
id: 1,
database: 'placeholder',
}, ],
};
fetch('/api/datasources')
.then(function(res) {
return res.json();
})
.then(json => {
data = json;
var counter = 0;
data.dbs.forEach(element => {
literals.settings.options[counter].label = element.database;
literals.settings.options[counter].value = element.id;
counter++;
});
});
return literals;
}
I was able to solve the problem with this code
fetch('/api/datasources')
.then(data => {
return data.json();
})
.then(data => {
data.forEach((element: { database: string; id: number }) => {
literals.settings.options[counter].label = element.database;
literals.settings.options[counter].value = element.id;
counter++;
});
});
I think there was something wrong with data.dbs but I'm still not sure what.
Insted of using then() functions, try using async/await where you can asyncronously execute functions & you can also wait till getting a promise.
In the above code you can write an async function & await till you get response from
fetch('/api/datasources')
// Do
async function getData() {
var data = await fetch('/api/datasources')
var jsonData = JSON.parse(data)
jsonData.dbs.forEach(element => {
literals.settings.options[counter].label = element.database;
literals.settings.options[counter].value = element.id;
counter++;
});
}
// Instead of
fetch('/api/datasources')
.then(function(res) {
return res.json();
})
.then(json => {
data = json;
var counter = 0;
data.dbs.forEach(element => {
literals.settings.options[counter].label = element.database;
literals.settings.options[counter].value = element.id;
counter++;
});
});
I'm new to Node/asynchronous coding and I realize this is a basic question that speaks to some fundamentals I'm missing, but for the life of me I just can't understand how this is supposed to work. I'm seeding a database using knex, by reading in data from a CSV and iterating through the rows in a for loop. Within that loop, I need to query another table in the database and return an associated value as part of the new row to be created.
I understand that knex returns a pending Promise, so it doesn't have access to the returned value yet. But I can't seem to get the structure right with async/await or Promise.all; I've tried it a bunch of ways based on other answers I've seen, but none seem to actually explain what's happening well enough that I can apply it successfully to my case. Any help hugely appreciated.
My seed file currently looks like this:
exports.seed = function(knex) {
const fs = require('fs');
function createImage(row, event_id) {
return {
name: row[1],
event_id: event_id
}
};
function get_event_id(loc) {
knex('events')
.where({location: loc})
.first()
.then(result => {console.log(result)}); // <-- this never executes
};
let images = [];
const file = fs.readFileSync('./data.csv');
const lines = file.toString().replace(/[\r]/g, '').split('\n');
for (i=1; i<lines.length; i++) {
var row = lines[i].split(',');
var event_id = (async function () { return await get_event_id(row[0]) }) ();
images.push(createImage(row, event_id));
};
console.log(images);
// Inserts seed entries
// return knex('table_name').insert(images);
};
Output:
[ { name: '003.jpg', event_id: undefined } ]
My data.csv is structured like:
Location,Name
Amsterdam,003.jpg,
...
You can change your for loop into an Array.map and use Promise.all on returned promises.
Also, seed will return a promise so invoke it properly
exports.seed = async function (knex) {
const fs = require('fs');
function createImage(row, event_id) {
return {
name: row[1],
event_id: event_id
}
};
function get_event_id(loc) {
return knex('events')
.where({ location: loc })
.first()
.then(result => { console.log(result); return result }); // <-- this never executes
};
const file = fs.readFileSync('./data.csv');
const lines = file.toString().replace(/[\r]/g, '').split('\n');
let promises = lines.map(async (line) => {
let [row] = line.split(',');
let event_id = await get_event_id(row)
return createImage(row, event_id)
});
let images = await Promise.all(promises);
console.log(images);
// Inserts seed entries
// return knex('table_name').insert(images);
};
The function below calls several asynchronous functions in a for loop. It's parsing different CSV files to build a single JavaScript object. I'd like to return the object after the for loop is done. Its returning the empty object right away while it does the asynchronous tasks. Makes sense, however I have tried various Promise / async /await combinations hopes of running something once the for loop has completed. I am clearly not understanding what is going on. Is there a better pattern to follow for something like this or am I thinking about it incorrectly?
async function createFormConfig(files: string[]): Promise<object>
return new Promise(resolve => {
const retConfig: any = {};
for (const file of files) {
file.match(matchFilesForFormConfigMap.get('FIELD')) ?
parseCsv(file).then(parsedData => {
retConfig.fields = parsedData.data;
})
: file.match(matchFilesForFormConfigMap.get('FORM'))
? parseCsv(file).then(parsedData => retConfig.formProperties = parsedData.data[0])
: file.match(matchFilesForFormConfigMap.get('PDF'))
? parseCsv(file).then(parsedData => retConfig.jsPdfProperties = parsedData.data[0])
: file.match(matchFilesForFormConfigMap.get('META'))
? parseCsv(file).then(parsedData => {
retConfig.name = parsedData.data[0].name;
retConfig.imgType = parsedData.data[0].imgType;
// console.log(retConfig); <- THIS CONSOLE WILL OUTPUT RETCONFIG LOOKING LIKE I WANT IT
})
: file.match(matchFilesForFormConfigMap.get('PAGES'))
? parseCsv(file).then(parsedData => retConfig.pages = parsedData.data)
: console.log('there is an extra file: ' + file);
}
resolve(retConfig); // <- THIS RETURNS: {}
});
This is the code I'm using to call the function in hopes of getting my 'retConfig' filled with the CSV data.
getFilesFromDirectory(`${clOptions.directory}/**/*.csv`)
.then(async (files) => {
const config = await createFormConfig(files);
console.log(config);
})
.catch(err => console.error(err));
};
First, an async function returns a Promise, so you dont have to return one explicitely.Here is how you can simplify your code:
async function createFormConfig(files: string[]): Promise<object> {
// return new Promise(resolve => { <-- remove
const retConfig: any = {};
// ...
// The value returned by an async function is the one you get
// in the callback passed to the function `.then`
return retConfig;
// }); <-- remove
}
Then, your function createFormConfig returns the config before it has finished to compute it. Here is how you can have it computed before returning it:
async function createFormConfig(files: string[]): Promise<object> {
const retConfig: any = {};
// Return a Promise for each file that have to be parsed
const parsingCsv = files.map(async file => {
if (file.match(matchFilesForFormConfigMap.get('FIELD'))) {
const { data } = await parseCsv(file);
retConfig.fields = data;
} else if (file.match(matchFilesForFormConfigMap.get('FORM'))) {
const { data } = await parseCsv(file);
retConfig.formProperties = data[0];
} else if (file.match(matchFilesForFormConfigMap.get('PDF'))) {
const { data } = await parseCsv(file);
retConfig.jsPdfProperties = data[0];
} else if (file.match(matchFilesForFormConfigMap.get('META'))) {
const { data } = await parseCsv(file);
retConfig.name = data[0].name;
retConfig.imgType = data[0].imgType;
} else if (file.match(matchFilesForFormConfigMap.get('PAGES'))) {
const { data } = await parseCsv(file);
retConfig.pages = data;
} else {
console.log('there is an extra file: ' + file);
}
});
// Wait for the Promises to resolve
await Promise.all(parsingCsv)
return retConfig;
}
async functions already return promises, you don't need to wrap the code in a new one. Just return a value from the function and the caller will receive a promise that resolves to the returned value.
Also, you have made an async function, but you're not actually using await anywhere. So the for loop runs through the whole loop before any of your promises resolve. This is why none of the data is making it into your object.
It will really simplify your code to only use await and get rid of the then() calls. For example you can do this:
async function createFormConfig(files: string[]): Promise<object> {
const retConfig: any = {};
for (const file of files) {
if (file.match(matchFilesForFormConfigMap.get('FIELD')){
// no need for the then here
let parsedData = await parseCsv(file)
retConfig.field = parsedData.data
}
// ...etc
At the end you can just return the value:
return retConfig