I am trying to scrap wikipedia page to fetch list of airlines by first scrapping first page and then going to each individual page of airline to get the website url. I have divided the code in two functions. One to scrap main page and get a new url, and second function to scrap another page from the created url to get the website name from that page. I have used request-promise module for getting the html and then cheerio to parse the data.
export async function getAirlinesWebsites(req,res) {
let response = await request(options_mainpage);
console.log(`Data`);
let $ = cheerio.load(response);
console.log('Response got');
$('tr').each((i,e)=>{
let children = '';
console.log('inside function ', i);
if($(e).children('td').children('a').attr('class') !== 'new') {
children = $(e).children('td').children('a').attr('href');
let wiki_url = 'https://en.wikipedia.org' + children;
console.log(`wiki_url = ${wiki_url}`);
let airline_url = getAirlineUrl(wiki_url);
console.log(`airline_url = ${airline_url}`);
}
})
And then the getAirlineUrl() function will parse another page based on the provided url.
async function getAirlineUrl(url){
const wiki_child_options = {
url : url,
headers : headers
}
let child_response = await request(wiki_child_options);
let $ = cheerio.load(child_response);
let answer = $('.infobox.vcard').children('tbody').children('tr').children('td').children('span.url').text();
return answer;
})
However when I console log the answer variable in the parent function, I get a [object Promise] value instead of a String. How do I resolve this issue?
Async function return promise.In case of that,you need to use then to get resolved response or use await.
This should work if other part of your code is ok.
export async function getAirlinesWebsites(req, res) {
let response = await request(options_mainpage);
console.log(`Data`);
let $ = cheerio.load(response);
console.log("Response got");
$("tr").each(async (i, e) => {
let children = "";
console.log("inside function ", i);
if ($(e).children("td").children("a").attr("class") !== "new") {
children = $(e).children("td").children("a").attr("href");
let wiki_url = "https://en.wikipedia.org" + children;
console.log(`wiki_url = ${wiki_url}`);
let airline_url = await getAirlineUrl(wiki_url);
console.log(`airline_url = ${airline_url}`);
}
});
}
Since your getAirlineUrl function returns a promise, you need to await that promise. You can't have await nested inside of the .each callback because the callback is not an async function, and if it was it wouldn't work still. The best fix is the avoid using .each and just use a loop.
export async function getAirlinesWebsites(req,res) {
let response = await request(options_mainpage);
console.log(`Data`);
let $ = cheerio.load(response);
console.log('Response got');
for (const [i, e] of Array.from($('tr')).entries()) {
let children = '';
console.log('inside function ', i);
if($(e).children('td').children('a').attr('class') !== 'new') {
children = $(e).children('td').children('a').attr('href');
let wiki_url = 'https://en.wikipedia.org' + children;
console.log(`wiki_url = ${wiki_url}`);
let airline_url = await getAirlineUrl(wiki_url);
console.log(`airline_url = ${airline_url}`);
}
}
}
Related
When I run this code I get undefined but it is clear that the ski and product_id are in the value form.
I want:
value="BTdtb4CBz3uSJ2qv"
value="adi-ss20-042"
but I get "undefined"
class TresBien {
async scrapeRaffleInfo() {
// scrape the form_key and sku values
const response = await axios(
"https://tres-bien.com/adidas-yeezy-boost-380-mist-fx9764-ss20"
);
console.log("response: ", response);
const html = await response.data;
const $ = cheerio.load(html);
const res = $('input[name="sku"]').val();
const ans = $('input[name="form_key"]').val();
console.log(res && ans);
}
}
const main = async () => {
const tb = new TresBien(
"https://tres-bien.com/adidas-yeezy-700-v3-alvah-h67799-ss20"
);
let checkoutSucc = await tb.scrapeRaffleInfo();
if (checkoutSucc) {
Logger.logEventSuccess("Raffle successfully entered");
}
};
main();
There are a few things wrong in your code:
as #Pointy mentioned, scrapeRaffleInfo() is returning undefined and you're trying to use it in checkoutSucc
Tresbien class doesn't offer any constructor, yet you're passing your url as a parameter to the constructor of that class.
You're not using any of axios library methods (like: get(), post(), put(), delete()). typically you need axios.get() but there in your code, you're just using axios()
I'm currently converting my local function into a lambda function and am running into a few hardships. In a previous method, I get an array of url fragments that I have to string along programmatically to a prefix to get the data back.
Logically, I figure the best way of doing this is to do a nested for loop. Go through the series of prefixes adding the url fragment and doing the fetch call.
It works fine in local, but lambda throws errors.
function getVariantData(data, cb) {
for (var i = 0; i < chapters.length; i++) {
let source = chapters[i];
// chapters = url prefix
data.forEach(async element => {
let res = await fetch(chapters[i] + element);
//element = url fragment
let body = await res.text();
createVariantsFile(element, source, body, cb);
});
}
}
this code runs fine but i've learned that lambdas are a bit more strict with forEach and async/awaits so I've changed my code to this and I've been dealing with a mess of issues. I havent gone past writing console.log because...well I haven't gotten past the error.
async function getVariantData(data, cb) {
for (var i = 0; i < chapters.length; i++) {
let source = chapters[i];
const promises = data.map((datum, index) => fetch(source+datum))
const chapterData = await Promise.all(promises)
console.log(chapterData)
// await data.map(async element => {
// return await (chapters[i] + element);
// createVariantsFile(element, source, body, cb);
// });
}
}
I'm fetching the stylesheet and replacing all CSS variables with the actual hex value it corresponds to when the user changes the color as desired.
I created an event handler so that when the user clicks the download button, all of the colors he/she selected, would be saved in the stylesheet at that moment, but it doesn't seem to work. I know it's an issue with my understanding of promises as a whole and async await
What I did.
const fetchStyleSheet = async () => {
const res = await fetch("./themes/prism.css");
const orig_css = await res.text();
let updated_css = orig_css;
const regexp = /(?:var\(--)[a-zA-z\-]*(?:\))/g;
let cssVars = orig_css.matchAll(regexp);
cssVars = Array.from(cssVars).flat();
console.log(cssVars)
for await (const variable of cssVars) {
const trimmedVar = variable.slice(6, -1)
const styles = getComputedStyle(document.documentElement)
const value = String(styles.getPropertyValue(`--${trimmedVar}`)).trim()
updated_css = updated_css.replace(variable, value);
}
console.log(updated_css)
return updated_css
}
const main = async () => {
const downloadBtn = document.getElementById('download-btn')
downloadBtn.addEventListener('click', () => {
const updated_css = fetchStyleSheet()
downloadBtn.setAttribute('href', 'data:application/octet-stream;charset=utf-8,' + encodeURIComponent(updated_css))
downloadBtn.setAttribute('download', 'prism-theme.css')
})
}
main()
I can't await the updated_css because it falls into the callback of the click event, which is a new function.
Then I did the following thinking it would work since it was top level.
const downloadBtn = document.getElementById('download-btn')
downloadBtn.addEventListener('click', async () => {
const updated_css = await fetchStyleSheet()
downloadBtn.setAttribute('href', 'data:application/octet-stream;charset=utf-8,' + encodeURIComponent(updated_css))
downloadBtn.setAttribute('download', 'prism-theme.css')
})
That gave me the following error TypeError: NetworkError when attempting to fetch resource.
I understand that calling fetchStyleSheet() only returns a promise object at first and to get the value (which is updated_css), I need to follow it with .then() or await it.
The await is the correct approach to deal with the fetchStyleSheet() call returning a promise, your problem is that the click on the link tries to follow the href attribute immediately - before you set it to that data url. What you would need to do instead is prevent the default action, asynchronously do your stuff, and then re-trigger the click when you're done. Also don't forget to deal with possible exceptions:
const downloadBtn = document.getElementById('download-btn')
downloadBtn.addEventListener('click', async (event) => {
if (!e.isTrusted) return // do nothing on the second run
try {
event.preventDefault()
const updated_css = await fetchStyleSheet()
downloadBtn.setAttribute('href', 'data:application/octet-stream;charset=utf-8,' + encodeURIComponent(updated_css))
downloadBtn.setAttribute('download', 'prism-theme.css')
downloadBtn.click() // simulate a new click
} catch(err) {
console.error(err) // or alert it, or put the message on the page
}
})
I am trying to access return data from a do while loop, but I am unable to do so.
I have stored the information in a new variable (starships) and then returned this variable, but it says starships is not defined. I see that this may be a scoping issue, how can I resolve this?
async function getData() {
const allResults = [];
let url = 'https://swapi.co/api/starships/';
do {
const res = await fetch(url);
const data = await res.json();
url = data.next;
allResults.push(...data.results);
console.log(allResults);
} while (url !== null)
let starships = allResults;
return starships;
}
console.log(starships);
You need to get the value which is returned from getData. The most obvious way to do this with the async/await structure you have is to just await it:
async function getData() {
const allResults = [];
let url = 'https://swapi.co/api/starships/';
do {
const res = await fetch(url);
const data = await res.json();
url = data.next;
allResults.push(...data.results);
console.log(allResults);
} while (url !== null)
let starships = allResults;
return starships;
}
async function doTheDo() {
const test = await getData();
console.dir(test);
}
doTheDo();
you can do this. starships is defined inside the loop. Additionally, you are not calling getData() function. You can store that return value like this
const result = await getData();
console.log(result);
or you can directly print like this. console.log(await getData())
async function getData() {
const allResults = [];
let url = 'https://swapi.co/api/starships/';
do {
const res = await fetch(url);
const data = await res.json();
url = data.next;
allResults.push(...data.results);
console.log(allResults);
} while (url !== null)
return allResults;
}
console.log(await getData());
Async functions return a promise, which means you have to access the return value with a .then().
However, you have another problem: starships is in the scope of the function getData() which you have defined, but not called.
So first lets call your function:
async function getData() {
const allResults = [];
// do stuff
let starships = allResults;
return starships;
}
console.log(getData());
Now you will see that your log value is [object Promise] which isn't so helpful in its current form. This is because the code outside the async function is running synchronously, which means we don't have the value yet, just a promise to maybe return the value sometime in the future.
So now we need to access the promise asynchronously using the .then() like so:
async function getData() {
const allResults = [];
// do stuff
let starships = allResults;
return starships;
}
getData().then(starships => {
console.log(starships);
});
Now you should see the info you were expecting to be logged.
You can also save promise to a variable and pass it around and access it elsewhere in your code like so:
async function getData() {
const allResults = [];
// do stuff
let starships = allResults;
return starships;
}
let starshipPromise = getData();
// time & code passes...
starshipPromise.then(starship => {
console.log(starship);
}).catch(error => {
// handle error
});
And don't forget to catch your rejected promises!
See the MDN docs on Async functions: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
And if you need more info on promises, go here: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
I made a small webpage that takes information from the "Yahoo Weather" API and displays it in divs on the page.
This is the JS:
const url = 'https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20weather.forecast%20where%20woeid%20in%20(select%20woeid%20from%20geo.places(1)%20where%20text%3D%22nome%2C%20ak%22)&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys';
let data = 1;
const getWeather = async function() {
const fetchWeather = await fetch(url);
const result = await fetchWeather.json();
return data=result;
}
getWeather();
const showData = async function(info) {
let html = '';
const newInfo = info.query.results.channel.item.forecast.map((item, index) => {
html += `<div id='item${index}'><p>Date: ${item.date}</p>`;
html += `<p>Day: ${item.day}</p>`;
html += `<p>Highest temp: ${item.high}</p>`;
html += `<p>Lowest temp: ${item.low}</p></div>`;
return html;
});
}
const display = async function() {
const info = await showData(data);
weatherInfo.innerHTML = data;
}
display();
My goal is that when the page loads, it displays the information gathered from the promise returned by the API.
I get this error:Uncaught (in promise) TypeError: Cannot read property 'results' of undefined
Basically as far as I understand, by the time the "display()" is invoked, the variable "data" doesn't have anything in it yet.
What I'm trying to achieve is that display() will only work after "data" is defined, but without a for loop or something like that.
Any help will be appreciated!
As far as I understand, by the time the "display()" is invoked, the variable "data" doesn't have anything in it yet.
Yes. Don't use a global data variable at all1. getWeather returns a promise that will fulfill with the data, so you know exactly when it becomes available:
getWeather().then(display);
with
async function getWeather () {
const fetchWeather = await fetch(url);
const result = await fetchWeather.json();
return result; // drop the `data =` assignment
}
async function display(data) {
// ^^^^
const info = await showData(data);
weatherInfo.innerHTML = info;
}
Alternatively, especially when you don't want to use a then chain, just put const data = await getWeather(); in the display function.
1: If you insist to store the data in the global scope because you want to use it in multiple places, put the promise for the data in the variable not the result itself.
Here is my solutiuon:
const url = 'https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20weather.forecast%20where%20woeid%20in%20(select%20woeid%20from%20geo.places(1)%20where%20text%3D%22nome%2C%20ak%22)&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys';
const getWeather = async function() {
const fetchWeather = await fetch(url);
return await fetchWeather.json();
}
const showData = async function(info) {
let html = '';
info.query.results.channel.item.forecast.map((item, index) => {
html += `<div id='item${index}'><p>Date: ${item.date}</p>`;
html += `<p>Day: ${item.day}</p>`;
html += `<p>Highest temp: ${item.high}</p>`;
html += `<p>Lowest temp: ${item.low}</p></div>`;
});
return html;
}
const display = async function() {
const data = await getWeather();
const info = await showData(data);
weatherInfo.innerHTML = info;
}
display();
https://plnkr.co/edit/1b0fpBji7y6sZPODPDjY?p=preview