I tried scraping with cheerio and everything was working. Then I tried to make the scraped links into working links by adding in the base url to the links, but suddenly this error showed up.
ReferenceError: Cannot access 'links' before initialization
I checked if I was getting the data from the site and axios side is working. But for some reason, cheerio stopped working. When I tried logging it gave me an empty array of elements like this,
LoadedCheerio(136) [Element, Element, Element, Element, Element]
I don't understand why. I need help to figure out about the problem.
Here's my code:
import axios from 'axios';
import * as cheerio from 'cheerio';
const baseUrl = 'https://gogoanime.tel/anime-list.html';
axios.get(baseUrl).then(res => {
const html = res.data;
const $ = cheerio.load(html);
const list = $('.listing a');
const links = list.each(function (i, el) {
$(el).attr('href');
});
});
Related
I have to build a page where the infos of countries (such as name, population, etc) are displayed using an API. I have managed to show the request results on the console using fetch, but I don't know how to display the results in the HTML page. I'm a beginner and very unexperienced. I know very little js.
I've seen other topics like this one, but they didn't really help solve my problem. One of them had an answer that suggested a code like the following, which worked fine until line 7 of the js script. However after adding the rest of the code I get an error that says Uncaught (in promise) TypeError: data is undefined. I don't understand why it says data is undefined when it has been used before.
function displayAll() {
let countriesList = [];
let url = "https://restcountries.com/v3.1/all";
fetch(url).then(function(response){
response.json().then(function(data) {
console.log(data); // works fine up until here
})
.then((data) => { // error says that 'data' is undefined
let countries = document.getElementById("countries-container");
data.results.forEach((item) => {
countriesList.push(item.name);
let div = document.createElement("div");
div.innerText = item.name;
countries.appendChild(div);
})
console.log(countriesList)
})
});
}
<body onload="displayAll()">
<header>Countries</header>
<div class="container" id="countries-container">
<!-- This is where the results will be displayed -->
</div>
<script src="./assets/scripts.js"></script>
</body>
I want to add new elements to serve as cards (one for each country).
Something tells me there will be more errors because the results printed in the console seem a bit too complicated to iterate, since it seems to be a matrix:
I'm not using any frameworks, since I still need to learn more about the basics before jumping into the more fancy stuff. The code is just HTML, CSS and JavaScript and I want to keep it that way if possible. I've tried learning React alreaddy and I think I'm not ready yet... Anyway, if anybody can help me with this, I'll be very grateful.
I've managed to solve the problem. Adding return data to the first .then solved the data undefined error, but then I got another error: TypeError: cannot read properties of undefined (reading forEach). This error disappeared after I deleted results from data.results.forEach
I could finally make the data appear in the HTML page, but it was shown as [object Object] and JSON.parse() would thow another error about an unexpected character somewhere in the original data. So I realized that I had to create the HTML elements to be able to show the data in the page. Obvious, I know. But as I said: I'm a total beginner.
So with that in mind, I created the following code and it works just fine!
Turns out I didn't really need a list of all the countries, so I eliminated that from the code.
// Searches and shows the data of all the countries in the homepage upon loading
function displayAll() {
let url = "https://restcountries.com/v3.1/all";
fetch(url).then(function(response){
response.json().then(function(data) {
console.log(data);
return data;
})
.then((data) => {
data.forEach((item) => {
showCountryHome(item);
})
})
});
}
// Creates a card for each country with the information I want
function showCountryHome(data) {
let countries = document.getElementById("countries-wrapper");
let div = document.createElement("div");
div.className = "card";
div.innerHTML = `<img src="${data.flags.svg}" width=200>
<p>${data.name.common}</p>
<p>Population: ${data.population}</p>
<p>Region: ${data.region}</p>
<p>Capital: ${data.capital}</p>`
countries.appendChild(div);
}
I’m running into an issue with generating dynamic pages in Next.js. I’m pulling data from Sanity, and I think I’ve got the code right, but every time I try load the page I get a type error - “the ‘id’ argument must be of type string. Received null”. The file path is ‘/post/[slug].js’, so that’s the variable name I’ve used throughout - I’m not sure where id is coming from, or if it’s something extra I need to pass, or if I’m not passing my slug right!
I’ve asked in the Sanity forums, and it doesn’t seem to be an API issue - I’m able to pull data in other parts of the app with no problem. In the console, it seems like the page compiles successfully, but this error comes up when attempting to load it. I’ve tried it with an empty div in the page component, to make sure it’s nothing in the presentation logic, with no luck.
I’ve put the full error message in a gist, and it looks like something with jest or next-server. For the life of me I can’t figure it out!
import client from '../../client’
const Post = ({ post }) => {
...
}
export default Post
export const getStaticPaths = async () => {
const posts = await client.fetch(`*[_type == "blog"]{ slug }`)
const paths = posts.map(post => ({
params: {
slug: post.slug.current
}
}))
return {
paths,
fallback: false }
}
export const getStaticProps = async ({ params }) => {
const post = await client.fetch(`*[_type == "blog" && slug.current == $slug]{ title, date, link, excerpt, body, slug }[0]`, {slug: params.slug})
return {
props: { post }
}
}
UPDATE: this seems to be an issue with either play.js or the configuration it uses - trying in a desktop environment doesn’t result in the same error.
I think the issue is your file name. the name should only be [slug].js, it should be located in pages/post. It should work perfectly when you access the page via the URL http://localhost:xxx/post/pulled-sanity-slug
I am trying to webscrape walmart's products. Here is the link I am trying to pull https://www.walmart.com/search/?query=&cat_id=91083 I am able to successfully scrape like 10 products from the page. Here is the code I am using.
const axios = require('axios');
const cheerio = require('cheerio');
axios.get('https://www.walmart.com/search/?query=&cat_id=91083').then( res => {
var combino1 = [];
const $ = cheerio.load(res.data);
$('a.product-title-link').each( (index, element) => {
const name = $(element)
.first().text()
combino1[index] = {name}
})
console.log(combino1);
})
When I search the dom with a.product-title-link it shows 40 products. Why I am able to only grab 10 and not 40?
Your issue is that a call with axios will only get you the HTML provided from the server
this means that any asynchronous calls that fetch products from other parts of their system, will never be in that request
a simple output of the data received to a new file, will show this fact
const fs = require('fs')
...
fs.writeFileSync('./data.html', res.data)
opening the new data.html file will only output 10 as the number of product-title-link found
For that you can't use axios but a web scraper library, for example, Puppeteer as with it, you can wait for all products to be loaded prior to transverse the DOM at that given time.
i hope you care. I have troble when scraping data in another website. this my case, when i see in page source data is empty, but when i see in inspect element i see the data. if you not understand see these image, first in inspect element and second in view page source
this is my code
const request = require('request');
const cheerio = require('cheerio');
const url = "https://pikobar.jabarprov.go.id/"
request(url, (error, response, html) => {
let $ = cheerio.load(html);
$('b').each((i, element) => {
let omg = $(element).text();
console.log(omg);
});
});
this is my result
i want text 388
If you see the document source (Ctrl+U), this element is really empty. That's because it's being filled afterwards, by a Ajax request. If you go to Network tab and reload, you will see all files being transferred. The url you are looking for is https://covid19-public.digitalservice.id/api/v1/rekapitulasi/jabar?level=prov which have the data you are looking for.
fetch("https://covid19-public.digitalservice.id/api/v1/rekapitulasi/jabar?level=prov")
.then(r => r.json())
.then(d => console.log(d.data.content.positif))
I am currently doing a course on Udemy on JavaScript and i have been stuck on a project for a couple of weeks, I have followed the instruction to the best of my abilities many times but i seem to be missing a vital part that is stopping the project form displaying an object.
The purpose of this project to display cocktail drinks and the ingredients when you submit your favourite cocktail into a form. I have four JavaScript files app.js this is the main section of the JS code, then CocktailAPI.js this holds a class that handles the API queries, then we have UI.js this is for the interface behaviour and lastly the part i have not yet reached is the COcktailDB.js.
The problem i am facing is that i have created the class that handles the API request and the tutor starts by converting it into json and then turn the response into a object which is then logged on the console log to prove that everything is working fine.The problem i am facing is that even though i have copied the tutor the object does not display on my console and i get a error message that reads:
Access to fetch at 'http://www.thecocktaildb.com/api/json/v1/1/search.php?s=vodka' from origin 'null' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
I have checked the url many times and even copied and paste the url to eliminate that as the problem, i have solved this problem before but ended up starting the project again because i encountered another problem and though starting again would solve it. However when i reached this point again i didn't not know what i done to solve the problem as i tried so many things.
As there is 2 files that are currently in use at the moment as i am still quite early into the project there is nothing on 2 files so i will only post the 2 js files.If this is not enough please let me know what i need to add.
app.js
//Instanciate the classes
const ui = new UI(),
cocktail = new CocktailAPI();
//Create the event listeners
function eventListeners() {
//Add event listeners when the form is submittted
const searchForm = document.querySelector('#search-form');
if (searchForm) {
searchForm.addEventListener('submit', getCocktails);
}
}
eventListeners();
//Get cocktail function
function getCocktails(e) {
e.preventDefault();
const searchTerm = document.querySelector('#search').value;
//Check something is on the search input
if (searchTerm === '') {
ui.printMessage('Please add something intot the form', 'danger');
} else {
//Query by name of the drink
cocktail.getDrinkByName(searchTerm)
.then(cocktails => {
console.log(cocktails);
})
}
}
Cocktail.js
class UI {
//Display a custom message
printMessage(message, className) {
const div = document.createElement('div');
//Add the HTML
div.innerHTML = `
<div class="alert alert-dismissable alert-${className}">
<button type="button" class="close" data-dismiss="alert">X</button>
${message}
</div>
`;
//Insert befrore
const reference = document.querySelector('.jumbotron h1');
const parentNode = reference.parentElement;
parentNode.insertBefore(div, reference);
//Remove after 3 seconds
setTimeout(() => {
document.querySelector('.alert').remove();
}, 3000);
}
}
cocktailAPI.js
class CocktailAPI {
//Get recipe by name
async getDrinkByName(name) {
//Search by name
const apiResponse = await fetch(`http://www.thecocktaildb.com/api/json/v1/1/search.php?s=${name}`);
//returns json respnse
cocktails = await apiResponse.json();
return {
cocktails
}
}
}
It may become clearer what i am trying to produce when you load up all the files
I understand that this may not be enough information but if you ask me i will be able to explain in more detail, but the code is mean to display the response of the API in the console log with all the properties
I will copy the code so that it can be viewed if any one want the files itself to look into more deeply.
This is happening because you are using http to communicate with the API. If you change http://www.thecocktaildb.com/api/json/v1/1/search.php?s=${name} to https://www.thecocktaildb.com/api/json/v1/1/search.php?s=${name} (notice the https) it should work for you.
class CocktailAPI {
//Get recipe by name
async getDrinkByName(name) {
try {
//Search by name
const apiResponse = await fetch(`https://www.thecocktaildb.com/api/json/v1/1/search.php?s=${name}`);
//returns json respnse
let cocktails = await apiResponse.json();
return { cocktails };
} catch (err) {
throw err;
}
}
}
let cocktails_api = new CocktailAPI();
cocktails_api
.getDrinkByName("vodka")
.then(vodka => { console.log(vodka) })
.catch(err => { console.log("ERR", err) });
Your request for the JSON data must be served over https:// and not http:// since thecocktaildb.com most probably added an Access-Control-Allow-Origin wildcard that only accept https requests and not http requests in their response header.
Just replace the http in your fetch request with an https like this:
fetch(`https://www.thecocktaildb.com/api/json/v1/1/search.php?s=${name}`);
Check and run the following code snippet for a practical example of the above:
async function hello(name) {
const apiResponse = await fetch(`https://www.thecocktaildb.com/api/json/v1/1/search.php?s=${name}`);
let cocktails = await apiResponse.json();
console.log(cocktails);
return cocktails;
}
hello("vodka");