<div class="view-content">
<div class="views-row views-row-1">
<div class="views-field">
<span class="field-content">
<a href="link1">Name for link1
<img src="image1">
</a>
</span>
</div>
<div class="views-field-title">
<span class="field-content">
<a href="link1">
</a>
</span>
</div>
</div>
<div class="views-row views-row-2">
<div class="views-field">
<span class="field-content">
<a href="link2">Name for Link2
<img src="image2">
</a>
</span>
</div>
<div class="views-field-title">
<span class="field-content">
<a href="link2">
</a>
</span>
</div>
</div>
I am using node with request, and cheerio to request the data and scrape accordingly.
I am seeking the href from link1 and link2, I got it to work for one link, but it does not scale out when I try to loop it.
const data ={
link:"div.views-field > span > a"
},
pageData = {};
Object.keys(data).forEach(k => {
pageData[k] = $(data[k]).attr("href");});
console.log(pageData);
Your approach with $(data[k]).attr("href"); is the right idea, but there's no loop here. There should be 2 elements matching this selector but your code only grabs the first.
Changing this to a [...$(data[k])].map(e => $(e).attr("href")) lets you get href attributes from all matching elements.
I'm not crazy about pageData being global and using a forEach when a map seems more appropriate, so here's my suggestion:
const $ = cheerio.load(html);
const data = {
link: "div.views-field > span > a",
};
const pageData = Object.fromEntries(
Object.entries(data).map(([k, v]) =>
[k, [...$(v)].map(e => $(e).attr("href"))]
)
);
console.log(pageData);
Related
I'm trying to scrape some data for a fun personal project, and I'm new to JS. I'm trying to get an array of team names from an HTMLCollection (using Chrome Dev Tools). If there's an easier way I'm open to it. So far:
let vMidCollection = document.getElementsByClassName("v-mid");
vMidCollection[0]
QUESTION: Each <a></a> contains a <span></span> but I need an array of team names (underlined). Thanks in advance.
You can try this using querySelectorAll
const teamNames = [...document.querySelectorAll('.v-mid span.teamName')].map(e => e.textContent)
console.log(teamNames)
<a class="v-mid">
<div>
<span class="teamName">Test </span>
</div>
</a>
<a class="v-mid">
<div>
<span class="teamName">Test 1</span>
</div>
</a>
<a class="v-mid">
<div>
<span class="teamName">Test 2</span>
</div>
</a>
You can use the following
let vMidCollection = document.getElementsByClassName("v-mid");
team_names = []
Array.from(vMidCollection).forEach( (elem) => {team_names.push(elem.querySelector(".teamName").innerText)})
console.log(team_names)
<a class="v-mid">
<div>
Stuff
</div>
<span class="teamName">Tam 1 </span>
</a>
<a class="v-mid">
<div>
Stuff
</div>
<span class="teamName">Team 2</span>
</a>
<a class="v-mid">
<div>
Stuff
</div>
<span class="teamName">Team3</span>
</a>
It finds every element inside vMidCollection that has class="teamname" and appends it text to team_names
db.collection('Buses').get().then((snapshot) = > {
snapshot.forEach((busDatas) = > {
busData = busDatas.data()
console.log(busData)
document.getElementById('bus-container-dynamic').innerHTML = `
<div class="single-room-area d-flex align-items-center
mb-50 wow fadeInUp" data-wow-delay="100ms">
<div class="room-thumbnail">
<img src="${busData.ImageLink}" alt="">
</div>
<div class="room-content">
<h2>${busData.TourName}</h2>
<h6>${busData.From} to ${busData.To}</h6>
<h4>₹ ${busData.SeatPrice} </h4>
<div class="room-feature">
<h6>Boarding Point <span>${busData.BoardingTime}</span></h6>
<h6>Dropping Point <span>${busData.DroppingTime}</span></h6>
<h6>Seats Left <span>${busData.SeatsLeft}</span></h6>
<h6>Total Time <span>${busData.TotalTime}</span></h6>
</div>
<a href="#" class="btn view-detail-btn">
View Details
<i class="fa fa-long-arrow-right" aria-hidden="true"></i>
</a>
</div>
</div>`
})})
I am using this code to display my code in html but the only one document is showing on the webpage , but when i print that data in console i am getting all the documents
Do not overwrite the contents of the element on each iteration, append to them.
In fact, use a variable to append to, then assign that to the element, so you only have to manipulate the DOM once.
This line:
document.getElementById('bus-container-dynamic').innerHTML = `...`;
Keeps re-writing the whole contents of #bus-container-dynamic at each iteration.
You could instead store all the data in one variable, then assign that to the element.
A short snippet to illustrate the solution.
const myData = [1,2,3,4,5];
// Create a variable here
let html = '';
myData.forEach( e => {
// Create your element's HTML inside the loop
html += e;
});
// Then assign it to the element
document.getElementById('my-element').innerHTML = html;
<div id="my-element"></div>
And this is how I would modify the code that you posted originally.
db.collection('Buses').get().then((snapshot) = > {
let html = '';
snapshot.forEach((busDatas) = > {
busData = busDatas.data()
console.log(busData)
html += `
<div class="single-room-area d-flex align-items-center
mb-50 wow fadeInUp" data-wow-delay="100ms">
<div class="room-thumbnail">
<img src="${busData.ImageLink}" alt="">
</div>
<div class="room-content">
<h2>${busData.TourName}</h2>
<h6>${busData.From} to ${busData.To}</h6>
<h4>₹ ${busData.SeatPrice} </h4>
<div class="room-feature">
<h6>Boarding Point <span>${busData.BoardingTime}</span></h6>
<h6>Dropping Point <span>${busData.DroppingTime}</span></h6>
<h6>Seats Left <span>${busData.SeatsLeft}</span></h6>
<h6>Total Time <span>${busData.TotalTime}</span></h6>
</div>
<a href="#" class="btn view-detail-btn">
View Details
<i class="fa fa-long-arrow-right" aria-hidden="true"></i>
</a>
</div>
</div>`
document.getElementById('bus-container-dynamic').innerHTML = html;
}) // End foreach
}) // End then
I have this HTML template:
<div class="item">
<span class="light">Date</span>
<a class="link" href="">2018</a>
(4pop)
</div>
<div class="item">
<span class="light">From</span>
<span>
<a class="link" href="" title="Bob" itemprop="url"><span itemprop="name">Bob</span></a>,
</span>
<span>
<a class="link" href="" title="Peter" itemprop="url"><span itemprop="name">Peter</span></a>
</span>
</div>
<div class="item">
<span class="light">For</span>
<a class="link" href="">Bak</a>,
<a class="link" href="">Cam</a>,
<a class="link" href="">Oli</a>
</div>
<div class="item">
<span class="light">Nat</span>
<a class="link" href="">Cool</a>
</div>
</div>
And my Javascript code:
var doc = new DOMParser().parseFromString(HTMLContent,'text/html');
var infos = doc.evaluate('//div[#class="item"]/span[1]', doc, null, XPathResult.ANY_TYPE, null);
var nodes = [];
for(var node = infos.iterateNext(); node; node = infos.iterateNext()) {
nodes.push(node);
console.log(node.textContent);
// Until here, all things works well ! Except the code from here:
var nodde = node.nextElementSibling.attributes;
nodde.forEach(function(item){
console.log(item);
});
}
My goal is to get the respective value for each categorie, for example:
Date = 2018, (4pop)
From = Bob, Peter
For = Bak, Cam, Oli
Nat = Cool
I tried to iterate: node.nextElementSibling.attributes but without any success !
Is there a way to get the expected result please ?
The HTML Document Object Model, DOM, is preferred as it's easier. XPATH is geared toward more rigid XML documents.
Using JavaScript, you would get data with something like:
var users = document.querySelectorAll('[itemprop=name]').textContent;
console.log(users);
In a hidden list I have a variable list with this data (in this example www.domain.com/2009 is the current URL):
<ul id="WalkingYears" style="visibility: hidden; display:none;">
<li id="Walk2011"><img src="some-imga.jpg"></li>
<li id="Walk2010"><img src="some-imgs.jpg"></li>
<li id="Walk2008"><img src="some-imgf.jpg"></li>
<li id="Walk2007"><img src="some-imgg.jpg"></li>
<li id="Walk2006"><img src="some-imgh.jpg"></li>
<li id="Walk2005"><img src="some-imgj.jpg"></li>
<li id="Walk2004"><img src="some-imgk.jpg"></li>
<li id="Walk2003"><img src="some-imgl.jpg"></li>
<li id="Walk2002"><img src="some-imgz.jpg"></li>
<li id="Walk2001"><img src="some-imgx.jpg"></li>
</ul>
The above list is auto-generated and I can change this if I like; for example into:
<div id="Walk2011" data-target="http://domain.com/2011" data-img="some-imga.jpg" data-title="2011"></div>
<div id="Walk2010" data-target="http://domain.com/2010" data-img="some-imgs.jpg" data-title="2010"></div>
<div id="Walk2008" data-target="http://domain.com/2008" data-img="some-imgd.jpg" data-title="2008"></div>
<div id="Walk2007" data-target="http://domain.com/2007" data-img="some-imgf.jpg" data-title="2007"></div>
<div id="Walk2006" data-target="http://domain.com/2006" data-img="some-imgg.jpg" data-title="2006"></div>
<div id="Walk2005" data-target="http://domain.com/2005" data-img="some-imgh.jpg" data-title="2005"></div>
<div id="Walk2004" data-target="http://domain.com/2004" data-img="some-imgj.jpg" data-title="2004"></div>
<div id="Walk2003" data-target="http://domain.com/2003" data-img="some-imgk.jpg" data-title="2003"></div>
<div id="Walk2002" data-target="http://domain.com/2002" data-img="some-imgl.jpg" data-title="2002"></div>
<div id="Walk2001" data-target="http://domain.com/2001" data-img="some-imgz.jpg" data-title="2001"></div>
You see that the current URL (www.domain.com/2009) is not showing in this list.
Now I'd like to fill the prev and next navigation, based on the current url, using the values mentioned above (title, href, image src):
<a href="http://domain.com/2008" title="2008" id="balk-prev-btn" class="prev-btn left">
<img src="some-imgd.jpg" alt="2008">
<span class="icon"></span>
</a>
<a href="http://domain.com/2010" title="2010" id="balk-next-btn" class="next-btn right">
<img src="some-imgs.jpg" alt="2010">
<span class="icon"></span>
</a>
I guess I need to
first find out what the current URL is
then compare it to the data in the list
somehow point out the prev and next page
Also when having selected a certain variable (the name of a walker) the links in the list will be different and the URL will be www.domain.com/walkername/2009:
<div id="Walk2011" data-target="http://domain.com/walkername/2011" data-img="some-imga.jpg" data-title="2011"></div>
<div id="Walk2010" data-target="http://domain.com/walkername/2010" data-img="some-imgs.jpg" data-title="2010"></div>
<div id="Walk2008" data-target="http://domain.com/didnotwalk/2008" data-img="some-imgd.jpg" data-title="2008"></div>
<div id="Walk2007" data-target="http://domain.com/didnotwalk/2007" data-img="some-imgf.jpg" data-title="2007"></div>
<div id="Walk2006" data-target="http://domain.com/walkername/2006" data-img="some-imgg.jpg" data-title="2006"></div>
<div id="Walk2005" data-target="http://domain.com/didnotwalk/2005" data-img="some-imgh.jpg" data-title="2005"></div>
<div id="Walk2004" data-target="http://domain.com/didnotwalk/2004" data-img="some-imgj.jpg" data-title="2004"></div>
<div id="Walk2003" data-target="http://domain.com/walkername/2003" data-img="some-imgk.jpg" data-title="2003"></div>
<div id="Walk2002" data-target="http://domain.com/didnotwalk/2002" data-img="some-imgl.jpg" data-title="2002"></div>
<div id="Walk2001" data-target="http://domain.com/didnotwalk/2001" data-img="some-imgz.jpg" data-title="2001"></div>
In this case the prev and next button should only show the links with the walker name in it :) and should look like this:
<a href="http://domain.com/walkername/2006" title="2006" id="balk-prev-btn" class="prev-btn left">
<img src="some-imgg.jpg" alt="2006">
<span class="icon"></span>
</a>
<a href="http://domain.com/walkername/2010" title="2010" id="balk-next-btn" class="next-btn right">
<img src="some-imgs.jpg" alt="2010">
<span class="icon"></span>
</a>
Can someone help me?
tnx!
Okay so if you have this layout, this script should do the job
<div id="Walk2011" data-target="http://domain.com/walkername/2011" data-img="some-imga.jpg" data-title="2011"></div>
<div id="Walk2010" data-target="http://domain.com/walkername/2010" data-img="some-imgs.jpg" data-title="2010"></div>
<div id="Walk2008" data-target="http://domain.com/didnotwalk/2008" data-img="some-imgd.jpg" data-title="2008"></div>
<div id="Walk2007" data-target="http://domain.com/didnotwalk/2007" data-img="some-imgf.jpg" data-title="2007"></div>
<div id="Walk2006" data-target="http://domain.com/walkername/2006" data-img="some-imgg.jpg" data-title="2006"></div>
<div id="Walk2005" data-target="http://domain.com/didnotwalk/2005" data-img="some-imgh.jpg" data-title="2005"></div>
<div id="Walk2004" data-target="http://domain.com/didnotwalk/2004" data-img="some-imgj.jpg" data-title="2004"></div>
<div id="Walk2003" data-target="http://domain.com/walkername/2003" data-img="some-imgk.jpg" data-title="2003"></div>
<div id="Walk2002" data-target="http://domain.com/didnotwalk/2002" data-img="some-imgl.jpg" data-title="2002"></div>
<div id="Walk2001" data-target="http://domain.com/didnotwalk/2001" data-img="some-imgz.jpg" data-title="2001"></div>
jQuery based script:
var xlocation = "http://www.domain.com/walkername/2009".match(/(\/[a-zA-Z]+\/)(\d+)/); //sorry for ugly regexp --> ["/walkername/2009", "/walkername/", "2009"], also here should be used window.location.href , but for example lets use static string;
//find and filter only links which have 'walkername' in data-tagert
$el = $('#WalkingYears div[id^=Walk]').filter(function(i,el){
return $(el).attr('data-target').indexOf(xlocation[1]) > 0;
}),
//sort if divs is scrambeled
$elSorted = $el.sort(sorter);
prev = jQuery.grep($elSorted,function(el,i){
return $(el).attr('data-title').replace(/^\D+/g, '')*1<xlocation[2]*1
})
next = jQuery.grep($elSorted,function(el,i){
return $(el).attr('data-title').replace(/^\D+/g, '')*1>xlocation[2]*1
})
var sorter = function(a,b){
var a = $(a).attr('data-title').replace(/^\D+/g, '')*1,
b = $(b).attr('data-title').replace(/^\D+/g, '')*1
return b-a
}
//ADD href to buttons...
$('#balk-prev-btn').prop('href',$(prev).first().attr('data-target'))
$('#balk-next-btn').prop('href',$(next).last().attr('data-target'))
You`ll need to check if prevEl and NextEl still exists in case if current page is first or last. Also you will need to review regexp used for parsing url :)
I have a piece of HTML like this:
<div id="contentblock">
<div id="producttile_137" class="producttile">
<a href="#" class="tile">
<img src="images/Bony A-Booster.jpg" alt="Bony A-Booster - 50 ml">
Bony A-Booster
<span class="price">€10.95</span>
</a>
</div>
<div id="producttile_138" class="producttile">
<a href="#" class="tile">
<img src="images/Bony B-Booster.jpg" blt="Bony B-Booster - 50 ml">
Bony B-Booster
<span class="price">€20.95</span>
</a>
</div>
<div>Aditional info</div>
</div>
I need to get all <img /> sources but with pure Javascript. I can get element by class name document.getElementsByClassName('producttile') but is it possible to traversing in pure JS to <img /> ang get src="" value?
You could use :
document.getElementsByClassName('class_name');
//OR
document.getElementsByTagName('img');
//OR
document.querySelectorAll('img');
All the previous methods are pure javascript and return nodes list so you could loop through them to get the src of every node.
Hope this helps.
You can use the function getElementsByTagName.
function listImages() {
var img = document.getElementsByTagName('img');
for (var i in img) {
if (img[i].src) {
console.log(img[i].src);
}
}
}
<div id="contentblock">
<div id="producttile_137" class="producttile">
<a href="#" class="tile">
<img src="images/Bony A-Booster.jpg" alt="Bony A-Booster - 50 ml">
Bony A-Booster
<span class="price">€10.95</span>
</a>
</div>
<div id="producttile_138" class="producttile">
<a href="#" class="tile">
<img src="images/Bony B-Booster.jpg" blt="Bony B-Booster - 50 ml">
Bony B-Booster
<span class="price">€20.95</span>
</a>
</div>
<div>Aditional info</div>
</div>
Show Images
one possible solution to get img and the src value:
var imgs = document.querySelectorAll('img')
var res = [].slice.call(imgs).map(x=>x.getAttribute("src"))
//or in es5
//.map(function(x){return x.getAttribute("src")})
//[].slice.call is necessary to transform nodeList in array
//otherwise you can use an regular for loop
console.log(res)
<div id="contentblock">
<div id="producttile_137" class="producttile">
<a href="#" class="tile">
<img src="images/Bony A-Booster.jpg" alt="Bony A-Booster - 50 ml">
Bony A-Booster
<span class="price">€10.95</span>
</a>
</div>
<div id="producttile_138" class="producttile">
<a href="#" class="tile">
<img src="images/Bony B-Booster.jpg" blt="Bony B-Booster - 50 ml">
Bony B-Booster
<span class="price">€20.95</span>
</a>
</div>
<div>Aditional info</div>
</div>
wanting to directly select an element based on it's src value you could also do this :
document.querySelector('[src="images/Bony A-Booster.jpg"]')
(assuming there is one element you can use querySlector() instead of querySelectorAll() )
You can use document.getElementsByTagName() to fetch all the img tags as a HTMLCollection. And then, you can iterate over the HTMLCollection and get the src attribute value by using getAttribute() method as below:
const imgElements = document.getElementsByTagName("img");
Array.from(imgElements).forEach(function(element){
console.log(element.getAttribute("src"));
});