Develop Reference

Python Selenium Scraping Javascript - Element not found

I am trying to scrape the following Javascript frontend website to practise my Javascript scraping skills:
https://www.oplaadpalen.nl/laadpaal/112618
I am trying to find two different elements by their xPath. The first one is the title, which it does find. The second one is the actual text itself, which it somehow fails to find. It's strange since I just copied the xPath's from Chrome browser.
from selenium import webdriver
link = 'https://www.oplaadpalen.nl/laadpaal/112618'
driver = webdriver.PhantomJS()
driver.get(link)
#It could find the right element
xpath_attribute_title = '//*[#id="main-sidebar-container"]/div/div[1]/div[2]/div/div[' + str(3) + ']/label'
next_page_elem_title = driver.find_element_by_xpath(xpath_attribute_title)
print(next_page_elem_title.text)
#It fails to find the right element
xpath_attribute_value = '//*[#id="main-sidebar-container"]/div/div[1]/div[2]/div/div[' + str(3) + ']/text()'
next_page_elem_value = driver.find_element_by_xpath(xpath_attribute_value)
print(next_page_elem_value.text)
I have tried a couple of things: change "text()" into "text", "(text)", but none of them seem to work.
I have two questions:
Why doesn't it find the correct element?
What can we do to make it find the correct element?

Selenium's find_element_by_xpath() method returns the first element node matching the given XPath query, if any. However, XPath's text() function returns a text node—not the element node that contains it.
To extract the text using Selenium's finder methods, you'll need to find the containing element, then extract the text from the returned object.

Keeping your own logic intact you can extract the labels and the associate value as follows :
for x in range(3, 8):
label = driver.find_element_by_xpath("//div[#class='labels']//following::div[%s]/label" %x).get_attribute("innerHTML")
value = driver.find_element_by_xpath("//div[#class='labels']//following::div[%s]" %x).get_attribute("innerHTML").split(">")[2]
print("Label is %s and value is %s" % (label, value))
Console Output :
Label is Paalcode: and value is NewMotion 04001157
Label is Adres: and value is Deventerstraat 130
Label is pc/plaats: and value is 7321cd Apeldoorn

I would suggest a slightly different approach. I would grab the entire text and then split one time on :. That will get you the title and the value. The code below will get Paalcode through openingstijden labels.
for x in range(2, 8):
s = driver.find_element_by_css_selector("div.leftblock > div.labels > div")[x].text
t = s.split(":", 1)
print(t[0]) # title
print(t[1]) # value
You don't want to split more than once because Status contains more semicolons.

Going with #JeffC's approach, if you want to first select all those elements using xpath instead of css selector, you may use this code:
xpath_title_value = "//div[#class='labels']//div[label[contains(text(),':')] and not(div) and not(contains(#class,'toolbox'))]"
title_and_value_elements = driver.find_elements_by_xpath(xpath_title_value)
Notice the plural elements in the find_elements_by_xpath method. The xpath above selects div elements that are descendants of a div element that had a class attribute of "labels". The nested label of each selected div must contain a colon. Furthermore, the div itself may not have a class of "toolbox" (Something that certain other divs on the page have), nor must it contain any additional nested divs.
Following which, you can extract the text within the individual div elements (which also contain the text from the nested label elements) and then split them using ":\n" which separates the title and value in the raw text string.
for element in title_and_value_elements:
element = element.text
title,value = element.split(":\n")
print(title)
print(value,"\n")

Since you want to practice JS skills you can do this also in JS, actually all the divs contain more data, you can see if you do paste this in the browser console:
labels = document.querySelectorAll(".labels");
divs = labels[0].querySelectorAll("div");
for (div of divs) console.log(div.firstChild, div.textContent);
you can push to an array and check only divs and that have label and return the resulted array in a python variable:
labels_value_pair.driver.execute_script('''
scrap = [];
labels = document.querySelectorAll(".labels");
divs = labels[0].querySelectorAll("div");
for (div of divs) if (div.firstChild.tagName==="LABEL") scrap.push(div.firstChild.textContent, div.textContent);
return scrap;
''')

I am getting empty values for sports-title and third

i am new to js.
can you tell me why I am getting empty values for sports-title and third.
since we have one div with content in it.
sports-title---->{"0":{}}
third---->{}
providing my code below.
findStringInsideDiv() {
/*
var str = document.getElementsByClassName("sports-title").innerHTML;
*/
var sportsTitle = document.getElementsByClassName("sports-title");
var third = sportsTitle[0];
var thirdHTML = third.innerHTML
//str = str.split(" ")[4];
console.log("sports-title---->" + JSON.stringify(sportsTitle));
console.log("third---->" + JSON.stringify(third));
console.log("thirdHTML---->" + JSON.stringify(thirdHTML));
if ( thirdHTML === " basketball football swimming " ) {
console.log("matching basketball---->");
var menu = document.querySelector('.sports');
menu.classList.add('sports-with-basketball');
// how to add this class name directly to the first div after body.
// but we are not rendering that div in accordion
//is it possible
}
else{
console.log("not matching");
}
}

When you call an object in the Document Object Model (DOM) using any of the GetElement selectors, it returns an object that can be considered that HTML element. This object includes much more than just the text included in the HTML element. In order to access the text of that element, you want to use the .textContent property.
In addition, an HTML class can potentially be assigned to several elements and therefore GetElementsByClassName returns an array so you would have to do the following, for example:
console.log("sports-title---->" + JSON.stringify(sportsTitle[0].textContent));
You can find a brief introduction to the DOM on the W3Schools Website. https://www.w3schools.com/js/js_htmldom.asp If you follow along it gives an overview of different aspects of the DOM including elements.

Maybe this would be helpful
As you see sportsTitle[0].textContent returns full heading and 0 is the index thus you get "0" when you stringify (serialize) sportsTitle. Why 0? Because you have one <h1> element . See this fiddle http://jsfiddle.net/cqj6g7f0/3/
I added second h1 and see the console.log and you get two indexes 0 and 1
if you want to get a word from element so get substring use substr() method https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/substr
One way is to change <h1> class attr to id and do sportsTitle.textContent;
and use substr() on this string
or
2nd way is to remain class attr and do sportsTitle[0].textContent;
and substr() on this string
The 2nd is the better way

Add child to E4X with my variable

This is using Mirth Connect which uses E4x and js.
Basically I have a variable that I want to populate the XML with.
var memberid = "1234";
var fieldsxml = new XML(<fieldvaluelist></fieldvaluelist>);
fieldsxml.field += <fieldvalue templatefieldid="446" value=#memberid/> //memberID
But its giving an error on the 3rd line: (I also tried just memberid without quotes)
DETAILS: TypeError: Open quote is expected for attribute "value"
associated with an element type "fieldvalue".
It works if the third line is this:
fieldsxml.field += <fieldvalue templatefieldid="446" value="memberid"/>
But that just adds the literal string "memberid" . I actually want value="1234" instead.
How can I do this?
Edit: The final XML should look like this.
<fieldvaluelist><fieldvalue templatefieldid="446" value="1234"/></fieldvaluelist>

You're almost there. Instead of using #memberId, use {memberId}:
fieldsxml.field += <fieldvalue templatefieldid="446" value={memberid}/>;

How to return certain number value with java script from html

I need custom javascript code for Google tag manger to create variable for transaction value. To do this I need a code that return number 30.99 from HTML:
<span class="price-data">zł30.99</span>
I wrote some code but it doesn’t work:
function() {
var orderValue = document.getElementsByClassName(‘price-data')[0].value;
return orderValue.match((\d*\.\d*)|(\d*));
}
Could you help me with that

See the following snippet. Check the console value.
var orderValues = document.getElementsByClassName('price-data');
var orderValue = orderValues[0].innerHTML;
var match = orderValue.match(/([\d\.]+)/);
console.log(match[0]);
<span class="price-data">zł30.99</span>

"I need custom javascript code for Google tag manger " - no, you don't. You need a DOM type variable with the selector method set to "CSS selector" and the selector set to your class name. This will return the text content of the first element that matches the selector.
Since GTM already carries a selector engine around you really gain nothing by writing custom Javascript.

Get last child of div in Javascript function [duplicate]

This question already has answers here:
How to select last child element in jQuery?
(6 answers)
Closed 9 years ago.
I am taking a fairly simple piece of code and wrapping into a function, the code builds a slide out menu from the items inside a div. I'm now trying to get the last child from a div:
experimentOne('#experimentOne');
function experimentOne(masterContainer) {
var experimentOneMenuButton = $('masterContainer :last-child');
... etc
However, this returns [] in the log. When I check the width, instead of the 100px it should be, it's 1420 which I'm guessing is the window width.
If I just get the last child of the ID by coding it in everything is fine, but I want the function to be as reusable as possible.
Thanks.

Try this if your masterContainer is an element:
var experimentOneMenuButton = $(':last-child', masterContainer);
Or this if it is a string for the element id
var experimentOneMenuButton = $('#'+masterContainer+' :last-child');

masterContainer is a variable containing the value of the parent container, so when you build the selector you need use it with string concatenation.
var experimentOneMenuButton = $(masterContainer + ' :last-child');
or you can use a context based lookup
var experimentOneMenuButton = $(':last-child', masterContainer);
I would recommend the second solution which make use of context bases lookup since it is neater.
Demo: Fiddle

element with id is selected with "#"
you need to target immediat children with ">", [space] target all descendants of element
$('#masterContainer>:last-child');

Well masterContainer inside the string will do nothing. I think it should be:
$(masterContainer).find(':last-child');
Also note that experimentOne('#experimentOne'); will send a string not a jQuery object. That would require experimentOne($('#experimentOne'));, which would leave:
experimentOne($('#experimentOne'));
function experimentOne(masterContainer) {
var experimentOneMenuButton = masterContainer.find(':last-child');
// Note the lack of jQuery wrapper around masterContainer in this version
Depends on whether you want to force a jQuery object in the parameter or in the function itself!

Your concatenation is wrong. Try that way:
$(masterContainer + ' :last-child')