So I'm trying to craft a website manipulation script to help automate teh creation of email mailboxes on our hosted provider.
I'm both new to Python and new to scripting web resources so if something looks weird or mediocre that's why :)
Here's my script:
import time
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
from selenium import webdriver
from selenium.webdriver.support.select import Select as driverselect
driver = webdriver.Firefox()
main_url = 'https://website.com:446'
opts = Options()
# noinspection PyDeprecation
# opts.set_headless()
#assert opts.headless # Operating in headless mode
browser = Firefox(options=opts)
browser.get(main_url)
search_form = browser.find_element_by_id('LoginName')
search_form.send_keys('username')
search_form = browser.find_element_by_id('Password')
search_form.send_keys('password')
search_form.submit()
time.sleep(5)
# provision = driverselect(driver.find_element_by_xpath("/html/body/div[2]/div[2]/nav/div/ul/li[4]"))
provision = driver.find_element_by_xpath('/html/body/div[2]/div[2]/nav/div/ul/li[4]/a/span[1]')
provision.submit()
# exchange = driver.find_element_by_name('Exchange')
# exchange.submit()
My error is:
Traceback (most recent call last): File
"/home/turd/PycharmProjects/Automate_NSGEmail/selenium_test.py", line
23, in provision =
driver.find_element_by_xpath('/html/body/div[2]/div[2]/nav/div/ul/li[4]/a/span1')
File "/home/turd/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py",
line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath) File "/home/turd/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py",
line 976, in find_element
return self.execute(Command.FIND_ELEMENT, { File "/home/turd/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py",
line 321, in execute
self.error_handler.check_response(response) File "/home/turd/.local/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py",
line 242, in check_response
raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: Unable to
locate element: /html/body/div[2]/div[2]/nav/div/ul/li[4]/a/span1
Now that Xpath value I copied straight from the dev tools on that page, here's what this block of code looks like from the site:
I'm trying to grab and 'click' on the one Active Dynamic-Menu item in the pic above. I think that menu is JS but I'm not 100% positive.
Anyway I'd be much obliged if anyone could help me narrow this down and grab that blasted element.
So I discovered the answer myself.. I had some wrong code at the beginning of my script:
driver = webdriver.Firefox()
main_url = 'https://website.com:446'
opts = Options()
# noinspection PyDeprecation
# opts.set_headless()
#assert opts.headless # Operating in headless mode
browser = Firefox(options=opts)
browser.get(main_url)
I changed this section to:
driver = webdriver.Firefox()
url = 'https://website.com:446'
opts = Options()
driver.maximize_window()
driver.get(url)
I was opening two instances of Firebox before, the driver.* lines would attempting to locate the xpath tags on the FF instance that was not logged in.
Derp.
Related
I am attempting to automate inputting values into a webpage. However, the major issue is that the Mechanize library does not work because my webpage has no forms that Mechanize's form.name recognizes. This is due to the input being a <input>.
I have spent the past hour researching alternatives to Mechanize that might work, but to no avail. Google is of no help as it only thinks I want to take data from a website.
My current code:
from mechanize import Browser
import csv
csv_file = 'city_names.csv' # file name
cities = [] # array to save values from csv into
with open(csv_file, 'r') as f:
reader = csv.DictReader(f)
for row in reader:
cities.append(row.get('cities'))
br = Browser()
br.set_handle_robots(False) # ignore robots
br.set_handle_refresh(False) # can sometimes hang without this
br.open("https://iafisher.com/projects/cities/world") # The website if you are curious
for form in br.forms():
print(form.name) # Prints nothing
for i in range(len(cities)):
br.select_form(class="city-input") # ISSUE IS THROWN HERE
control = br.form.find_control("controlname")
# Browser passes through unknown attributes (including methods)
# to the selected HTMLForm (from ClientForm).
br[control] = [cities[i]] # (the method here is __setitem__)
response = br.submit() # submit current form
The input value as seen in developer tools:
<input data-v-018e983a="" id="city-input" type="text" placeholder="Try 'Tokyo' or 'Kingston, Jamaica'" autocomplete="off" spellcheck="false" class="city-input ">
If there is any alternative to Mechanize or a method in Mechanize that would work, it would be appreciated.
After trying multiple different search options, I finally stumbled across a tutorial that manually tells Selenium what to do.
Final code:
from selenium import webdriver
import csv
from selenium.webdriver.common.keys import Keys
csv_file = 'city_names.csv'
cities = []
with open(csv_file, 'r') as f:
reader = csv.DictReader(f)
for row in reader:
cities.append(row.get('cities'))
chromedriver_location = "C:/Users/blais/Downloads/chromedriver"
driver = webdriver.Chrome(chromedriver_location)
driver.get('https://iafisher.com/projects/cities/world')
submit = '//*[#id="city-input"]'
for c in range(len(cities)):
driver.find_element_by_xpath(submit).send_keys(cities[c] + Keys.ENTER)
I want to send a message to this website with Python.
It is to say to do the following but with python :
That's why I tried the following script with Selenium:
api_location = 'http://iphoneapp.spareroom.co.uk'
api_search_endpoint = 'flatshares'
api_details_endpoint = 'flatshares'
location = 'http://www.spareroom.co.uk'
details_endpoint = 'flatshare/flatshare_detail.pl?flatshare_id='
def contact_room(self, room_id):
url = '{location}/{endpoint}/{id}?format=json'.format(location=self.api_location, endpoint=self.api_details_endpoint, id=room_id)
from selenium import webdriver
driver = webdriver.Chrome()
# Go to your page url
driver.get(url)
# Get button you are going to click by its id ( also you could use find_element_by_css_selector to get element by css selector)
button_element = driver.find_element_by_id('button id')
button_element.click()
But it returns:
C:\Users\antoi\Documents\Programming\projects\roomfinder>python test_message.py
Traceback (most recent call last):
File "C:\Python36\lib\site-packages\selenium\webdriver\common\service.py", line 76, in start
stdin=PIPE)
File "C:\Python36\lib\subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "C:\Python36\lib\subprocess.py", line 997, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test_message.py", line 21, in <module>
contact_room(13829371)
File "test_message.py", line 14, in contact_room
driver = webdriver.Chrome() # Optional argument, if not specified will search path.
File "C:\Python36\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 73, in __init__
self.service.start()
File "C:\Python36\lib\site-packages\selenium\webdriver\common\service.py", line 83, in start
os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
While I already added it in the PATH:
I am javascript learner. If you have tips and time to show how to answer the question as well in Javascript I am always happy to learn :)
The chromedriver needs to be in the path of your python script or you need to add it to your driver:
driver_path = 'Path\to\your\Driver'
driver = webdriver.Chrome(executable_path = driver_path)
Why are you using webdriver.Firefox() if you talk about Chrome?
I'm trying to save a portion of a webpage and save it as html file.
I can do it manually like this:
When I use F12 (developer tools) in Chrome or Mozilla and use the selector to select the position of the website I want, I see a div and I copy the Xpath. Then I copy the HTML of that element and paste into a notepad editor and save it as HTML.
I've used before Selenium IDE but I don't find a way to save the content of the Xpath of that div.
Is there a way to do it with a combination of Selenium IDE and JavaScript or Python?
Maybe someone could suggest me how to achieve this.
Thanks
this is just selenium example, not your particular answer.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time
import random
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
seed = 1
random.seed(seed)
driver = webdriver.Chrome()
driver.get("https://www.myntra.com/")
element = driver.find_element_by_xpath("//*[#id='desktop-header-cnt']/div[2]/div[3]/input")
# Put the word "history" in the search box and hit enter
element.send_keys("pantaloons")
element.send_keys(Keys.RETURN)
time.sleep(3)
for i in range(1000):
time.sleep(1)
for i in range(120):
actions = ActionChains(driver)
actions.send_keys(Keys.ARROW_DOWN)
actions.perform()
time.sleep(0.10)
element=driver.find_element_by_xpath(" //*[#id='desktopSearchResults']/div[2]/section/div[2]/ul/li[12]/a")
element.click()
time.sleep(1)
#
#
# # Get a list of elements (videos) that get returned by the search
# search_results = driver.find_elements_by_id("video-title")
#
# # Click randomly on one of the first five results
# search_results[random.randint(0,10)].click()
#
# # Go to the end of the page (I don't know if this is necessary
#
# #
# time.sleep(4)
#
# # Get the recommended videos the same way as above. This is where the problem starts, because recommended_videos essentially becomes the same thing as the previous page's search_results, even though the browser is in a new page now.
# while True:
# recommended_videos = driver.find_elements_by_xpath("//*[#id='dismissable']/div/a")
# print(recommended_videos)
# recommended_videos[random.randint(1,4)].click()
# time.sleep(4)
You can try to dump page source and parse it, or dump only element source.
Page source to pageSource variable (Java):
String pageSource = driver.getPageSource();
Element source to elementSource variable (Java):
WebElement element = driver.findElement(By.id("id"));
String elementSource = element.getAttribute("innerHTML");
I want to create a VBA macro on excel which at a click of button would open a browser (chrome or safari) login to a website, extract the desired float value, then populate a given cell in the sheet with that value.
There are examples online on how to achieve this using internet explorer but this is not available as on a mac. I have also seen guides using Selenium but this doesn't appear to work on mac.
The javascript itself is along the lines of (after opening a browser at a certain website):
document.getElementById("username").value = "username"
document.getElementById("password").value = "password"
document.getElementsByClassName("button")[0].click()
value = parseFloat(document.getElementsByClassName("value")[1].innerText.slice(1))
I've solved this by using a combination of python-selenium and xlwings. My VBA calls RunPython ("import python_script; python_script.fun()")
python_script.py
import xlwings as xw
from selenium import webdriver
def fun():
# Creates a reference to the calling Excel file
wb = xw.Book.caller()
# opens chrome
chrome_driver_path = '/usr/local/bin/chromedriver'
driver = webdriver.Chrome(chrome_driver_path)
# open website and login
driver.get('url')
driver.find_element_by_id('username').send_keys('username')
driver.find_element_by_id('password').send_keys('password')
driver.find_element_by_name('buttonId').click()
# finds member price sum
table_body = driver.find_element_by_xpath("//*[#class='classname']").text
price = float(table_body.split()[2][1:])
# closes chrome
driver.quit()
# changes cell value
sheet = wb.sheets['sheetname']
sheet.range('cell').value = price
I'm trying to scrape javascript data from a site. Currently I'm given myself the challenge of trying to scrape the number of Followers from this website. Here's my code so far:
import os
from selenium import webdriver
import time
chromedriver = "/Users/INSERT USERNAME/Desktop/chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)
driver.get("http://freelegalconsultancy.blogspot.co.uk/")
time.sleep(5)
title = driver.find_element_by_class_name
print title
As you can see, I've got a chromedriver file located on my desktop. When I execute the code, I get the following result:
<bound method WebDriver.find_element_by_class_name of <selenium.webdriver.chrome.webdriver.WebDriver (session="dd9e5d3f429bc2810c30ebe7067e4e22")>>
I tried iterating into this with a for loop but it returned an error. Does anyone know how I can get the Javascript data and ultimately get the number of followers?
EDIT:
So as per request, I have changed my code to this:
import os
from selenium import webdriver
import time
chromedriver = "/Users/INSERT USERNAME/Desktop/chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)
driver.get("http://freelegalconsultancy.blogspot.co.uk/")
time.sleep(5)
title = driver.find_element_by_class_name("member-title")
print title
But I now get this error:
Traceback (most recent call last):
File "C:\Users\INSERT USERNAME\Desktop\blogger_v.1.py", line 11, in <module>
title = driver.find_element_by_class_name("member-title")
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 413, in find_element_by_class_name
return self.find_element(by=By.CLASS_NAME, value=name)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 752, in find_element
'value': value})['value']
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"class name","selector":"member-title"}
(Session info: chrome=53.0.2785.143)
(Driver info: chromedriver=2.24.417431 (9aea000394714d2fbb20850021f6204f2256b9cf),platform=Windows NT 6.1.7601 SP1 x86_64)
Any ideas on how I can get around it?
EDIT:
So I've changed my code to this:
import os
from selenium import webdriver
import time
chromedriver = "/Users/INSERT USERNAME/Desktop/chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)
driver.get("http://freelegalconsultancy.blogspot.co.uk/")
time.sleep(5)
title = driver.find_element_by_class_name("item-title")
print title
And I get this result:
<selenium.webdriver.remote.webelement.WebElement (session="5fe8fb966edd26fdf808da07f99d4109", element="0.9924860218635834-1")>
How would I go about just printing all the javascript? Is this even possible?
You need to provide the class name you're looking for as a parameter.
title = driver.find_element_by_class_name("TheNameOfTheClass")