I'm trying to make an auto-login twitter bot. But when I try to send_keys to passwords field, I can't. (Only passwords field doesn't work, the similar code for send_keys to username and phone_number works).
Error Message: "selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable".
So I tried to use execute_script instead.
driver.execute_script("arguments[0].value=arguments[1];", password_input, TWITTER_PASSWORD)
The line above works too. But I don't know how to send Enter key to arguments[0]
driver.execute_script("arguments[0].submit();", password_input)
Tried this but doesn't work. (Forgive me if this line is completely wrong cause it looks like this method take JS code and I don't know JS)
Please help me with this. Any help for this problem or just my code in general would be appreciated.
""" If you want to test the code, remember to give values to TWITTER_EMAIL, TWITTER_PHONE_NUMBER and TWITTER_PASSWORD. """
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
import time
TWITTER_EMAIL =
TWITTER_PHONE_NUMBER =
TWITTER_PASSWORD =
URL = r"https://twitter.com/i/flow/login"
s = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=s)
driver.get(url=URL)
def twitter_auto_login():
time.sleep(10)
username_input = driver.find_element(By.CSS_SELECTOR, "input")
username_input.send_keys(TWITTER_EMAIL)
time.sleep(2)
username_input.send_keys(Keys.ENTER)
time.sleep(5)
phone_num_input = driver.find_element(By.CSS_SELECTOR, "input")
phone_num_input.send_keys(TWITTER_PHONE_NUMBER)
time.sleep(2)
phone_num_input.send_keys(Keys.ENTER)
time.sleep(5)
password_input = driver.find_element(By.CSS_SELECTOR, "input")
# driver.execute_script("arguments[0].click();", password_input)
driver.execute_script("arguments[0].value=arguments[1];", password_input, TWITTER_PASSWORD)
# https://stackoverflow.com/questions/52273298/what-is-arguments0-while-invoking-execute-script-method-through-webdriver-in
time.sleep(2)
driver.execute_script("arguments[0].submit();", password_input)
UPDATE: So I found exactly what is my stupid mistake here, in the third time entering input (the password input time), username is also an input tag will appear again and will be an unreachable tag. just use find_elements and select the second element of that list will get us the right password input tag. But the solution from wado is better, just use it in case u face the same problem
No need to use driver.execute_script. In your case you're just locating the elements in a wrong way. You should do something like this:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
TWITTER_EMAIL = "email#email.com"
TWITTER_PASSWORD = "lalala"
URL = r"https://twitter.com/i/flow/login"
s = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=s)
driver.get(url=URL)
def twitter_auto_login():
username_input = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//input[#autocomplete='username']")))
username_input.send_keys(TWITTER_EMAIL)
username_input.send_keys(Keys.ENTER)
password_input = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//input[#autocomplete='current-password']")))
password_input.send_keys(TWITTER_PASSWORD)
password_input.send_keys(Keys.ENTER)
twitter_auto_login()
Note that I used explicit waits that are way better than those implicit waits (that makes you waste a lot of time senseless).
Via JQuery, you can use the following Javascript to simulate the enter event:
driver.execute_script("var script = document.createElement('script');
script.src = 'https://code.jquery.com/jquery-3.6.0.min.js';
var e = $.Event( 'keypress', { which: 13 } );
arguments[0].trigger(e);", password_input)
It sounds like you want to submit whatever form that input is part of:
arguments[0].closest('form').submit()
Related
Im trying to use selenium web driver to extract data and i get to see that one of the links that i want to click does not have a href. The html tags i see in inspect element are also not available in the page source.I badly want the link to be clicked and proceed to the next page.
The anchor tag that i see during inspect is as below and this seems to be having a angular JS
< a id="docs" ng-click="changeFragment('deal.docs')">
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get('URL here');
time.sleep(5) # Let the user actually see something!
username = driver.find_element_by_name('USERID')
username.send_keys('12345')
password = driver.find_element_by_name('PASSWORD')
password.send_keys('password')
#search_box.submit()
driver.find_element_by_id("submitInput").submit()
time.sleep(5) # Let the user actually see something!
lnum = driver.find_element_by_name('Number')
lnum.send_keys('0589403823')
checkbox = driver.find_element_by_name('includeInactiveCheckBox').click()
driver.find_element_by_id("searchButton").click()
time.sleep(5)
driver.execute_script("changeFragment('deal.docs')").click()
driver.quit()
I tried to use find element by xpath and script but both didnt work .
The url im trying access cant be shared as it can be accessed only through a specific network
My code inputs text into the text area of the web page , line by line, how to make it insert the entire text all at once instead, is there a solution for this?
because line by line takes a lot of time
def Translated_Content(content):
driver= webdriver.Chrome("C:\\Users\\shricharan.arumugam\\Desktop\\PDF2txt\\chromedriver.exe")
driver.get('https://translate.shell.com/')
input_box = driver.find_element_by_id('translateText')
input_box.send_keys(content)
translate_button = driver.find_element_by_id('translate')
translate_button.click()
translated_text_element= driver.find_element_by_id('translatedText')
time.sleep(4)
translated_text=translated_text_element.get_attribute('value')
driver.close()
return translated_text
You can change the text of textbox/textarea through JavaScript DOM API in silent way, not from front UI:
long_string= <the long string>
input_box = driver.find_element_by_id('translateText')
driver.execute_script('arguments[0].value=arguments[1]', input_box, long_string)
To send the entire chunk of text into the <textarea> using selenium through Python to speed up the process you can inject a script and use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
myText = """No, there is no way to hide the console window of the chromedriver.exe
in the .NET bindings without modifying the bindings source code. This is seen
as a feature of the bindings, as it makes it very easy to see when your code
hasn't correctly cleaned up the resources of the ChromeDriver, since the console window
remains open. In the case of some other languages, if your code does not properly clean up
the instance of ChromeDriver by calling the quit() method on the WebDriver object,
you can end up with a zombie chromedriver.exe process running on your machine."""
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
driver=webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get('https://translate.shell.com/')
translate_from = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "textarea.form-control#translateText")))
translate_from._parent.execute_script("""
var elm = arguments[0], text = arguments[1];
if (!('value' in elm))
throw new Error('Expected an <input> or <textarea>');
elm.focus();
elm.value = text;
elm.dispatchEvent(new Event('change'));
""", translate_from, myText)
driver.find_element_by_css_selector("input#translate").click()
Browser Snapshot:
I am trying to download a csv file from a website using selenium, but I am failing in the last step.
I fail at selecting the format of the file and then to click on export. Does someone as any idea on how to do it? There's a free registration process to be able to connect to the website, you would have to register with your email address to try. I have attached a picture and circled in red the part I struggle to automate picture. Below is the working code up until before the last step i would like to complete.
Thank you very much for your help!
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
driver = webdriver.PhantomJS()
driver.get("http://www.sem-o.com/MarketData/pages /default.aspx?ReturnUrl=%2fMarketData%2fPages%2fDynamicReports.aspx")
#log-in
##############
elem = driver.find_element_by_name("ctl00$PlaceHolderMain$FBALoginId$username")
elem.clear()
elem.send_keys("EMAIL")
elem.send_keys(Keys.RETURN)
elem = driver.find_element_by_id("ctl00_PlaceHolderMain_FBALoginId_password")
elem.clear()
elem.send_keys("PASSWORD")
#elem.send_keys(Keys.RETURN)
elem = driver.find_element_by_id(r"ctl00_PlaceHolderMain_FBALoginId_btnLogin")
elem.click()
##############
#retrieve files of interest
##############
elem = Select(driver.find_element_by_id("ctl00_ctl00_g_f5e6fa98_faa2_4210_85e9_780934d96ab8_cmbReportGroup"))
elem.select_by_visible_text('Forecast Data')
elem = Select(driver.find_element_by_id("ctl00_ctl00_g_f5e6fa98_faa2_4210_85e9_780934d96ab8_cmbSelectReport"))
elem.select_by_visible_text("Four Day Load Forecast")
elem = driver.find_element_by_id(r"ctl00_ctl00_g_9ab92c0a_eb10_4b6c_ad1b_7277cbdab462_btnGenerateLocalReport")
elem.click()
elem = driver.find_element_by_id(r"ctl00_ctl00_g_9ab92c0a_eb10_4b6c_ad1b_7277cbdab462_prm_GetFromDate_prm_GetFromDateDate")
elem.clear()
elem.send_keys("01/01/2017")
elem = driver.find_element_by_id(r"ctl00_ctl00_g_9ab92c0a_eb10_4b6c_ad1b_7277cbdab462_prm_GetToDate_prm_GetToDateDate")
elem.clear()
elem.send_keys("15/01/2017")
elem = driver.find_element_by_id(r"ctl00_ctl00_g_9ab92c0a_eb10_4b6c_ad1b_7277cbdab46 2_btnGenerateLocalReport")
elem.click()
What is the error you are getting?
Which step is it failing at, is it the actual clicking on export? If this is the case, check the way you are identifying the element in the Google dev console. Go to F12 dev tools, then in the console type:
$$('#ctl00_ctl00_g_9ab92c0a_eb10_4b6c_ad1b_7277cbdab46.2_btnGenerateLocalReport')
This will show all elements with that selected id. If it's an empty array, then the selector is invalid.
You may have to experiment with the id selector as that space looks a bit odd. More information is needed on the error you are getting and why it's failing to provide a better answer.
I'm trying to download all xml files from a webpage. The process requires locating xml file download link one after the other, and once such a download link is clicked it leads to a form which needs to be submitted for the download to begin. The issue I'm facing lies in the iteration of these loops, once the first file is downloaded from the webpage I receive an error:
"selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of stale: either the element is no longer attached to the DOM or the page has been refreshed"
The "97081 data-extension xml" is the 2nd downloadable file in the iteration. I've hereby attached the code, any suggestions to rectify this will be much appreciated.
import os
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
fp = webdriver.FirefoxProfile()
fp.set_preference("browser.download.folderList", 2)
fp.set_preference("browser.download.manager.showWhenStarting", False)
fp.set_preference("browser.download.dir", "F:\Projects\Poli_Map\DatG_Py_Dat")
fp.set_preference("browser.helperApps.neverAsk.saveToDisk", "text/xml")
driver = webdriver.Firefox(firefox_profile=fp)
driver.get('https://data.gov.in/catalog/variety-wise-daily-market-prices-data-cauliflower')
wait = WebDriverWait(driver, 10)
allelements = driver.find_elements_by_xpath("//a[text()='xml']")
for element in allelements:
element.click()
class FormPage(object):
def fill_form(self, data):
driver.execute_script("document.getElementById('edit-download-reasons-non-commercial').click()")
driver.execute_script("document.getElementById('edit-reasons-d-rd').click()")
driver.find_element_by_xpath('//input[#name = "name_d"]').send_keys(data['name_d'])
driver.find_element_by_xpath('//input[#name = "mail_d"]').send_keys(data['mail_d'])
return self
def submit(self):
driver.execute_script("document.getElementById('edit-submit').click()")
data = {
'name_d': 'xyz',
'mail_d': 'xyz#outlook.com',
}
time.sleep(5)
FormPage().fill_form(data).submit()
time.sleep(5)
window_before = driver.window_handles[0]
driver.switch_to_window(window_before)
driver.back()
I found a workaround for you, no need to submit any fields.
You need to get the ID in the class field in the bottom of this picture (here for instance its 962721)
Then, use this URL like so :
https://data.gov.in/node/962721/download
This was found just doing a bit of "reverse-engineering". When you do web scraping, always have a look at the .js files and at your networking tab to see all the requests made.
The URL is:site
By using selenium with Firefox 47.0.2 binary and python 3.6.0, from this page I click on “Pesquisar” button and in the next page I fill in the form with tha date range (format d/m/y) and click again on the new “Pesquisar” button, then I get a list of PDF documents and I want to download them.
When I print the page_source, I can see the links generated, but I don’t understand why selenium can’t locate those links.
The simplified code is as follows:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from datetime import datetime, date, timedelta
from calendar import monthrange
import time
driver = webdriver.Firefox(firefox_profile=profile, firefox_binary=binary, capabilities=capabilities)
driver.maximize_window()
wait = WebDriverWait(driver, 10)
months = range(1, 13)
limits = monthrange(2017, 8)
#num_docs = limites[1]-limites[0]
date_input_begin = '{num:0{width}}'.format(num=limits[0], width=2) + '08' + '2017'
date_input_end = '{num:0{width}}'.format(num=limits[1], width=2) + '08' + '2017'
today = datetime.now().date()
date = today
date = date - timedelta(24)
driver.get("http://dje.trf2.jus.br/DJE/Paginas/Externas/inicial.aspx")
driver.find_element_by_id("ctl00_ContentPlaceHolder_ctrInicial_btnPesquisar").click()
wait.until(EC.presence_of_element_located(
(By.XPATH, '//*[#id="ctl00_ContentPlaceHolder_ctrFiltraPesquisaDocumentos_btnFiltrar"]')))
select1 = Select(driver.find_element_by_id("ctl00_ContentPlaceHolder_ctrFiltraPesquisaDocumentos_ddlAreaJudicial"))
select1.select_by_index(3)
select2 = Select(driver.find_element_by_id("ctl00_ContentPlaceHolder_ctrFiltraPesquisaDocumentos_ddlRegistrosPaginas"))
select2.select_by_index(6)
element_date_begin = driver.find_element_by_id(
'ctl00_ContentPlaceHolder_ctrFiltraPesquisaDocumentos_tbxDataInicial')
element_date_begin.clear()
element_date_begin.send_keys(date_input_begin)
element_date_end = driver.find_element_by_id(
'ctl00_ContentPlaceHolder_ctrFiltraPesquisaDocumentos_tbxDataFinal')
element_date_end.clear()
element_date_end.send_keys(date_input_end)
driver.find_element_by_id('ctl00_ContentPlaceHolder_ctrFiltraPesquisaDocumentos_btnFiltrar').submit()
wait.until(EC.presence_of_element_located((By.ID, 'ctl00_ContentPlaceHolder_ctrFiltraPesquisaDocumentos_btnFiltrar')))
wait.until(EC.element_to_be_clickable((By.ID, 'ctl00_ContentPlaceHolder_ctrFiltraPesquisaDocumentos_btnFiltrar')))
time.sleep(5)
driver.find_element_by_id('ctl00_ContentPlaceHolder_ctrFiltraPesquisaDocumentos_btnFiltrar').click()
wait.until(EC.presence_of_element_located(
(By.XPATH, '//*[#id="ctl00_ContentPlaceHolder_ctrListaDiarios_udtVisualizaAdmRj_lblNomeCaderno"]')))
driver.find_element_by_xpath(
'//*[#id="ctl00_ContentPlaceHolder_ctrListaDiarios_udtVisualizaAdmRj_grvCadernos_ct102_lnkData"]').click()
But when I look for the links by ID or XPATH, I get the following error:
File "C:\Users\b2002032064079\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: {"method":"xpath","selector":"//*[#id=\"ctl00_ContentPlaceHolder_ctrListaDiarios_udtVisualizaAdmRj_grvCadernos_ct102_lnkData\"]"}
I’m a newbie at scraping and I’d be very thankful for any help! Thank you!
First of all: Which browser are you using?
2: Your site is slow. Maybe try giving more waiting time.
3: Is the xpath correct? I think that the problem is with the XPATH
try using XPath helper on chrome to check.