Develop Reference

Getting 403 when using Selenium to automate checkout process

I am trying to create a script using python and selenium to automate the checkout process at bestbuy.ca.
I get all the way to the final stage where you click to review the final order, but get the following 403 forbidden message (as seen in the network response) when I try to click through to the final step.
Is there something server side that has detected that I am using selenium and preventing me to proceed?
How can I hide the fact that it is selenium being used?
These are the options I am using for selenium:
options = Options()
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("start-maximized")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(options=options)
I currently have 10 second delays after each action(ie open page, wait, click add to cart, wait, click checkout, wait)
I have implemented a random useragent to be used on each run:
import fake_useragent
ua = UserAgent()
userAgent = ua.random
options.add_argument(f'user-agent={userAgent}')
I have also modified my chromedriver binary as per the comments in THIS THREAD
Error seen when proceeding to order review page:

After much testing the last few days, here are the options that have allowed me to bypass the restrictions I was facing.
Modified cdc_ string in my chromedriver
Chromedriver options:
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("--disable-extensions")
options.add_experimental_option('useAutomationExtension', False)
options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_driver = webdriver.Chrome(options=options)
Change the property value of the navigator for webdriver to undefined:
chrome_driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
After all three of these were implemented I no longer faced any 403 error when navigating the site and the cart/checkout process.

In my case, either using code to control the browser, or simply starting Chrome through python and manually using the browser always leads to the 403 error, even just adding a product to the cart.
As you said, I think that this site someway knows that the user is using Selenium or some sort of automation tool and the server is blocking API requests.
Searching in stackoverflow I found this https://stackoverflow.com/a/52108199/3228768 but editing the chromedriver results anyway in a failure.
The only way I completed the flow is settings this options:
u = 'https://www.bestbuy.ca/en-ca/category/appliances/26517'
# relevant part start here
options = webdriver.ChromeOptions()
options.add_argument("--disable-blink-features")
options.add_argument("--disable-blink-features=AutomationControlled")
# relevant part ends here
driver = webdriver.Chrome(executable_path=r"chromedriver.exe", options=options)
driver.maximize_window()
driver.get(u)
In this way I managed to add a product to the cart. I think you could use it to proceed the flow until checkout.
Let me know.

Try this one: https://github.com/ultrafunkamsterdam/undetected-chromedriver
It avoids the selenium detection quite well, I've been having good result with it so far. Headless is not guaranteed though.
import undetected_chromedriver as uc
driver = uc.Chrome()
driver.get('https://www.bestbuy.ca/')

Selenium will not show javascript loaded table in webpage with Python

I am trying to scrape the following webpage: https://steamdb.info/app/730/graphs/
(I have gained permission from the website)
The problem is that the "Monthly Breakdown" table seems to be loaded by Javascript, and BeautifulSoup does not work. When using Selenium to open the webpage, it says that to see the table "You must have Javascript enabled.", which should be enabled when using Selenium. Here is my code:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--enable-javascript")
browser = webdriver.Chrome(options=options)
browser.maximize_window()
url = "https://steamdb.info/app/730/graphs/"
browser.get(url)
Any ways to solve this problem?
How the page should look:
How it looks on Selenium:

Try this and see if you no longer get that error message:
options.add_argument("javascript.enabled", True)
You may also need to look into "waits" here to make sure the async operation on the webpage has time to load.

Update:
To enable or disable javascript :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
#value 1 enables it , if you set to 2 it disables it
chrome_options.add_experimental_option( "prefs",{'profile.managed_default_content_settings.javascript': 1})
driver =webdriver.Chrome(r".\chromedriver.exe",options=chrome_options)
driver.get("https://www.google.com")
THis wont solve your issue
you can check if the javascript is enabled by typing below in your address bar:
chrome://settings/content/javascript?search=javascript
you can see that even if its enabled the website won't load properly.
it seems they have enabled security to avoid using selenium in thier website
Previous answer:
There is no command line argument called --enable-javascript chromium project try the above javascript-harmony flag instead.
below are the full list of supported chrome flags:
https://peter.sh/experiments/chromium-command-line-switches/#login-profile
please add screenshots and other information if more help is needed

Can't turn off Javascript using Selenium

I'm trying to turn off javascript via the profile when opening using Selenium. This has work previously but now I've updated Selenium/Firfox I can't get it to work.
profile = webdriver.FirefoxProfile()
profile.set_preference('javascript.enabled', False)
driver = webdriver.Firefox(profile)
driver.implicitly_wait(30)
driver.get("http://www.enable-javascript.com/")
All other settings seem to change while using profile.set_preference() on other option and javascript.enabled exists and is set to True when I look at the Firefox settings about:config. Is it possible Javascript is being set to True after loading the profile or something?
FF version 43.0.3
Selenium version 2.48.0
Any suggestions on why this could be happening?
UPDATE
Adding profile.add_extension("path/to/noscript_security_suite-2.9.xpi"); to the above code with the downloaded extension as #alecxe suggested fixed the issue.

This issue affects selenium starting with 2.46.0, javascript.enabled is being ignored:
Firefox driver 2.46.0 regression - unable to set to non-js
As a workaround, load the noscript addon, see:
How to disable Javascript when using Selenium by JAVA?

How to test inline installation of Chrome extension in Protractor? [duplicate]

I am using selenium for some browser automation. I need to install an extension in the browser for my work. I am doing it as follows:
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
executable_path = "/usr/bin/chromedriver"
options = Options()
options.add_extension('/home/TheRookie/Downloads/extensionSamples/abhcfceiempjmchhhdhbnkbimnfpckgl.crx')
browser = webdriver.Chrome(executable_path=executable_path, chrome_options=options)
The browser is starting fine but I am prompted with a pop-up to confirm that I want to add the extension as follows:
and after I get this pop-up, Python soon returns with the following exception:
selenium.common.exceptions.WebDriverException: Message: u'unknown
error: failed to wait for extension background page to load:
chrome-extension://abhcfceiempjmchhhdhbnkbimnfpckgl/toolbar.html\nfrom
unknown error: page could not be found:
chrome-extension://abhcfceiempjmchhhdhbnkbimnfpckgl/toolbar.html\n
(Driver info: chromedriver=2.12.301324
(de8ab311bc9374d0ade71f7c167bad61848c7c48),platform=Linux
3.13.0-39-generic x86_64)'
I tried handling the popup as a regular JavaScript alert using the following code:
alert = browser.switch_to_alert()
alert.accept()
However, this doesn't help. Could anyone please tell me how do I install this extension without the popup or a way to accept the popup? Any help would be greatly appreciated. Thanks!

Usually, you cannot test inline installation of a Chrome extension with just Selenium, because of that installation dialog. There are a few examples in the wild that show how to use external tools outside Selenium to solve this problem, but these are not very portable (i.e. platform-specific) and rely on a state of Chrome's UI, which is not guaranteed to be consistent.
But that does not mean that you cannot test inline installation. If you replace chrome.webstore.install with a substitute that behaves like the chrome.webstore.install API (but without the dialog), then the end-result is the same for all intents and purposes.
"Behaves like chrome.webstore.install" consists of two things:
Same behavior in error reporting and callback invocation.
An extension is installed.
I have just set up such an example on Github, which includes the source code of the helper extension/app and a few examples using Selenium (Python, Java). I suggest to read the README and the source code to get a better understanding of what happens: https://github.com/Rob--W/testing-chrome.webstore.install.
The sample does not require the tested extension to be available in the Chrome Web store. It does not even connect to the Chrome Web store. In particular, it does not check whether the site where the test runs is listed as a verified website, which is required for inline installation to work.

I had some really big code which I would have to re-write if I had to use Java. Luckily, python has a library for automating GUI events called ldtp. I used that to automate the clicking on the "Add" button. I did something on the following lines:
from ldtp import *
from threading import Thread
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
def thread_function():
for i in range(5):
if activatewindow('Confirm New Extension'):
generatekeyevent('<left><space>')
break
time.sleep(1)
def main():
executable_path = "/usr/bin/chromedriver"
options = Options()
options.add_extension('/home/TheRookie/Downloads/extensionSamples/abhcfceiempjmchhhdhbnkbimnfpckgl.crx')
thread.start()
browser = webdriver.Chrome(executable_path=executable_path, chrome_options=options)
Hope it helps somebody.

Can I/how can I translate a Selenium webdriver test script from node.js over to phantomjs - ghostdriver?

I recently began working with Selenium and to make life easier to start I was using node to run my scripts so that I could visually monitor the tests. My challenge now is to convert it so that it can be run as a headless test. Unfortunately, most of the resources that I have come across only address using phantomjs and ghostdriver with Java or Python. My boss wants me to run the test through phantomjs without Java or Python. Eventually these tests will be run remotely through a Linux VM on a server without a GUI. Currently I am testing using Mac OS X 10.8 and still have many bridges to cross in order to get to my goal.
My most important question firstly, is it possible to run a script from phantomjs through a port without the use of Java or Python? I have spent hours poring through as many resources as I could come across and I've come up with no solution.
If so, how can I properly initialize the test to run headless? Here is how I scripted the start of my functioning test. I want to properly switch the capabilities from firefox to phantomjs and be able to run it headless using the appropriate port. The rest of the test navigates to a specific site, logs in through a widget, then does further navigation to the area which I will build further tests on which to manipulate after I get this working.
var webdriver = require('selenium-webdriver'),
SeleniumServer = require('selenium-webdriver/remote').SeleniumServer;
var server = new SeleniumServer("Path/selenium-server-standalone-2.39.0.jar", {
port: 8910
});
server.start();
var driver = new webdriver.Builder().
usingServer(server.address()).
withCapabilities(webdriver.Capabilities.firefox()).
build();
The test works perfectly, but I am new to this so there might be something foolish that I am overlooking. Please let me know what adjustments to make so that it will run headless through phantom. When I attempt to use node to run the script after switching capabilities to phantomjs it produces
"/Selenium/node_modules/selenium-webdriver/phantomjs.js:22
LogLevel = webdriver.logging.LevelName,
^
TypeError: Cannot read property 'LevelName' of undefined
at Object.<anonymous> (/Selenium/node_modules/selenium-webdriver/phantomjs.js:22:33)
That's a read only file that I can't adjust, any attempts that I made to define "LogLevel" or "LevelName" to the appropriate corresponding value (DEBUG, etc.) were fruitless.
And if I run it through phantomjs itself I get -
"Error: Cannot find module 'path'
phantomjs://bootstrap.js:289
phantomjs://bootstrap.js:254 in require"
(It also lists module 'http') -- (and various undefined function errors)
I feel that with that instance I didn't properly organize where the files for Selenium, phantomjs, and ghostdriver should go in order to play nice. I also removed the server setup portion and instead ran this first, then the script separately.
phantomjs --webdriver=8910
But it yielded the same result. All of my research to fix these issues turned up instructions for Java and Python but not Javascript by itself. Rather than chase through many rabbit holes I figured it wise to consult better minds.
If you know better than I do and that it is fruitless to attempt this without Java or Python, please let me know. If you know where the issue lies within my script and could propose a fix please let me know. I hope that I have properly described the nature of my issue and if you need more information I will do my best to provide it to you.
This is my second week working with Javascript so if you believe I am making a noob error you very well may be correct. Please, keep in mind that the script works through node with selenium webdriver.
Many thanks for your time!!!
~Isaac

This was a bit tricky but here is the solution I've pieced together:
var webdriver = require('selenium-webdriver'),
SeleniumServer = require('selenium-webdriver/remote').SeleniumServer,
server = new SeleniumServer('/path/to/selenium/selenium-server-standalone-2.41.0.jar', {
port: 4444
}),
capabilities = webdriver.Capabilities.phantomjs();
capabilities.set('phantomjs.binary.path', 'path/to/phantom/bin/phantomjs');
var promise = server.start().then(function() {
var client = new webdriver.Builder().
usingServer(server.address()).withCapabilities(
capabilities
).build();
return {
'client': client,
'server': server
};
}, function(err) {
console.log('error starting server', err);
});
You can then use the promise with selenium's mocha-compatible test framework to hold the test till the server has started.
I found the documentation really helpful once i figured out the navigation is on the far right of the page. Here's the URL: http://selenium.googlecode.com/git/docs/api/javascript/module_selenium-webdriver.html
Then you'll be stuck where I am. Getting selenium-webdriver to quiet down.