Download a file using Watir Webdriver and phantomjs - javascript

I am using Watir Webdriver and a headless(phantomjs) browser to goto a website, login into it and click and download a file using javascript submit button.When I click on submit, I am redirected with 302 to a different address that i can see under my Network.This is url of the file to download.I am degugging using screenshots so i can see the phantomjs is working fine but after it hits on submit button, nothing happens.This whole procedure is working fine on firefox too.Using watir webdriver, how can i get that link and save it in database and redirect my phantomjs to download the file using that link?I tried reading github pull requests, official documentation and blog posts but i am unable to reach to any solution.Please provide me with suggestions or solutions. Even one word suggestion is also appreciated as it might help me to approach the problem.I have tried getting 'http request headers' but didn't succeed.I have browser.cookie.to_a and browser.headers is giving me only object like this Watir::HTMLElementCollection:0x000000024b88c0.Thank you

I was not to find solution to my question using Phantomjs but I have solved the problem using watirwebdriver(0.9.1) headless and firefox(44.0).
These are the settings i have used.
profile = Selenium::WebDriver::Firefox::Profile.new
profile['download.prompt_for_download'] = false
profile['browser.download.folderList'] = 2 # custom location
profile['browser.download.dir'] = download_directory
profile['browser.helperApps.neverAsk.saveToDisk'] = "application/pdf"
profile['pdfjs.disabled'] = true
profile['pdfjs.firstRun'] = false
headless = Headless.new
headless.start
browser = Watir::Browser.new(:firefox, :profile => profile)
browser.goto 'www.google.com'
browser.window.resize_to(1280, 720)
puts browser.title
puts browser.url

Related

window.onload with querystring Not working on deployed MVC6 project

I have been developing a website that can display some data. In the table, there is a button in each row to open a new window, where the user can see data related to that line.
I am using MVC 6 and I used Javascript to open the window and passing the 'id' parameter through querystring.
My code is:
The Parent View .cshtml:
The Button:
wButtonClass = "btn btn-warning openW";
<a href="javascript:void(0);" class="#wButtonClass" data-id=#id.ToString()>#buttonText</a>
The Script:
<script>
$(document).ready(function () {
$(".openW").click(function (e) {
var x = $(this).data("id");
var new_window = window.open('/MyView/HandleButton?id='+x, '_blank', 'left=200,top=150,width=1000,height=800,toolbar=1,resizable=0');
});
});
</script>
The Controller:
public IActionResult HandleButton(int id)
{
//Filling the List
return PartialView("DataView", myList);
}
So, it is working just fine from VS, but when I deploy the project to an IIS server (not on my machine), it opens the new window, and says "404 - Not Found", although the URL in the newly opened window is correct (the value is passed)
What could be the problem?
I've been looking through several forum questions, but couldn't find an answer.
Update 1:
Well, technicalliy, in the popup window, the URL is:
http://localhost/MyView/HandleButton?id=5
And it says in the 404 Error Details:
Requested URL http://localhost:80/MyView/HandleButton?id=5
Physical Path C:\inetpub\wwwroot\MyView\HandleButton
Update 2:
Well, I have finally found an answer. It looks very silly, but solved the problem.
I found it here:
IIS 8 Can't see partial view
I had to change the URL and add the application name:
So, instead of: /MyView/HandleButton?id='+x
I Typed: /MyWebApp/MyView/HandleButton?id='+x
Well, it works on the server, but doesn't work in VS. It will be fine (I just change the URL everytime I debug), but is there a way to do this more elegantly?
This could be expected if the routing is incorrect and the URL is incorrect. However, since your URL is correct and you still seeing a 404 error, it likely is an issue with the server configuration.
Verify that the IIS server has the correct permissions to access the
files and folders of your application.
Check that the IIS server has the correct .NET Framework version
installed and that your application is targeting the correct version.
Make sure that the IIS server has the correct MVC version installed
and that your application is targeting the correct version.
Check that the IIS server is configured to handle requests to the
.cshtml files.
Make sure that the MVC routing is configured correctly in the
web.config file of your application.
Check whether the IIS server is running in 32-bit or 64-bit mode.
Make sure that your application is built for the same mode.
Confirm that the server is able to find the DLLs and assemblies that
your application needs.
Check the event logs for any info.
This is italicized, and so
is this.
This is bold, just like this.
useEffect(() => {
var aScript = document.createElement("script");
aScript.type = "text/javascript";
aScript.src = "./js/main.js";
}, []);
You can combine them
if you really have to.

Selenium will not show javascript loaded table in webpage with Python

I am trying to scrape the following webpage: https://steamdb.info/app/730/graphs/
(I have gained permission from the website)
The problem is that the "Monthly Breakdown" table seems to be loaded by Javascript, and BeautifulSoup does not work. When using Selenium to open the webpage, it says that to see the table "You must have Javascript enabled.", which should be enabled when using Selenium. Here is my code:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--enable-javascript")
browser = webdriver.Chrome(options=options)
browser.maximize_window()
url = "https://steamdb.info/app/730/graphs/"
browser.get(url)
Any ways to solve this problem?
How the page should look:
How it looks on Selenium:
Try this and see if you no longer get that error message:
options.add_argument("javascript.enabled", True)
You may also need to look into "waits" here to make sure the async operation on the webpage has time to load.
Update:
To enable or disable javascript :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
#value 1 enables it , if you set to 2 it disables it
chrome_options.add_experimental_option( "prefs",{'profile.managed_default_content_settings.javascript': 1})
driver =webdriver.Chrome(r".\chromedriver.exe",options=chrome_options)
driver.get("https://www.google.com")
THis wont solve your issue
you can check if the javascript is enabled by typing below in your address bar:
chrome://settings/content/javascript?search=javascript
you can see that even if its enabled the website won't load properly.
it seems they have enabled security to avoid using selenium in thier website
Previous answer:
There is no command line argument called --enable-javascript chromium project try the above javascript-harmony flag instead.
below are the full list of supported chrome flags:
https://peter.sh/experiments/chromium-command-line-switches/#login-profile
please add screenshots and other information if more help is needed

python : disable download popup when using firefox with selenium

I have script that using selenium and firefox to automating download action.
The problem is whenever I run script I always get pop up from firefox keep asking what kinds of action I would like to do, even though I set download path in firefox preference. I checked files and folders to create master mimeTypes.rdf for all users, but I couldn't find mine.(I'm using ubuntu). I found ~/.mozilla/firefox but there was no file for directory of my profile name nor any file has an extension like .rdf
here is the criminal's pic that making me crazy
firefox download popup
below is what I've done to disable the popup.
profile = FirefoxProfile()
profile.set_preference("browser.download.panel.shown", False)
profile.set_preference("browser.helperApps.neverAsk.openFile", 'application/zip')
profile.set_preference("browser.helperApps.neverAsk.saveToDisk", 'application/zip')
profile.set_preference("browser.download.folderList", 2)
profile.set_preference("browser.download.dir", "/home/i-06/Downloads")
driver = webdriver.Firefox(firefox_profile=profile)
I have spent many hours trying to suppress that "save or open" pop-up that appears when downloading a file using the firefox driver with selenium (python 3.x). None of the many suggestions involving various values for profile.set_preference worked for me. Maybe I missed something.
Still, I finally got it working by the other method that is recommended : using an existing firefox profile.
You can tweak your default (or custom) profile to the file save behaviour you want. Type the following in the firefox address bar and make changes here :
about:preferences#applications
Then the only setting up you need to do to download the file into your current working directory is :
from selenium import webdriver
fp = webdriver.FirefoxProfile(<your firefox profile directory>)
fp.set_preference("browser.download.folderList",2)
fp.set_preference("browser.download.dir", os.getcwd())
driver = webdriver.Firefox(firefox_profile=fp)
If you have a typical ubuntu setup, you can find your default firefox profile dir by viewing ~/.mozilla/firefox/profile.ini
In that .ini file, look for Path under [Profile0]
I doubt you need to define both. Remove the below line from your code
profile.set_preference("browser.helperApps.neverAsk.openFile", 'application/zip')
Also sometime the MIME type of zip file can be different based on the server. It could be any of below
application/octet-stream
multipart/x-zip
application/zip
application/zip-compressed
application/x-zip-compressed
So in Network tab check what is the content type you are getting and add that to your profile to make sure the dialog doesn't come
I removed profile.set_preference("browser.helperApps.neverAsk.openFile", 'application/zip') as Tarun Lalwani suggest and it still work. But my problem was that I put application/mp4 instead of video/mp4. You could check MIME type here.

Selenium Webdriver Automatic Redirecting

Problem Description:
I am currently trying to set up a Selenium webdriver in Java.
However every time I try to load this specific webpage: Expected Website I end up with this Unintentional Website. No matter which driver I use (Firefox,Chrome,Edge), I somehow always get redirected and I did not find any solution to overcome this. Please note that the page loads some JS during the page loading process. This might be causing this redirection.
However if I use a standard browser I get the Expected Website as wished.
Goal:
Load this website with a Selenium webdriver: Expected Website
Additional Information:
The code I am using so far:
System.setProperty("webdriver.gecko.driver", "E:/Downloads/geckodriver.exe");
File pathToBinary = new File(
"C:/Program Files (x86)/Mozilla Firefox/firefox.exe");
FirefoxBinary ffBinary = new FirefoxBinary(pathToBinary);
FirefoxProfile firefoxProfile = new FirefoxProfile();
driver = new FirefoxDriver(ffBinary,firefoxProfile);
driver.get("https://www.liketoknow.it/featured");
try {
Thread.sleep(10000);
}catch (InterruptedException e) {}
driver.quit();
The reason why this happens is the following:
If you want to open this page you have to have granted access. To have so you have to first login on the main webpage.
For other people having a similar issues of getting redirected:
Use a different user agent when you set up your Webdriver and switch to a either mobile or PC/MAC, depending on your needs.
Cheers
Have you seen that /featured is redirecting to the root site literally?
I would go for that first. Probably is something related with it and you will end with the same result if you connect to https://www.liketoknow.it/ in the first place.

Using selenium to download a file via window.open

I'm trying to scrape a webpage where clicking a link results in a new window popping open that immediately downloads a csv. I haven't been able to figure out the format of the url since it's fairly dense javascript (and one function is called via the onClick property while another is called as part of the href property. I have not worked with Selenium before, so I was hoping to confirm before getting started that what I want to do is possible. I had read somewhere that downloading files via new popup windows is not necessarily something I can do with Selenium.
Any advice would be greatly appreciated. A this is possible would be very helpful as would as here's how you'd do it even sketched in broad detail. Thanks much!
To be clear, my difficulties primarily stem from the fact that I can't figure out how the URL to download the file is generated. Even looking at the Google chrome network calls, I am not seeing where it is, and it would probably take me many hours to track this down, so I am looking for a solution that relies on clicking specific text in the browser rather than disentangling the cumbersome machinery behind the scenes.
Here's how I download files using Firefox webdriver. It's essentially creating a browser profile so that the default download location for certain file types are set. You can then verify if the file exists at that location.
import os
from selenium import webdriver
browser_profile = webdriver.FirefoxProfile()
# add the file_formats to download
file_formats = ','.join(["text/plain",
"application/pdf",
"application/x-pdf",
"application/force-download"])
preferences = {
"browser.download.folderList": 2,
"browser.download.manager.showWhenStarting": False,
"browser.download.dir": os.getcwd(), # will download to current directory
"browser.download.alertOnEXEOpen": False,
"browser.helperApps.neverAsk.saveToDisk": file_formats,
"browser.download.manager.focusWhenStarting": False,
"browser.helperApps.alwaysAsk.force": False,
"browser.download.manager.showAlertOnComplete": False,
"browser.download.manager.useWindow": False,
"services.sync.prefs.sync.browser.download.manager.showWhenStarting": False,
"pdfjs.disabled": True
}
for pref, val in preferences.items():
browser_profile.set_preference(pref, val)
browser_binary = webdriver.firefox.firefox_binary.FirefoxBinary()
browser = webdriver.Firefox(firefox_binary=browser_binary,
firefox_profile=browser_profile)
# set the file name that will be saved as when you download is complete
file_name = 'ABC.txt'
# goto the link to download the file from it will be automatically
# downloaded to the current directory
file_url = 'http://yourfiledownloadurl.com'
browser.get(file_url)
# verify if the expected file name exists in the current directory
path = os.path.join(os.getcwd(), file_name)
assert os.path.isfile(path)

Categories

Resources