I have code in place where I'm using EO.WebBrowser to get the html from a page using the EO.WebView Request:
var cookie = new EO.WebBrowser.Cookie("cookie", "value");
cookie.Path = path;
cookie.Domain = domain;
var options = new BrowserOptions();
options.EnableWebSecurity = false;
Runtime.SetDefaultOptions(options);
var request = new Request(url);
request.Cookies.Add(cookie);
webView.LoadRequestAndWait(request);
Finally I use the following to get the HTML I need:
webView.GetDOMWindow().document.body.outerHTML
My issue is that this is very slow and although I can get it to run it locally, I can not get it to run on Azure server code. Is there a way to do the same thing using HttpWebRequest?
You can use JavaScript:
var data = (string)webView.EvalScript("document.body.outerHTML");
No, HttpWebRequest (and other similar "get me HTML response") methods will only give you HTML itself and will not run JavaScript on the page.
For server side processing of dynamic HTML consider using proper headless internet browser? instead of trying to convince regular IE to work correctly without UI.
the eo.webbrowser runs multi-process like chrome and unsupported by many cloud service environment.
just use WebClient or HttpWebRequest or RestSharp or something like that can do http requests to get the response html.
Related
I am trying to integrate SigWeb API into my application. I want to use the digital signature in all modern browsers. Now it works only in IE, using ActiveX object. The problem is that I can not use the encryption key.
Let me explain:
Old version js code looks like:
SigPlus1.AutoKeyStart();
SigPlus1.AutoKeyData = "Some Key Data";
SigPlus1.AutoKeyFinish();
SigPlus1.EncryptionMode = 2;
SigPlus1.SigCompressionMode = 1;
var strSignature = SigPlus1.SigString;
New version (using SigWebTablet.js):
AutoKeyStart();
SetAutoKeyData("Some Key Data");
AutoKeyFinish();
SetEncryptionMode(2);
SetSigCompressionMode(1);
var strSignature = GetSigString();
The value of strSignature is passed to the server and converted to .jpg file. On the server side (java) I am using following code:
ClassLoader firma = (com.topaz.sigplus.SigPlus.class).getClassLoader();
sigObj = (SigPlus)Beans.instantiate(firma, "com.topaz.sigplus.SigPlus");
sigObj.autoKeyStart();
sigObj.setAutoKeyData("Some Key Data"); // the same data in front-end
sigObj.autoKeyFinish();
sigObj.setEncryptionMode (2);
sigObj.setSigCompressionMode(1);
sigObj.setSigString(strSignature);
The problem in setSigString method - it doesn't set the new value (with the old code SigPlus1.SigString works), but if I disable setAutoKeyData - it works fine.
The tablet model: T-LBK766SE-BHSB-R
I have found a solution which works for me. Before signature capture it needs to reset the encryption mode, i.e. to call SetEncryptionMode(0)
UPD (05/03/2017). Guys from dev support recommended me to use AutoKeyAddData function instead AutoKeyStart and AutoKeyFinish functions. I have tested and it works for me. I think this better solution of this issue.
http://www.sigplusweb.com/sigwebtablet_autokeydemo.htm
I'm trying to load a webpage in my application background. following code shows How I am loading a page:
request = (HttpWebRequest)WebRequest.Create("http://example.com");
request.CookieContainer = cookieContainer;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
Stream st = response.GetResponseStream();
StreamReader sr = new StreamReader(st);
string responseString = sr.ReadToEnd();
sr.Close();
st.Close();
}
as you know, the server responses HTML codes or some javascript codes, but there are many codes which added to the webpage by javascripts functions. so I have to interpret or compile the first HTTP response.
I tried to use System.Windows.Forms.WebBrowser object to load the webpage completely, but this is a weak engine to do this.
so I tried to use CEFSharp (Chromium embedded Browser), it's great and works fine but I have trouble with that. following is how I use CEFSharp to load a webpage:
ChromiumWebBrowser MainBrowser = new ChromiumWebBrowser("http://Example/");
MainBrowser.FrameLoadEnd+=MainBrowser.FrameLoadEnd;
panel1.Controls.Add(MainBrowser);
MainBrowser.LoadHtml(responseString,"http://example.com");
it works fine when I use this code in Form1.cs and when I add MainBrowser to a panel. but I want to use it in another class, actually ChromiumWebBrowser is part of another custom object and the custom object works in background. also it would possible 10 or 20 custom objects work in a same time. in this situation ChromiumWebBrowser doesn't work any more!
second problem is the threading issue, when I call this function MainBrowser.LoadHtml(responseString,"http://example.com");
it doesn't return any results, so I have to pause the code execution by using Semaphore and wait for the result at this event: MainBrowser.FrameLoadEnd
so I wish my code be some thing like this:
request = (HttpWebRequest)WebRequest.Create("http://example.com");
request.CookieContainer = cookieContainer;
string responseString="";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
Stream st = response.GetResponseStream();
StreamReader sr = new StreamReader(st);
responseString = sr.ReadToEnd();
sr.Close();
st.Close();
}
string FullPageContent = SomeBrowserEngine.LoadHtml(responseString);
//Do stuffs
Can you please show me how to do this? do you know any other web browser engines that work like what I want?
please tell me if I'm doing any things wrong with CEFSharp or other concepts.
I want to scrape my college website (moodle) with node.js but I haven't found a headless browser able to do it. I have done it in python in just 10 lines of code using RoboBrowser:
from robobrowser import RoboBrowser
url = "https://cas.upc.edu/login?service=https%3A%2F%2Fatenea.upc.edu%2Fmoodle%2Flogin%2Findex.php%3FauthCAS%3DCAS"
browser = RoboBrowser()
browser.open(url)
form = browser.get_form()
form['username'] = 'myUserName'
form['password'] = 'myPassword'
browser.submit_form(form)
browser.open("http://atenea.upc.edu/moodle/")
print browser.parsed
The problem is that the website requires authentication. Can you help me? Thanks!
PD: I think this can be useful https://www.npmjs.com/package/form-scraper but I can't get it working.
Assuming you want to read a 3rd party website, and 'scrape' particular pieces of information, you could use a library such as cheerio to achieve this in Node.
Cheerio is a "lean implementation of core jQuery designed specifically for the server". This means that given a String representation of a DOM (or part thereof), cheerio can traverse it in much the same way as jQuery can.
An example from Max Ogden show how you can use the request module to grab HTML from a remote server and then pass it to cheerio:
var $ = require('cheerio')
var request = require('request')
function gotHTML(err, resp, html) {
if (err) return console.error(err)
var parsedHTML = $.load(html)
// get all img tags and loop over them
var imageURLs = []
parsedHTML('a').map(function(i, link) {
var href = $(link).attr('href')
if (!href.match('.png')) return
imageURLs.push(domain + href)
})
}
var domain = 'http://substack.net/images/'
request(domain, gotHTML)
Selenium has support for multiple languages and multiple platforms and multiple browsers.
How can you use Javascript to parse out the URL of a page?
First of all, you have to decide whether you want to do this on the client or on the server. On the server, you can load the XML and use XPath to locate the part of the XML DOM tree that contains the site:
//site/name[text() = 'Blah00']
When using JavaScript on the client, a better solution would be to have a server which keeps the current status per site (use a database or some in-memory structure). Then use AJAX requests to ask the server for the information for a certain site. jQuery will make your life much easier.
I have solved this:
<script>
function mySiteURL() {
var myURL = window.location.href;
var dashIndex = myURL.lastIndexOf("-");
var dotIndex = myURL.lastIndexOf(".");
var result = myURL.substring(dashIndex + 1, dotIndex);
}
</script>
Is there any way to work with external Data, for example Textfiles ?
With JS there isn´t any option to do I/O, for a good reason.
The purpose is, that i want to validate data from a file or a database with the data represented in the APP (comparison tests)…
Is there any trick for the UI Automation tool ?
Text file
This is something I came up with based on performTaskWithPathArgumentsTimeout method:
var target = UIATarget.localTarget();
var host = target.host();
var result = host.performTaskWithPathArgumentsTimeout(
"/bin/cat", ["/Users/username/Documents/test.txt"], 5)
var content = result.stdout;
UIALogger.logMessage(content);
Content of the test.txt:
Hello World!
This is a test.
And this is how it looks in the Editor Log:
The downside is that you'll have to manually parse file's content.
Database
The only feasible option I can see, is to use this same method in conjunction with Command Line Shell For SQLite. But, alas, I haven't tried this approach, so I can't guarantee it'll work.