Develop Reference

Kotlin, how can I read a dynamic website as text?

As titled, I'm trying to read the content of sites like this one, which appears to be javascript based.
I tried using plain jdk lib, then jsoup and then htmlunit, but I couldn't get anything useful out of it (I see just the source code or just the title or null):
val url = URL("https://registry.terraform.io/providers/hashicorp/tls/latest/docs/data-sources/certificate")
val connection = url.openConnection()
val scanner = Scanner(connection.getInputStream())
scanner.useDelimiter("\\Z")
val content = scanner.next()
scanner.close()
println(content)
val doc = Jsoup.connect("https://registry.terraform.io/providers/hashicorp/tls/latest/docs/data-sources/certificate").get()
println(doc.text())
WebClient().use { webClient ->
val page = webClient.getPage<HtmlPage>("https://registry.terraform.io/providers/hashicorp/tls/latest/docs/data-sources/certificate")
val pageAsText = page.asNormalizedText()
println(pageAsText)
}
WebClient(BrowserVersion.FIREFOX).use { webClient ->
val page = webClient.getPage<HtmlPage>("https://registry.terraform.io/providers/hashicorp/tls/latest/docs/data-sources/certificate")
println(page.textContent)
}
It should be something easy peasy, but I cant see what's wrong

In order for this to be possible, you need something to execute the JS that modifies the DOM.
It might be a bit overkill depending on the use case, and probably won't be possible if you're on Android, but one way to do this is to launch a headless browser separately and interact with it from your code. For instance, using Chrome Headless and the Chrome DevTools Protocol. If you're interested, I have written a Kotlin library called chrome-devtools-kotlin to interact with a Chrome browser in a type-safe way.
There might be simpler options, though. For instance maybe you can run an embedded browser instead with JBrowserDriver and still use JSoup to parse the HTML, as mentioned in this other answer.

Regarding HtmlUnit:
the page has initially no content, all you see is rendered from javascript magic on the client side using one of this spa frameworks.
It looks like there is some feature check in the beginning that figures out the js support in HtmlUnit does not have all the required features and based on this you only get a hint like "Please enable Javascript to use this application".
You can use
page.asXml()
to have a look at the content trough HtmlUnit's eyes.
You can open an HtmlUnit issue on github but i fear adding support for this will be a longer story.

Angular Universal and browser feature checks

When developing a web app with jQuery or normal JavaScript, it is commonplace to check for feature availability first. So for example, if I want to use the document.oncopy event, I should first have something like this to ensure my code doesn't break for lesser browsers:
if ("oncopy" in document) {
// Feature is available
}
I'm a bit puzzled about how this would work in Angular2. I could still use the same if I expect to only run in the browser, but I'm specifically told to leave the DOM alone if I want to use Angular Universal and depend on templates or the DomRenderer instead. This allows the page to be pre-rendered on the server and provides a truly impressive performance gain.
But suppose I want a specific div to be invisible if the document.oncopy is unavailable. My understanding is that this is not recommended:
<div *ngIf="hasFeature()">...</div>
and
hasFeature() {
return 'oncopy' in document;
}
because then I'm still manipulating the DOM. Note that my example is about the document.oncopy but I could choose any feature whatsoever that doesn't have universal support.
I tested this using Chris Nwamba's tutorial on Scotch and added the following to the end of his Home template:
<div *ngIf="hasFeature()">Feature is supported</div>
<div *ngIf="!hasFeature()">Feature is NOT supported</div>
Update: Interestingly, it gave different results on different browsers. On Chrome 55, it executed as it would normally and showed the "Feature is supported" message. On IE11, I received the "not supported" message. In both instances the server log shows a EXCEPTION: document is not defined message, but the page still seems perfectly okay.
So what is the correct way to check for browser features if I want to use Angular Universal?
Update:
I also toyed around with using a field in the template and assigning that field from one of the life cycle hooks. ngAfterContentInit seemed like a fine candidate, but also causes an error on the server. It still runs fine in the browser with no weird effects (that I have noticed so far).

There are two ways to approach this:
Do the check only once the server is done rendering and the client is completely initialised (including the replay of user events done by preboot.js).
Return a reasonable default when the page is running on the server and perform the actual check only in the browser.
I started looking at the first option, but none of the Angular2 life cycle events will help with this. In fact, you can clearly see them all executing on the server and only then on the client.
I then started looking for something usable in preboot.js but quickly realised it was more complex than it needed to be.
So onto option 2 I went. It turns out checking for the browser is as easy as importing and checking isBrowser.
import { isBrowser } from "angular2-universal";
#Component({
// All the usual stuff
})
export class MyComponent {
// ...
hasFeature(): boolean {
return isBrowser && 'oncopy' in document;
}
// ...
}
And then use the template as I showed in the question.
To check if you're running on the server, import and use isNode in exactly the same way. There doesn't seem to be an obvious way to distinguish between Node and ASP.NET Core, but perhaps it's best not to write too much code that specific to platform.

Browser.ExecScript() stopped working after updating windows

I've set up a simple testbed for WatiN (ver 2.1) which reads:
var browser = new IE();
browser.GoTo("http://www.google.co.il"); // webpage doesn't matter really
browser.RunScript("alert(123)");
This works only if KB3025390 is not installed. Installing it breaks the above test with an UnAuthorizedAccessException which has HRESULT set to E_ACCESSDENIED. What gives? Is there any workaround?
Update: Using IWebBrowser2.Navigate2 along with "javascript:console.log(123)" type of scripts works however
it makes me feel uneasy using such a backchannel
the scripts run through this back-channel of .Navigate2() may only have a max length of about 2070 chars (give or take) otherwise they get forcibly truncated to this length leading to javascript errors upon attempting to run them
using .Navigate2(), even with the most trivial script, will clog the ready state of Internet Explorer for good in the sense that it will be set to READYSTATE_LOADING without any hope of getting rid of it. In simple terms this means that once you use this hack, you either have to perform every single subsequent operation in WatiN in a "dont-wait-for-webpage-to-load" fashion (GoToNoWait, ClickNoWait etc) lest your code freezes upon waiting for the browser to turn back to READYSTATE_COMPLETE (which will never come about ofcourse as already mentioned).
there appears to be a much broader issue here in the sense that I can't even access the properties of an IHtmlWindow2 object p.e. window.document throws an unauthorized exception again making it virtually impossible to transfer over to the C# world the return-values of the scripts I'm running (using Expando etc) for documents other than window.top.document (for the window.top.document window there is IWebBrowser2.Document which does the trick)
Update#2: The folks over at the selenium project have also noticed this issue:
https://code.google.com/p/selenium/issues/detail?id=8302
A bug report has been created as well:
https://connect.microsoft.com/IE/feedback/details/1062093/installation-of-kb3025390-breaks-out-of-process-javascript-execution-in-ie11
Update#3: IHTMLWindow2.setInterval and IHTMLWindow2.setTimeout also throw UnauthorizedAccess exceptions. These methods are not marked as deprecated in:
http://msdn.microsoft.com/ko-kr/library/windows/desktop/aa741505%28v=vs.85%29.aspx
yet they have wounded up suffering from the same cutbacks all the same.
Update#4: I gave the approach recommended in this post a shot:
https://stackoverflow.com/a/18546866/863651
In order to dynamically invoke the "eval" method of the IHTMLWindow2 object (or any other method really). Got the same "System.UnauthorizedAccessException" as above. So no joy here either.
Microsoft recommends using "eval" over "execscript" however after the above experiment I suspect that they are refering to accessing "eval" only from within the browser.
As far as I can tell thus far, when it comes to the full-fledged IE11+ using "eval" out-of-process (via COM) appears to have been completely prohibited along with any other function-invocation of the window object, the only exception being the back-channel of the .Navigate2() mentioned above.

It turns out Microsoft eventually backpedaled on its decision to kill off .execScript at COM-level. Just install the latest updates for Windows including kb3025390: One of the updates for IE that came after kb3025390 brings back .execScript functionality at COM-level
Note, however, that .execScript is not accessible through IE's javascript anymore. In that context it's gone for good.

fyi: this one is also not working
ieInstance.Document.Script.<methodNameString>(<commaSeperatedParameterString>)
try this worked for me at some places but not all places
ieObject.Navigate "javascript:<methodNameString>(<commaSeperatedParameterString>)", Null, "_parent"
or
ieObject.Navigate2 "javascript:"<methodNameString>(<commaSeperatedParameterString>)", Null, "_parent"
now trying to find out solution using eval

I have found a way around the problem of an update installing automatically. You can just create a simple batch file with following content.
{code}
#echo off
wusa /uninstall /kb:3025390/quiet /norestart
END
{code}
Then go to task scheduler, create a new task for this batch file to run every one hour or day as per your requirements. Add it as a system task so it runs in the background and does not affect the running automations.

JavaScript for automation in OSX Yosemite

Can we send a click event to a button inside a webview in Cocoa app using JavaScript?
I am trying to use the script editor under utilities (Yosemite) to record but unfortunately not able to record any events inside the webview.
I tried using the sample code under UI automation section provided in the Apple documentation with the testapp (cocoa app with webview) at https://developer.apple.com/library/mac/releasenotes/InterapplicationCommunication/RN-JavaScriptForAutomation/index.html.
TestApp = Application('TestApp')
TestApp.activate()
delay(1)
SystemEvents = Application('System Events')
TestApp = SystemEvents.processes['TestApp']
TestApp.document.getElementById('testid').click(); // stuck at this last line not sure if I can
//even call the document object in this way. Getting error undefined variable document.

You're trying to use browser/DOM Javascript to access native app UI elements. Although this environment uses Javascript like a browser does, the underlying object model is not the DOM you're used to seeing on a web page. That's why you're seeing that document is undefined.
As implied by the little snippet in the "UI Automation" section of the docs, you need to access the window and button objects. The exact path you'll use depends on your TestApp, of course, but it might look something like:
TestApp.windows[0].buttons[0].click()
(It is probably also possible to scrobble through these arrays by control ID using whose, or the like, but don't have experience with that.)

In what situation would document.open() return null?

I'm trying to understand an intermittent script error that I am seeing in a JavaScript intensive thin-client application running under Internet Explorer 6 and Windows XP. The root cause of the problem is that the following function call returns a null value (however it does succeed without an error):
var doc = targetWindow.document.open("text/html","_replace");
Where targetWindow is a window object.
Neither targetWindow nor targetWindow.document is null and so I'm struggling to understand why this call would return null. My interpretation of the documentation is that this method shouldn't ever return null.
This code has been unchanged and working perfectly for many years - until I understand why this is happening I'm not sure either how I might handle this, or what might have changed to cause this to start happening.
What might cause this function call to return null?

According to the documentation you should be passing "replace", not "_replace". Try this instead:
var doc = targetWindow.document.open("text/html", "replace");
Since you say your code has worked for years, then it is likely that something has changed and the above suggestion may not be the issue. However, it is still worth a try.
Have you changed any js files / libraries you are using in your application lately? Also, are you using any browser plugins within the page? It is possible that a newer version of either of these could be somehow affecting your call to "document.open".

document.open() does not have any parameters by W3C standard. Check out this link: http://www.w3.org/TR/DOM-Level-2-HTML/html.html#ID-72161170
I recommend you to use W3C documentation instead of Microsoft's one because with W3C you are sure it works on all modern browsers, while Microsoft is well known for adding extensions that, of course, works only in their own products. It's called EEE (Embrace, extend and extinguish).
Simply use document.open() without arguments. There are ways to manipulate user history, but that's called bad programming practice. History is user's private data and web application should not try to manipulate it.