Develop Reference

Can't retrieve HTML code generated by JavaScript (I use Java+Selenium)

I have run into a problem during I tried scrape data from website niche.com
I know that website is use JavaScript to render pages.
I have used Java and Selenium (tried FireFox and Chrome, both give the same results).
I tried download page https://www.niche.com/k12/search/best-schools/c/santa-clara-county-ca/
and save rendered page into text file on my laptop.
The code:
private WebDriver driver = new ChromeDriver();
public final String load(String url) {
String html = "";
try {
driver.navigate().to(url);
html = gotHtml();
} catch (Exception ex) {
logger.warn(ex);
}
return html;
}
public String gotHtml() {
try {
return ((JavascriptExecutor) driver).executeScript("return document.documentElement.outerHTML;").toString();
} catch (Exception ex) {
logger.error(ex);
}
return driver.getPageSource();
}
On all other websites the code "return document.documentElement.outerHTML;" returned the rendered page, but for this one that code return not rendered page (page with javascript code, but without rendered html code).
This div must contains the rendered HTML code, but I got it empty:
<div class="platform__wrapper" id="app"><!-- react-empty: 1 --></div>
Any ideas?

Android load pdf in Webview locally from SDcard

I am trying to load a pdf from my sd onto webview.
I know it is not possible to direct load a pdf in webview. So I have a html file which renders the file and load the pdf.
sample from http://www.worldwidewhat.net/2011/08/render-pdf-files-with-html5/
Android Code:
final WebView webView = (WebView) findViewById(R.id.magazineWebView);
final ProgressBar spinner = (ProgressBar) findViewById(R.id.magazineSpinner);
webView.setBackgroundColor(0xFFFFFFFF);
webView.getSettings().setJavaScriptEnabled(true);
webView.getSettings().setSaveFormData(true);
webView.getSettings().setBuiltInZoomControls(mgznCanZoom);
webView.getSettings().setDisplayZoomControls(false);
webView.getSettings().setLoadWithOverviewMode(true);
webView.getSettings().setUseWideViewPort(true);
webView.getSettings().setPluginState(WebSettings.PluginState.ON);
webView.setWebViewClient(new WebViewClient() {
#Override
public void onPageFinished(WebView view, String url) {
spinner.setVisibility(View.GONE);
}
#Override
public boolean shouldOverrideUrlLoading(WebView view, String url) {
view.loadUrl(url);
return true;
}
});
String webPath = "http://mysite/demo/pdf-js/index.htm"; //WORKS
String file_name2 = "pdf-js/index.htm"; //LOAD HTML BUT NOT PDF
String file_uri = MAGAZINE_FOLDER + file_name2;
String sdPath = Uri.parse("file://"
+ Environment.getExternalStorageDirectory()
+ file_uri).toString();
webView.loadUrl(sdPath); //webPath
Erros from logcat:
I/chromium: [INFO:CONSOLE(0)] "XMLHttpRequest cannot load file:///storage/emulated/0/MyApp/Magazine/pdf-js/compressed.tracemonkey-pldi-09.pdf. Cross origin requests are only supported for protocol schemes: http, data, chrome, https.", source: file:///storage/emulated/0/MyApp/Magazine/pdf-js/index.htm (0)
I/chromium: [INFO:CONSOLE(52)] "Uncaught TypeError: Cannot read property 'Symbol(Symbol.iterator)' of null", source: file:///storage/emulated/0/MyApp/Magazine/pdf-js/lib/pdf.js (52)
I/AppLifecycle: onActivitySaveInstanceState
But ->
If I store this file in the cloud and load it as above, it will work fine.
Anyone know why this wont load locally ?
Thanks guys.
P.S I know there are lot of these question online. I have looked around and not found a solution.
I am not able to use librarys like this https://github.com/JoanZapata/android-pdfview

Download WebView content in WInRT application

I'm trying to build a universal rss application for Windows 10 that could be able to download the content of the full article's page for offline consultation.
So after spending a lot of time on stackoverflow I've found some code:
HttpClientHandler handler = new HttpClientHandler { UseDefaultCredentials = true, AllowAutoRedirect = true };
HttpClient client = new HttpClient(handler);
HttpResponseMessage response = await client.GetAsync(ni.Url);
response.EnsureSuccessStatusCode();
string html = await response.Content.ReadAsStringAsync();
However this solution doesn't work on some web page where the content is dynamically called.
So the alternative that remains seems to be that one: load the web page into the Webview control of WinRT and somehow copy and paste the rendered text.
BUT, the Webview doesn't implement any copy/paste method or similar so there is no way to do it easily.
And finally I found this post on stackoverflow (Copying the content from a WebView under WinRT) that seems to be dealing with the same exact problematic as mine with the following solution;
Use the InvokeScript method from the webview to copy and paste the content through a javascript function.
It says: "First, this javascript function must exist in the HTML loaded in the webview."
function select_body() {
var range = document.body.createTextRange();
range.select();
}
and then "use the following code:"
// call the select_body function to select the body of our document
MyWebView.InvokeScript("select_body", null);
// capture a DataPackage object
DataPackage p = await MyWebView.CaptureSelectedContentToDataPackageAsync();
// extract the RTF content from the DataPackage
string RTF = await p.GetView().GetRtfAsync();
// SetText of the RichEditBox to our RTF string
MyRichEditBox.Document.SetText(Windows.UI.Text.TextSetOptions.FormatRtf, RTF);
But what it doesn't say is how to inject the javascript function if it doesn't exist in the page I'm loading ?

If you have a WebView like this:
<WebView Source="http://kiewic.com" LoadCompleted="WebView_LoadCompleted"></WebView>
Use InvokeScriptAsync in combination with eval() to get the document content:
private async void WebView_LoadCompleted(object sender, NavigationEventArgs e)
{
WebView webView = sender as WebView;
string html = await webView.InvokeScriptAsync(
"eval",
new string[] { "document.documentElement.outerHTML;" });
// TODO: Do something with the html ...
System.Diagnostics.Debug.WriteLine(html);
}

Android Web-View : Inject local Javascript file to Remote Webpage

It has been asked many times before, I browsed through everything, no clear answers yet.
Question simplified: Is it possible to inject local Javascript file (from asset or storage) to remote webpage loaded in an Android Web-View? I know that it is possible to inject such files to local Webpages (Assets HTML) loaded in a Web-View.
Why do I need this to work? : To make browsing experience faster, by avoiding downloading of bigger files such as Js and CSS files every time. I want to avoid Web-View Caching.

There is a way to 'force' the injection of your local Javascript files from local assets (e.g., assets/js/script.js), and to circumvent the 'Not allowed to load local resource : file:///android_assets/js/script.js ...' issue.
It is similar to what described in another thread (Android webview, loading javascript file in assets folder), with additional BASE64 encoding/decoding for representing your Javascript file as a printable string.
I am using an Android 4.4.2, API level 19 Virtual Device.
Here are some code snippets:
[assets/js/script.js]:
'use strict';
function test() {
// ... do something
}
// more Javascript
[MainActivity.java]:
...
WebView myWebView = (WebView) findViewById(R.id.webView);
WebSettings webSettings = myWebView.getSettings();
webSettings.setJavaScriptEnabled(true);
webSettings.setAllowUniversalAccessFromFileURLs(true);
myWebView.setWebViewClient(new WebViewClient() {
#Override
public boolean shouldOverrideUrlLoading(WebView view, String url) {
return false;
}
#Override
public void onPageFinished(WebView view, String url) {
super.onPageFinished(view, url);
injectScriptFile(view, "js/script.js"); // see below ...
// test if the script was loaded
view.loadUrl("javascript:setTimeout(test(), 500)");
}
private void injectScriptFile(WebView view, String scriptFile) {
InputStream input;
try {
input = getAssets().open(scriptFile);
byte[] buffer = new byte[input.available()];
input.read(buffer);
input.close();
// String-ify the script byte-array using BASE64 encoding !!!
String encoded = Base64.encodeToString(buffer, Base64.NO_WRAP);
view.loadUrl("javascript:(function() {" +
"var parent = document.getElementsByTagName('head').item(0);" +
"var script = document.createElement('script');" +
"script.type = 'text/javascript';" +
// Tell the browser to BASE64-decode the string into your script !!!
"script.innerHTML = window.atob('" + encoded + "');" +
"parent.appendChild(script)" +
"})()");
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
});
myWebView.loadUrl("http://www.example.com");
...

loadUrl will work only in old version use evaluateJavascript
webview.evaluateJavascript("(function() { document.getElementsByName('username')[0].value='USERNAME';document.getElementsByName('password')[0].value='PASSWORD'; "+
"return { var1: \"variable1\", var2: \"variable2\" }; })();", new ValueCallback<String>() {
#Override
public void onReceiveValue(String s) {
Log.d("LogName", s); // Prints: {"var1":"variable1","var2":"variable2"}
}
});

Yes, you could use shouldInterceptRequest() to intercept remote url loading and return local stored content.
WebView webview = (WebView) findViewById(R.id.webview);
webview.setWebViewClient(new WebViewClient() {
#Override
public WebResourceResponse shouldInterceptRequest (final WebView view, String url) {
if (url.equals("script_url_to_load_local")) {
return new WebResourceResponse("text/javascript", "UTF-8", new FileInputStream("local_url")));
} else {
return super.shouldInterceptRequest(view, url);
}
}
});

Be careful using evaluateJavascript: if there is a syntax error or exception thrown in your javascript it will call your onReceiveValue with a null. The most common way to support both SDK 19 as well as lower seems to be like this:Fill form in WebView with Javascript
Also if you get terribly desperate for some kind of browser functionality (in my case, never could figure out how to get DRM to work well) you could use a bookmarklet within normal chrome, which works only if you type the bookmark name into the omnibox but does work and does inject javascript.
Also be aware that with the default WebView you can't use javascript alerts to test anything, they don't show. Also be aware that "video" by default (like html <video> tags) doesn't "really work" by default and also DRM video doesn't work by default, they're all configure options :\

Javascript is not working in webView when Loaded from asset folder but working from both http server & localhost

I am trying to load offline version of python documentation into an webview from assetfolder. The offline docs work perfectly in my pc web browser in offline but not working properly in webview (something like jquery is missing).
#SuppressLint("SetJavaScriptEnabled")
public class MainActivity extends Activity {
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
WebView webView = (WebView) findViewById(R.id.wrapper);
webView.getSettings().setJavaScriptEnabled(true);
webView.setWebChromeClient(new WebChromeClient());
webView.loadUrl("file:///android_asset/python/index.html");
}
}
And this error message is shown when I tried to load the home page or navigate to any page.
09-24 01:03:02.789: E/Web Console(479): ReferenceError: Can't find variable: $ at file:///android_asset/python/index.html:164
And the above error is for a Jquery code snippet ( I think this for that the jquery library isn't loading)
<script type="text/javascript">$('#searchbox').show(0);</script>
But when I load those pages from my local server localhost or http server, this is working perfectly. What did I miss?
Edit
Showing nothing in the webView. After using loadDataWithBaseURL:
#SuppressLint("SetJavaScriptEnabled")
public class MainActivity extends Activity {
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
WebView webView = (WebView) findViewById(R.id.wrapper);
webView.getSettings().setJavaScriptEnabled(true);
webView.setWebChromeClient(new WebChromeClient());
AssetManager assetManager = getAssets();
String htmlPage = null;
InputStream input;
try {
input = assetManager.open("python/index.html");
int size = input.available();
byte[] buffer = new byte[size];
input.read(buffer);
input.close();
// byte buffer into a string
htmlPage = new String(buffer);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
webView.loadDataWithBaseURL("file:///android_asset/", htmlPage, "text/html", "utf-8", "");
}
}

Your html page should be having reference to
<script type="text/javascript" src="http://code.jquery.com/jquery-1.6.4.min.js">
</script>
Since you say the docs should load offline the jquery js is not being loaded. You could probably bundle jquery along with your application and reference it locally like this
<script src="file:///android_asset/js/jquery-1.8.2.min.js"></script>
Also include
<script src="file:///android_asset/js/jquery.mobile-1.3.2.min.js"></script>
You might also have to load your html page using loadDataWithBaseURL as seen below instead of loadUrl.
AssetManager assetManager = getAssets();
String htmlPage=null;
InputStream input;
try {
input = assetManager.open("python/index.html");
int size = input.available();
byte[] buffer = new byte[size];
input.read(buffer);
input.close();
// byte buffer into a string
htmlPage = new String(buffer);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
webView.loadDataWithBaseURL("file:///android_asset/", htmlPage, "text/html", "utf-8", "");
Note: jquery-1.8.2.min.js and jquery.mobile-1.3.2.min.js files should be present in your assets folder.
Hope this helps.

The problem was - I extracted those web files from a phonegap app's asset folder; those had some additional files ad different structure. That's why it was not working in Android naive environment !