Develop Reference

Get the source DOM/HTML of an external page from URL

I want to download source code of page from url in Chrome extension.
I have something like this. But don't know how to format text to html.
Or if this formatting works how to display the source code even in console.
fetch('https://www.transfermarkt.com/robert-lewandowski/profil/spieler/38253').then(r => r.text()).then(result => {
// Result now contains the response text, do what you want...
console.log("Fetch result: " + result);
var parser = new DOMParser();
var source_code = parser.parseFromString(result, "text/html");
console.log("Source code: " + source_code);
});
For example I would like to get text from span "dataValue".
How can I achieve that ?

The first console.log below is the textual representation.
The second is the DOM object where you can get stuff to show.
fetch('https://www.transfermarkt.com/robert-lewandowski/profil/spieler/38253').then(r => r.text()).then(result => {
// Result now contains the response text, do what you want...
// console.log("Source code: " + result); // textual representation
var parser = new DOMParser();
var DomObject = parser.parseFromString(result, "text/html");
console.log("Name: " + DomObject.querySelector("title").textContent);
[...DomObject.querySelectorAll(".dataItem")].forEach(item => {
if (item.textContent.trim() === "Joined:") {
console.log("Joined:",item.nextElementSibling.textContent);
}
});
})
P.S. Don't forget to add the site (or "<all_urls>") in manifest.json:
ManifestV2: "permissions": ["*://*.transfermarkt.com/"]
ManifestV3: "host_permissions": ["*://*.transfermarkt.com/"]

Download embedded PDF loaded in WKWebView

While loading HTML5 page from url I'm getting pdf somewhere in that page and I have to download that pdf or save it as base64.
This is where the pdf is in the HTML code. I cannot simply hit the 'src' URL and get the pdf.
< embed width="100%" height="100%" name="plugin" id="plugin" src="https://myurl.com/fileToOpen.pdf” type="application/pdf" internalinstanceid="8" title="">
Any JS which can help me get base64 string of that or any other method to download?

This question asked sometimes back,
But if someone looking for swift solution with WKWebView to download .pdf or any file on File manager, this is how I ended-up
class WebPortalVC: UIViewController, WKNavigationDelegate,WKUIDelegate, UIDocumentInteractionControllerDelegate,URLSessionDownloadDelegate {
override following function, that will intercept url, in our case we check ulr ending with .pdf and .csv and redirect to open with file manger view. which enable view file, download and save on device storage, airdrop or share with other apps
just add following functions and check.
func webView(_ webView: WKWebView, decidePolicyFor navigationAction: WKNavigationAction, decisionHandler: #escaping (WKNavigationActionPolicy) -> Void) {
if let url = navigationAction.request.url {
print("fileDownload: check :: \(url)")
let extention = "\(url)".suffix(4)
if extention == ".pdf" || extention == ".csv"{
print("fileDownload: redirect to download events. \(extention)")
DispatchQueue.main.async {
self.downloadPDF(tempUrl: "\(url)")
}
decisionHandler(.cancel)
return
}
}
decisionHandler(.allow)
}
func downloadPDF(tempUrl:String){
print("fileDownload: downloadPDF")
guard let url = URL(string: tempUrl) else { return }
let urlSession = URLSession(configuration: .default, delegate: self, delegateQueue: OperationQueue())
let downloadTask = urlSession.downloadTask(with: url)
downloadTask.resume()
//showHUD(isShowBackground: true); //show progress if you need
}
func documentInteractionControllerViewControllerForPreview(_ controller: UIDocumentInteractionController) -> UIViewController {
print("fileDownload: documentInteractionControllerViewControllerForPreview")
return self
}
func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didFinishDownloadingTo location: URL) {
// create destination URL with the original pdf name
print("fileDownload: urlSession")
guard let url = downloadTask.originalRequest?.url else { return }
print("fileDownload: urlSession \(url)")
let documentsPath = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0]
let destinationURL = documentsPath.appendingPathComponent(url.lastPathComponent)
// delete original copy
try? FileManager.default.removeItem(at: destinationURL)
// copy from temp to Document
do {
try FileManager.default.copyItem(at: location, to: destinationURL)
myViewDocumentsmethod(PdfUrl:destinationURL)
print("fileDownload: downloadLocation", destinationURL)
DispatchQueue.main.async {
NBMaterialToast.showWithText(self.view, text: "Download Completed", duration: NBLunchDuration.long)
}
} catch let error {
print("fileDownload: error \(error.localizedDescription)")
}
// dismissHUD(isAnimated: false); //dismiss progress
}
func myViewDocumentsmethod(PdfUrl:URL){
print("fileDownload: myViewDocumentsmethod \(PdfUrl)")
DispatchQueue.main.async {
let controladorDoc = UIDocumentInteractionController(url: PdfUrl)
controladorDoc.delegate = self
controladorDoc.presentPreview(animated: true)
}
}

Update
From the Docs they say
The Fetch API provides an interface for fetching resources (including
across the network). It will seem familiar to anyone who has used
XMLHttpRequest
You can also use the following string to get the base64 String from the WKWebview
let s = "path = document.getElementById(\"plugin\").src\n" +
"\n" +
"fetch(path).then(function (response) {\n" +
" response.body.getReader().read().then(function(result) {\n" +
" return btoa(String.fromCharCode.apply(null, result.value));\n" +
" }).then(function(b64) {\n" +
" window.webkit.messageHandlers.myInterface.postMessage(b64);\n" +
" });\n" +
"});"
both fetch and xmlhttp works async.. all you need to do is wait when the processing gets completed pass it to the Swift using javascript's bridge to ios (WKScriptMessageHandler)
Use the following code to get the base64 string from javascript to Swift.
I am using WKScriptMessageHandler to get the callback from Javascript when the base64 string is ready to be consumed. In the String s you just need to pass the url of the pdf and it will do a ajax request to get the pdf file and then convert it to base64 string.
import UIKit
import WebKit
class ViewController: UIViewController {
#IBOutlet weak var btnPDF: UIButton!
#IBOutlet weak var webViewParentView: UIView!
var activityIndicator: UIActivityIndicatorView?
var webView: WKWebView!
#objc func didSelect(_ sender: UIView){
let s="var xhr = new XMLHttpRequest();\n" +
"xhr.open(\'GET\', \"https://codingexceptions.com/wkwebview/dummy.pdf\", true);\n" +
"\n" +
"xhr.responseType = \'arraybuffer\';\n" +
"\n" +
"xhr.onload = function(e) {\n" +
" if (this.status == 200) {\n" +
" var uInt8Array = new Uint8Array(this.response);\n" +
" var i = uInt8Array.length;\n" +
" var binaryString = new Array(i);\n" +
" while (i--)\n" +
" {\n" +
" binaryString[i] = String.fromCharCode(uInt8Array[i]);\n" +
" }\n" +
" var data = binaryString.join(\'\');\n" +
"\n" +
" var base64 = window.btoa(data);\n" +
"\n" +
"window.webkit.messageHandlers.myInterface.postMessage(base64);" +
"\n" +
" }\n" +
"};\n" +
"\n" +
"xhr.send();\n"
webView.configuration.userContentController.add(self, name: "myInterface")
webView?.evaluateJavaScript(s, completionHandler: {(string,error) in
print(error ?? "no error")
})
}
func setupWebView(){
webView = WKWebView.init(frame: CGRect(x: 0, y: 0, width: webViewParentView.frame.width, height: webViewParentView.frame.height))
webView.navigationDelegate = self
webViewParentView.addSubview(webView)
activityIndicator = UIActivityIndicatorView(activityIndicatorStyle: .gray)
activityIndicator?.center = self.view.center
self.view.addSubview(activityIndicator!)
webView.load(URLRequest(url: URL(string: "https://codingexceptions.com/wkwebview/index.php")!))
activityIndicator?.startAnimating()
}
override func viewDidLoad() {
super.viewDidLoad()
btnPDF.addTarget(self, action: #selector(self.didSelect(_:)), for: .touchUpInside)
}
override func viewDidAppear(_ animated: Bool) {
super.viewDidAppear(animated)
setupWebView()
}
}
extension ViewController: WKScriptMessageHandler{
func userContentController(_ userContentController: WKUserContentController, didReceive message: WKScriptMessage) {
print("Message received: \(message.name) with body: \(message.body)")
}
}
extension ViewController: WKNavigationDelegate{
func webView(_ webView: WKWebView, didFinish navigation: WKNavigation!) {
self.activityIndicator?.stopAnimating()
self.activityIndicator?.removeFromSuperview()
self.activityIndicator = nil
}
}
Update: To get the source from the embed tag as pointed in #Tarun's answer
just put the below line in the starting of string variable s and pass the url in xhr.open
var url = document.getElementById("plugin").src

PS: Using answer as comments, as I need formatting
You should execute the below JavaScript in the webview
path = document.getElementById("plugin").src
fetch(path).then(function (response) {
response.body.getReader().read().then(function(result) {
return btoa(String.fromCharCode.apply(null, result.value));
}).then(function(b64) {
window.pdf_data = b64;
});
});
Then you can execute another query to access the window.pdf_data assuming get return value from a javascript execution is possible?

Are you looking to download the PDF to your iPhone or to your Mac? Generally, it's not possible to download PDFs directly to your iPhone because the iPhone doesn't natively have the capability to store PDFs. You'd need to have iBooks or iCloud Drive and then open the PDF in another window, followed by manually downloading it. There would still need to be user-interaction before you download the PDF, meaning the user would have to approve the download. Direct download via the injection of JavaScript into a WKWebView instance is is not possible.

XPathEvaluator in Firefox addon

I am attempting to follow this article to evaluate an XPath expression. My code is copy/pasted from the article:
// Evaluate an XPath expression aExpression against a given DOM node
// or Document object (aNode), returning the results as an array
// thanks wanderingstan at morethanwarm dot mail dot com for the
// initial work.
function evaluateXPath(aNode, aExpr) {
var xpe = new XPathEvaluator();
var nsResolver = xpe.createNSResolver(aNode.ownerDocument == null ?
aNode.documentElement : aNode.ownerDocument.documentElement);
var result = xpe.evaluate(aExpr, aNode, nsResolver, 0, null);
var found = [];
var res;
while (res = result.iterateNext())
found.push(res);
return found;
}
However, I'm getting this error:
Message: ReferenceError: XPathEvaluator is not defined
Is Mozilla's article out of date, perhaps? Is there a more up-to-date article available on parsing XML in an SDK add-on?
Edit. When I tried it this way:
var {Cc, Ci} = require("chrome");
var domXPathEvaluator = Cc["#mozilla.org/dom/xpath-evaluator;1"].createInstance(Ci.nsIDOMXPathEvaluator);
I got a long error message:
- message = Component returned failure code: 0x80570019 (NS_ERROR_XPC_CANT_CREATE_WN) [nsIJSCID.createInstance]
- fileName = undefined
- lineNumber = 14
- stack = #undefined:14:undefined|#resource://helloworld-addon/index.js:14:25|run#resource://gre/modules/commonjs/sdk/addon/runner.js:145:19|startup/</<#resource://gre/modules/commonjs/sdk/addon/runner.js:86:7|Handler.prototype.process#resource://gre/modules/Promise-backend.js:920:23|this.PromiseWalker.walkerLoop#resource://gre/modules/Promise-backend.js:799:7|this.PromiseWalker.scheduleWalkerLoop/<#resource://gre/modules/Promise-backend.js:738:39|Promise*this.PromiseWalker.scheduleWalkerLoop#resource://gre/modules/Promise-backend.js:738:7|this.PromiseWalker.schedulePromise#resource://gre/modules/Promise-backend.js:762:7|this.PromiseWalker.completePromise#resource://gre/modules/Promise-backend.js:705:7|handler#resource://gre/modules/commonjs/sdk/addon/window.js:56:3|
- toString = function () /* use strict */ toString
edit 2. Here, I'll just post my whole code, because it's clear something stranger than I thought is going on. I've created a hello-world addon using the Mozilla tutorials including this one to display a popup. I've modified that further so that it will append text to a file, and modified that further to, I hope, parse and modify XML. So the resulting add-on is supposed to take text entered in the popup and append it to an XML file.
var data = require("sdk/self").data;
var text_entry = require("sdk/panel").Panel({
contentURL: data.url("text-entry.html"),
contentScriptFile: data.url("get-text.js")
});
const fooFile = "/Users/sabrina/Documents/addon/foo.xml";
var {Cc, Ci} = require("chrome");
var parser = Cc["#mozilla.org/xmlextras/domparser;1"].createInstance(Ci.nsIDOMParser);
//var domXPathEvaluator = Cc["#mozilla.org/dom/xpath-evaluator;1"].createInstance(Ci.nsIDOMXPathEvaluator);
var foo = parser.parseFromString(readTextFromFile(fooFile), "application/xml");
// Create a button
require("sdk/ui/button/action").ActionButton({
id: "show-panel",
label: "Show Panel",
icon: {
"16": "./icon-16.png",
"32": "./icon-32.png",
"64": "./icon-64.png"
},
onClick: handleClick
});
// Show the panel when the user clicks the button.
function handleClick(state) {
text_entry.show();
}
text_entry.on("show", function() {
text_entry.port.emit("show");
});
text_entry.port.on("text-entered", function (text) {
console.log(text);
// appendTextToFile(text, "/Users/sabrina/Documents/addon/output.txt");
appendFoo(text);
text_entry.hide();
});
function appendFoo(text) {
var newNode = foo.createElement("blah");
newNode.innerHTML = text;
var mainFoo = evaluateXPath(foo, '/foo')[0];
mainFoo.appendChild(newNode);
foo.save(fooFile);
}
function evaluateXPath(aNode, aExpr) {
var xpe = new XPathEvaluator();
var nsResolver = xpe.createNSResolver(aNode.ownerDocument == null ?
aNode.documentElement : aNode.ownerDocument.documentElement);
//var result = domXPathEvaluator.evaluate(aExpr, aNode, null,
// domXPathResult.ORDERED_NODE_ITERATOR_TYPE, null);
var found = [];
var res;
while (res = result.iterateNext())
found.push(res);
return found;
}
function readTextFromFile(filename) {
var fileIO = require("sdk/io/file");
var text = null;
if (fileIO.exists(filename)) {
var TextReader = fileIO.open(filename, "r");
if (!TextReader.closed) {
text = TextReader.read();
TextReader.close();
}
}
console.log(arguments.callee.name + ": have read " + text + " from " + filename);
return text;
}
function writeTextToFile(text, filename) {
var fileIO = require("sdk/io/file");
var TextWriter = fileIO.open(filename, "w");
if (!TextWriter.closed) {
TextWriter.write(text + "\n");
console.log(arguments.callee.name + ": have written " + text + " to " + filename);
TextWriter.close();
}
function appendTextToFile(text, filename) {
var textplus = readTextFromFile(filename) + text;
writeTextToFile(textplus, filename);
}
I run at the command line using jpm run which opens Firefox Developer Edition. I click the addon button, the popup comes up, I enter text, I hit return, and I see this in the console:
JPM undefined Starting jpm run on Sabrina's Helloworld Addon
Creating XPI
JPM undefined XPI created at /var/folders/gg/r_hp4hzs0gdfy70f__l18fmr0000gn/T/#helloworld-addon-0.0.1.xpi (46ms)
Created XPI at /var/folders/gg/r_hp4hzs0gdfy70f__l18fmr0000gn/T/#helloworld-addon-0.0.1.xpi
JPM undefined Creating a new profile
console.log: helloworld-addon: readTextFromFile: have read <?xml version="1.0" encoding="UTF-8"?>
<foo><blah>eek</blah><foo>
from /Users/sabrina/Documents/addon/foo.xml
console.log: helloworld-addon: ook
console.error: helloworld-addon:
JPM undefined Message: ReferenceError: XPathEvaluator is not defined
Stack:
evaluateXPath#resource://gre/modules/commonjs/toolkit/loader.js -> resource://helloworld-addon/index.js:63:9
appendFoo#resource://gre/modules/commonjs/toolkit/loader.js -> resource://helloworld-addon/index.js:57:19
#resource://gre/modules/commonjs/toolkit/loader.js -> resource://helloworld-addon/index.js:50:2
emitOnObject#resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modules/commonjs/sdk/event/core.js:112:9
emit#resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modules/commonjs/sdk/event/core.js:89:38
portEmit#resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modules/commonjs/sdk/content/sandbox.js:343:7
emitOnObject#resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modules/commonjs/sdk/event/core.js:112:9
emit#resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modules/commonjs/sdk/event/core.js:89:38
onContentEvent/<#resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modules/commonjs/sdk/content/sandbox.js:384:5
delay/<#resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modules/commonjs/sdk/lang/functional/concurrent.js:38:20
notify#resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modules/commonjs/sdk/timers.js:40:9

Non-authoritative, speculative answer
In a different question, Wladimir Palant (author of Adblock Plus, presumably he has good knowledge of firefox) said:
Yes, a lot of global classes available in the window context aren't there in SDK modules which are sandboxes.
Source: https://stackoverflow.com/a/10522459/3512867
This could explain why XPathEvaluator is not defined in the SDK addon.
The logical conclusion would be to use Firefox's Components object to access the nsIDOMXPathEvaluator interface. Which brings up the following error:
NS_ERROR_XPC_CANT_CREATE_WN
Looking into it takes us to this, from mozillazine's forums user "lithopsian":
That means it can't create a wrapper for a non-javascript interface.
Source: http://forums.mozillazine.org/viewtopic.php?f=19&t=2854793
I am unable to judge the credibility of that statement and while the linked bug reports seem to be relevant, I can not attest they actually are:
https://bugzilla.mozilla.org/show_bug.cgi?id=994964
https://bugzilla.mozilla.org/show_bug.cgi?id=1027095
https://bugzilla.mozilla.org/show_bug.cgi?id=1029104
Unless those informations are confirmed (or dispelled) by people with a deeper knowledge of Firefox's internal workings, I can only hesitantly conclude that the nsIDOMXPathEvaluator interface can simply not work in an SDK addon.

Sending all Javascript console output into a DOM element

How does one send all console output into a DOM element so it can be viewed without having to open any developer tools? I'd like to see all output, such as JS errors, console.log() output, etc.

I found the accepted answer above helpful but it does have a couple issues as indicated in the comments:
1) doesn't work in Chrome because "former" does not take into account the this context no long being the console, the fix is to use the JavaScript apply method.
2) It does not account for multiple arguments being passed to console.log
I also wanted this to work without jQuery.
var baseLogFunction = console.log;
console.log = function(){
baseLogFunction.apply(console, arguments);
var args = Array.prototype.slice.call(arguments);
for(var i=0;i<args.length;i++){
var node = createLogNode(args[i]);
document.querySelector("#mylog").appendChild(node);
}
}
function createLogNode(message){
var node = document.createElement("div");
var textNode = document.createTextNode(message);
node.appendChild(textNode);
return node;
}
window.onerror = function(message, url, linenumber) {
console.log("JavaScript error: " + message + " on line " +
linenumber + " for " + url);
}
Here is an updated working example with those changes.
http://jsfiddle.net/eca7gcLz/

This is one approach for a quick solution:
Javascript
var former = console.log;
console.log = function(msg){
former(msg); //maintains existing logging via the console.
$("#mylog").append("<div>" + msg + "</div>");
}
window.onerror = function(message, url, linenumber) {
console.log("JavaScript error: " + message + " on line " +
linenumber + " for " + url);
}
HTML
<div id="mylog"></div>
Working Example http://jsfiddle.net/pUaYn/2/

Simple console.log redefinition, without error handling:
const originalConsoleLog = console.log
console.log = (...args) => {
args.map(arg => document.querySelector("#mylog").innerHTML += arg + '<br>')
}
console.log = originalConsoleLog

Server Side Logging Of Client Side Javascript Crashes

I have a large complex web app with thousands of lines of Javascript. There is a small set of intermittent Javascript bugs that are report by users.
I think these are epiphenomena of race conditions - something has not initialised correctly and the Javascript crashes causing 'down stream' js not to run.
Is there anyway to get Javascript execution crashes to log back server side?
All the js logging libraries like Blackbird and Log4JavaScript are client-side only.

I have written a remote error logging function using window.onerror as suggested by #pimvdb
Err = {};
Err.Remoterr = {};
Err.Remoterr.onerror = function (msg, errorfileurl, lineno) {
var jsonstring, response, pageurl, cookies;
// Get some user input
response = prompt("There has been an error. " +
"It has been logged and will be investigated.",
"Put in comments (and e-mail or phone number for" +
" response.)");
// get some context of where and how the error occured
// to make debugging easier
pageurl = window.location.href;
cookies = document.cookie;
// Make the json message we are going to post
// Could use JSON.stringify() here if you are sure that
// JSON will have run when the error occurs
// http://www.JSON.org/js.html
jsonstring = "{\"set\": {\"jserr\": " +
"{\"msg\": \"" + msg + "\", " +
"\"errorfileurl\": \"" + errorfileurl + "\", " +
"\"pageurl\": \"" + pageurl + "\", " +
"\"cookies\": \"" + cookies + "\", " +
"\"lineno\": \"" + lineno + "\", " +
"\"response\": \"" + response + "\"}}}";
// Use the jquery cross-browser post
// http://api.jquery.com/jQuery.post/
// this assumes that no errors happen before jquery has initialised
$.post("?jserr", jsonstring, null, "json");
// I don't want the page to 'pretend' to work
// so I am going to return 'false' here
// Returning 'true' will clear the error in the browser
return false;
};
window.onerror = Err.Remoterr.onerror;
I deploy this between the head and body tags of the webpage.
You will want to change the JSON and the URL that you post it to depending on how you are going to log the data server side.

Take a look at https://log4sure.com (disclosure: I created it) - but it is really useful, check it out and decide for yourself. It allows you to log errors/event and also lets you create your custom log table. It also allows you to monitor your logs real-time. And the best part, its free.
You can also use bower to install it, use bower install log4sure
The set up code is really easy too:
// setup
var _logServer;
(function() {
var ls = document.createElement('script');
ls.type = 'text/javascript';
ls.async = true;
ls.src = 'https://log4sure.com/ScriptsExt/log4sure.min.js';
var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(ls, s);
ls.onload = function() {
// use your token here.
_logServer = new LogServer("use-your-token-here");
};
})();
// example for logging text
_logServer.logText("your log message goes here.")
//example for logging error
divide = function(numerator, divisor) {
try {
if (parseFloat(value) && parseFloat(divisor)) {
throw new TypeError("Invalid input", "myfile.js", 12, {
value: value,
divisor: divisor
});
} else {
if (divisor == 0) {
throw new RangeError("Divide by 0", "myfile.js", 15, {
value: value,
divisor: divisor
});
}
}
} catch (e) {
_logServer.logError(e.name, e.message, e.stack);
}
}
// another use of logError in window.onerror
// must be careful with window.onerror as you might be overwriting some one else's window.onerror functionality
// also someone else can overwrite window.onerror.
window.onerror = function(msg, url, line, column, err) {
// may want to check if url belongs to your javascript file
var data = {
url: url,
line: line,
column: column,
}
_logServer.logError(err.name, err.message, err.stack, data);
};
// example for custom logs
var foo = "some variable value";
var bar = "another variable value";
var flag = "false";
var temp = "yet another variable value";
_logServer.log(foo, bar, flag, temp);