Develop Reference

How to get the contents of the body tag of the page through the browser extension?

Why is variable body empty? how to get body tag content?
I am writing code for google chrome extension. I plan to use the
extension for personal use only. For web scraping.
To finally analyze the text. I want to play with text.
background.js
chrome.tabs.onUpdated.addListener(function(tabId, changeInfo, tab) {
if(document.readyState == "complete"){
var url = changeInfo.url;
// I want to save the url to a file(url.txt)
var body = document.body.innerText;
// why is body empty?????
var pattern = /[A-Z].*?\./g;
var result = text.match(pattern);
result.forEach(myFunction);
function myFunction(item) {
text += item+"\n";
}
// I want to save the text to a file(collection.txt)
}
});
manifest.json
{
"name": "Parser",
"version": "1",
"manifest_version": 2,
"background": {
"scripts":["background.js"]
},
"permissions": [
"tabs",
"activeTab",
"storage",
"http://*/*",
"https://*/*"
]
}

Based on the manifest.json you posted it looks like you are running your code as a background script.
Background scripts don't have direct access to the loaded page content - this is why the body is empty in your code.
Instead, you will need to use a content script to access the page content and then message that data across to your background script for processing.
Here is an example setup using a background and content script that should let you retrieve and process the page content when a tab loads (not tested, but should point you in the right direction).
Thanks to the ResourceOverride extension which I used as a reference for writing the example below.
background.js
// background.js
chrome.runtime.onMessage.addListener(function(message, sender){
if (!message || typeof message !== 'object' || !sender.tab){
// Ignore messages that weren't sent by our content script.
return;
}
switch (message.action){
case 'receiveBodyText': {
processBodyText(sender.tab, message.bodyText);
break;
}
}
});
function processBodyText(tab, bodyText){
var url = tab.url;
// I want to save the url to a file(url.txt)
// TODO: Process your bodyText
var pattern = /[A-Z].*?\./g;
var result = text.match(pattern);
result.forEach(myFunction);
function myFunction(item) {
text += item+"\n";
}
// I want to save the text to a file(collection.txt)
}
content.js
// content.js
window.addEventListener('load', function(){
let bodyText = document.body.innerText;
chrome.runtime.sendMessage({
action: 'receiveBodyText',
bodyText: bodyText
});
});
manifest.json
// manifest.json
{
"name": "Parser",
"version": "1",
"manifest_version": 2,
"background": {
"scripts":["background.js"]
},
"content_scripts": [{
"matches" : [
"http://*/*",
"https://*/*"
],
"js": ["content.js"]
}],
"permissions": [
"tabs",
"activeTab",
"storage",
"http://*/*",
"https://*/*"
]
}
Information and docs
A note on WebExtension API differences between firefox and chrome:
Chrome uses the chrome namespace, Firefox uses the future-standard browser namespace.
So code written in Chrome would use: chrome.tabs.onUpdated(...)
and the equivalent in Firefox would be: browser.tabs.onUpdated(...)
Be aware of that when reading the docs and reading example extensions.
Background scripts
do not have access to the loaded page
have full access to the WebExtensions API
Content scripts
have full access to the loaded page
have only limited access to the WebExtensions API
Chrome content scripts docs
MDN content scripts docs
Docs on communication between content scripts and background scripts
WebExtensions API
Chrome WebExtensions API reference
MDN WebExtensions API reference
other useful links
MDN WebExtensions example Github repository
MDN "Anatomy of a WebExtension"
MDN detailed browser WebExtensions support tables
ResourceOverride extension - this is a fairly complex extension that uses both background and content scripts. I this used as a reference/example to better understand how extensions are written.

Porting WebExtension from Chrome to Firefox?

I made a working Chrome extension that is not packaged and is just a directory on my computer. I found out that I should be able to port this to Firefox rather easily.
I followed the "Porting a Google Chrome extension" guide on MDN and found that my manifest file is perfect.
I then followed the instructions on how to perform "Temporary Installation in Firefox" of the extension.
However, when I click on any file inside the directory, nothing happens. The extension doesn't load. Any advice? I know the extension works in Chrome fine and loads without error.
manifest.json:
{
"manifest_version": 2,
"name": "ER",
"description": "P",
"version": "1.0",
"browser_action": {
"default_icon": "icon.png",
"default_popup": "popup.html"
},
"background": {
"scripts": [ "background.js" ]
},
"content_scripts": [
{
"matches": [ "SiteIwant" ],
"js": [ "ChromeFormFill.js" ],
"run_at": "document_idle"
}
],
"permissions": [
"*://*/*",
"cookies",
"activeTab",
"tabs",
"https://ajax.googleapis.com/"
],
"externally_connectable": {
"matches": ["SiteIwant"]
}
}
ChromeFormFill.js:
// JavaScript source c
console.log("inside content");
console.log(chrome.runtime.id);
document.getElementById("ID").value = chrome.runtime.id.toString();
Background.js
chrome.runtime.onMessage.addListener(
function (request, sender, sendResponse) {
if (request.data === "info") {
console.log("Recieving Info");
return true;
}
});
chrome.tabs.create(
{
url: 'myUrl'
active: true
}, function (tab) {
chrome.tabs.executeScript(tab.id, { file: 'Run.js', runAt: "document_idle" });
});
Run.js will just alert('hi').
It just won't do anything when I try to load it on Firefox; nothing will happen.

Issues:
In manifest.json:
externally_connectable is not supported:1
Firefox does not support externally_connectable. You can follow bug 1319168 for more information. There is, currently, no expected time when this will be implemented.
You will need to communicate between the code on your site and the WebExtension using a different method. The way to do so is to inject a content script and communicate between the site's code and the content script. The common ways to do this are CustomEvent() or window.postMessage(). My preference is CustomEvent().
Using window.postMessage() is like yelling your message outside and hoping that either nobody else is listening, or that they know that they should ignore the message. Other people's code that is also using window.postMessage() must have been written to ignore your messages. You have to write your code to ignore any potential messages from other code. If either of those were not done, then your code or the other code can malfunction.
Using CustomEvent() is like talking directly to someone in a room. Other people could be listening, but they need to know about the room's existence in order to do so, and specifically choose to be listening to your conversation. Custom events are only received by code that is listening for the event type which you have specified, which could be any valid identifier you choose. This makes it much less likely that interference will happen by mistake. You can also choose to use multiple different event types to mean different things, or just use one event type and have a defined format for your messages that allows discriminating between any possible types of messages you need.
matches value needs to be valid (assumed to be intentionally redacted):
You have two lines (one with a trailing ,, one without; both syntactically correct):
"matches": ["SiteIwant"]
"SiteIwant" needs to be a valid match pattern. I'm assuming that this was changed away from something valid to obfuscate the site that you are working with. I used:
"matches": [ "*://*.mozilla.org/*" ]
In Background.js:
The lines:
url: 'myUrl'
active: true
need to be:
url: 'myUrl',
active: true
[Note the , after 'myUrl'.] In addition, myUrl needs to be a valid URL. I used:
url: 'http://www.mozilla.org/',
A Firefox 48 bug (now long fixed):
Your line:
chrome.tabs.executeScript(tab.id, { file: 'Run.js', runAt: "document_idle" });
In Firefox 48 this line is executed prior to the tab being available. This is a bug. It is fixed in Firefox Developer Edition and Nightly. You will need one of those to test/continue development.
Issues in ChromeFormFill.js:
Another Firefox 48 bug (now long fixed):
chrome.runtime.id is undefined. This is fixed in Developer Edition and Nightly.
Potential content script issue:
I'm going to assume your HTML has an element with an ID = 'ID'. If not, your document.getElementById("ID") will be null. You don't check to see if the returned value is valid.
Running your example code
Once all those errors were resolved, and running under Firefox Nightly, or Developer Edition, it worked fine. However, you didn't have anything that relied on being externally_connectable, which won't function.
agaggi noticed that I had forgotten to include this issue in the original version of my answer.

Access DOM elements data document->Iframe ->Iframe->method chrome extension

I'm trying to access some data from an iframe nested within an iframe, from developers console:
Object.keys(document.getElementById("contentBody").
contentDocument.getElementById('rawContent').
contentDocument.defaultView.window.messages)
["29c736c0ed25463c8436f4990ab6c6ec.zip",
"235819a8cf11488e83f0336603b71711.zip",
"66c9260590834d9698568c8a676ef406.zip",
"fae95e31cb424cd6ad21302217ef2cdc.zip",
"f554f712141047aa9aa24f765073e305.zip",
"e5c41819578240e0868f43ab6301aeb3.zip"]
That's what I expect back, but I've tried to get that very same info from a google chrome extension that I'm developing and for some reason I cant access messages array, this is the manifest file and contentscript.js (I've tried everything that came to my mind and searching for a few hours without success :/):
content.js
var iframeContentBody = document.getElementById('contentBody');
var innerDocContentBody = iframeContentBody.contentDocument;
var iframeRawContent = innerDocContentBody.getElementById('rawContent');
var innerDocRawContent = iframeRawContent.contentDocument; // iframeRawContent is undefined here
console.log(iframeRawContent.messages); // this prints undefined
manifest:
{
"manifest_version": 2,
"name": "Read Comments",
"description": "Read all comments from the current forum",
"version": "1.0",
"content_scripts": [{
"matches": ["*://*.forum.net/*"],
"js": ["content.js"]
}],
"browser_action": {
"default_title": "Read Comments"
},
"permissions": ["activeTab", "tabs"]
}
Gists to set everything up:
HTML Example
after downloading and placing these 3 files in the same folder, run this:
python -m SimpleHTTPServer 80 # You may need to run it with sudo
then go to localhost/test.html and you're all set, if you test the line that I posted in the console you should see [1,2,3]
Extension example
this is the extension code
Developers console:
Chrome extension with "all_frames": true
Hacky solution: Partial solution
In this gist there is a way to do it, it's hard to detect when the iframe has been loaded, and it's harded to detect when the iframe inside the another iframe has been loaded, so a setTimeout gives enough time to get it done, then adding a script element to the dom seems to bypass all security measures that chrome extensions may have and it does get the content of the attribute without any other issue, still this seems hacky and it's not what I'm trying to do, I'm looking for a clean solution or a clean way to access the dom of a nested iframe as the example code states...
Thanks, any suggestion is welcome.

This was my solution after all, between what we talk over comments and my research over docs and other threads:
Content script:
(function () {
document.addEventListener("DOMContentLoaded", function () {
contentBody = document.getElementById("contentBody");
contentBody.addEventListener("load", function () {
rawContent = contentBody.contentDocument.getElementById("rawContent");
if (rawContent) {
var s = document.createElement("script");
s.src = chrome.extension.getURL('injected.js');
s.onload = function() {
this.parentNode.removeChild(this);
};
(document.head||document.documentElement).appendChild(s);
}
});
});
})();
Injected file:
keys = Object.keys(document.getElementById("contentBody").contentDocument.getElementById("rawContent").contentDocument.defaultView.window.messages);
console.log(keys);
Manifest:
{
"manifest_version": 2,
"name": "Read Comments",
"description": "Read all comments from the current forum",
"version": "0.0.1",
"content_scripts": [{
"matches": ["*://localhost/*"],
"run_at": "document_start",
"js": ["content.js"]
}],
"browser_action": {
"default_title": "Read Comments"
},
"permissions": [
],
"web_accessible_resources": ["content.js", "injected.js"]
}
As a simple explanation the main issue was the asyc load of iframes and the moment when the extension code ran, so after listening to a lot of events and discarding the ones that doesn't have the required elements on the dom everything went fine...

For completeness, here’s a version with "all_frames":true. There are two problems to work around: (1) getting messages from the inner frame to the top and (2) getting messages from the isolated world of the webpage to the isolated world of the content script (I assume you’re wanting to do more than just write messages to the console). This solves both at once by using postMessage.
if ( window.top !== window.parent ) {
var s = document.createElement("script");
s.textContent = "postMessage(messages,'*');";
s.onload = function() {
this.parentNode.removeChild(this);
};
document.head.appendChild(s);
} else if ( window.top === window ) {
addEventListener('message',function(e) {
console.log(e.data);
});
}
I must confess I’ve not actually tested it out. You may need to try making the injected script send a message from the webpage.

Enable Cross-Origin XMLHttpRequest in devtool

I'm working on a devtool Chrome extension and would like to log all XHR request of the current website. So in my injected script ea "messageback-script" I have this:
(function(open) {
XMLHttpRequest.prototype.open = function(method, url, async) {
console.log('The URL is: ' + url);
};
})(XMLHttpRequest.prototype.open);
And I have given required permissions in the manifest.json:
{
"name": "DevToolsPanel",
"version": "0.1",
"description": "Log XHR requests",
"devtools_page": "devtools.html",
"background": {
"scripts": [
"background.js"
]
},
"permissions": [
"tabs",
"http://*/*",
"https://*/*"
],
"manifest_version": 2
}
The problem is that I don't see anything in the console log (blank). What am I doing wrong?

Having examined the code you are using, I can say that this script is evaluated via chrome.tabs.executeScript, making it a Content Script.
Content Scripts live in an isolated context, and as such modifying the XHR prototype only affects XHR in your content script(s). There's a canonical question describing this problem and workarounds.
However, since you're writing a devtools extension, you have a simpler solution via chrome.devtools.inspectedWindow.eval(), which directly executes code in the page's own context.
// In the devtool panel code
chrome.devtools.inspectedWindow.eval("/* Your code here*/");

Communication between scripts in Chrome Extension

I know there are many variations of this question already in existence here, but none of them seem to work for me.
Details:
I'm writing an extension that pulls some email data from emails you send in gmail. In order to achieve this I am using this version of Gmailr https://github.com/joscha/gmailr.
In effect, I have three content scripts: Gmailr.js and main.js (which are pretty much identical to those in the link above) allow me to pull out the information I'm looking for. Then content.js I use to send a message to the background page of the extension.
The problem is that from gmailr.js and main.js I cannot use any of the Chrome APIs, and I'm not really sure why, so I can't send messages from these back to the background page.
That is why I made content.js which can communicate with the background page. However, it does not seem to be able to see anything the other content scripts do. For example, main.js inserts a div at the top of the page. When I try to attach an event listener to a button in this div from content.js, I am told that no such element exists.
How can I get the data pulled out by main.js to be seen by content.js? (I also tried to put the data in local storage, then trigger a custom event listener to tell content.js to read local storage, but no luck because they don't seem to be able to hear each other's event being triggered).
Any insight or alternatives are much appreciated.
(I can post code if necessary, but it's fragmented and long)
My manifest file:
{
"manifest_version": 2,
"name": "Email extractor",
"description": "Extracts data from emails",
"version": "1.0",
"background": {
"script": "background.js"
},
"content_scripts": [
{
"matches": [
"*://mail.google.com/*",
"*://*/*"
],
"js": [
"lib/yepnope.js/yepnope.1.5.4-min.js",
"lib/bootstrap.js",
"main.js",
"gmailr.js",
"content.js"
],
"css": [
"main.css"
],
"run_at": "document_end"
}
],
"permissions": [
"tabs",
"storage",
"background",
"*://mail.google.com/*",
"*://*/*"
],
"browser_action": {
"default_icon": "img/icon.png",
"default_popup": "popup.html"
},
"web_accessible_resources" : [
"writeForm.js",
"disp.js",
"/calendar/jsDatePick.min.1.3.js",
"/calendar/jsDatePick_ltr.min.css",
"lib/gmailr.js",
"lib/jquery-bbq/jquery.ba-bbq.min.js",
"content.js",
"main.js",
"background.js"
]
}
This is main.js:
Gmailr.init(function(G) {
sender = G.emailAddress();
G.insertTop($("<div id='gmailr'><span></span> <span id='status'></span>)");
el = document.getElementById("testid");
el.addEventListener('click', mg, false);
var status = function(msg) {
G.$('#gmailr #status').html(msg); };
G.observe(Gmailr.EVENT_COMPOSE, function(details) {
....
status(" user: " + user);
console.log('user:', user);
//now try to send a message to the background page
//this always returns the error that method sendMessage does not exist for undefined
chrome.runtime.sendMessage({greeting: "test from gmailr"}, function(response) {
console.log("did it send?");
});
});
});
gmailr.js is quite long and is also not my own code but it can be seen here: http://pastebin.com/pK4EG9vh

Hi perhaps 3 likely reason to your problem :
The way you send messages to bgp from main.js and gmailr.js are perhaps wrong because you must arrive to communicate from any content script to your bgp. (in your manifest content script key the gmailr.js is missing). Show us your code it would help.
You seems to have a problem with the moment you search from content.js to access to the element created in main.js. Do you try to access your element with the jQuery $("").on() method ? A simple test must be to declare a function in one cs and to use it in another. If it's not working it's a manifest problem. The order you declare .js file in manifest content script key is important also.
try to in the manifest content script array "run_at":"document_end"
Hope it help !