Develop Reference

Grabbing partial data from the clipboard in javascript

I have copied an Excel table which is about a million rows. When I look at the clipboard on my system, it seems to contain about 250MB of data. However, I only need to grab the styling information from it, for example:
This entire data comes out to (far) less than 1MB of data. Is there a way to read the clipboard as if it were a file or stream, so that I can just do, for example:
clipboard.read(1024)
Otherwise, if I do the straight:
evt.clipboardData.getData('text/html')
And grab the section of data I want after getting the data, it takes me over 10 seconds to do! Whereas I believe the event should only take 0.1s or so, if I'm able to read the clipboard data in a partial manner, as if it were a file.
What can I do here? Is it possible to use FileReader on the clipboard? If so, how could that be done?

The Clipboard.read API cited in the comments can return clipboard contents as a list of ClipboardItem objects, from which you can then obtain a Blob object, which you can then .slice to perform partial reads. (The MDN claims that Clipboard.read returns a DataTransfer object, but this disagrees with the specification, so I assume this is stale information, or simply in error.)
const perm = await navigator.permissions.query({ name: 'clipboard-read' });
switch (perm.state) {
case 'granted':
case 'prompt':
break;
default:
throw new Error("clipboard-read permission not granted");
}
const items = await navigator.clipboard.read();
for (const item of items) {
const blob = await item.getType('text/html');
const first1M = await blob.slice(0, 1048576).arrayBuffer();
/* process first1M */
}
However, the Clipboard API is nowhere near universally available as of yet. Firefox ESR 78.9 doesn’t implement it, and by the state of MDN it hardly seems to be on Mozilla’s radar at all. (I haven’t tried other browsers yet; perhaps in Chrome it’s already usable.)

After a lot a research, this is not possible in Javascript, there is no support for stream manipulation using the clipboard object so u have to read the entire content at once.
However, u can use MacOS (inferred from your picture) native tools for processing the clipboard data: pbcopy and pbpaste, and they are extremely fast, orders of magnitude faster than Javascript, so u can delegate the heavy processing of the text to them.
So, after u copy the 250MB of text, you can slice it and read only the first n bytes (in this case 1024) and substitute the content of the clipboard with that, so now u it will be available for u to use it in Javascript:
pbpaste | cut -b 1-1024 | pbcopy
If u need any documentation about each terminal command, u can run man command_name. Extracting the first 1024 bytes of the clipboard took less than a second with this approach.
I tested it with a sample text file of 390MB created with python with this script:
c = 30000000
with open('sample.txt', 'w') as file:
file.writelines('a sample code' for i in range(c))

Accessing a local file using HTML/javascript

I am trying to access local files. The method works with Firefox (and was surprised Edge) but not Chrome.
The files in question are 2 html files each containing a huge tables that are used as databases. The tables are basic tables (table, tbody for each group, tr's, and td's with data).
The method I am using is to load the html files into 2 hidden iframes then accessing the tables inside - html file 1 is a master spell list and html file 2 is similar file for a pencil and paper RPG. Works beautifully in Firefox - tables are read into memory, selects/options are all loaded up, popups and page modifications (showing results of what you selected, memory versions of tables modified as needed, generated customized function working - if this file exists at loadup it automatically updates the memory versions of the tables, if the tables are modified - user is shown the function and can copy/save by using a text editor to local file system). Again beautifully.
But Chrome is a different matter. I can load the files in the iframes, but can't access the tables within. It throws an error about cross server access even though all files are in the same directory (the master html file, functions.js file, 2 table files, and if generated and saved by user the customization.js).
So my question is: is there a way to load/import/access a second or third html file in the main html that will work in FF, Chrome, Edge, and most other modern browsers without changing any security settings?
I would love something as simple as how js and the iframe files can be loaded () and accessable. Can xmlrequest work on local files (I could load and render the tables)?
I would like to share the files with the other players, but can't assume browser choices, security settings, and some may not be technically minded enough to make or want said changes.
PS: I am not looking to write any files back to file system, user is the only one with those options.
OK, the other methods (using new tag attributes) failed so looking into a way to hijack the tag and use JSON.
Another user here posted this code (I have cleaned it up - easier to read - and added the suggested but not included part of the code - adding/initializing rowIx and its incrementer)
function getTable() {
var jsonArr = [];
var obj = {};
var jsonObj = {};
var rowIx = 0
//this assumes only one table in the whole page, and table has column headers
var thNum = document.getElementsByTagName('th').length;
var arrLength = document.getElementsByTagName('td').length;
for(i = 0; i < arrLength; i++){
if(i%thNum === 0){ obj = {}; }
var head = document.getElementsByTagName('th')[i%thNum].innerHTML;
var content = document.getElementsByTagName('td')[i].innerHTML;
obj[head] = content;
if(i%thNum === 0){
jsonObj[rowIx++] = obj;
}
}
return JSON.stringify({"Values": jsonObj})
}
the caller then displays (in a P tag using .innerText since .innerHTML tries to render the data; there are p and br tags in some of the table cells) the returned value so it can be copy/pasted/saved in a separate .js file.
Testing the JSON.parse function in the original HTML (that contains the table I want to later import elsewhere) works just fine, although not like the original: array.Values[x].property vs array.rows[x].cells[y].innerHTML but I can work with that.
format:
{"Values":{"numeric index":{7 key/value pairings},{pattern repeated 122 more times}}}
But when the data is placed in a separate js file, it won't parse back to the original data (error is found when developer options/web console is activated, see below).
source HTML file (has the table database, generates the JSON data for copy/paste/save)
large Table (style="display:none;" which hides it, 123 rows by 7 cells each)
the above function getTable
var test1 = getTable()
update p tag using .innerText for copying with test1 data
var schematics = JSON.parse(test1)
alert(schematics.Values[0].Name)
(all of that works)
js File contents (schematics.json.js)
var schematics = JSON.parse( copy/pasted data goes here );
html file
<script language="javascript" src="schematics.json.js"></script>
<script language="javascript">
alert(schematics.Values[0].name); //data restored test
function rebuildTable(){
//use schematics data to rebuild hidden table
)
</script>
<script language="javascript" src="_functions.js"></script>
all other code is in the last script tag
Web Console, reported error
unexpected character at line 1 column 2 of the JSON data
So what am I doing wrong with the JSON containing js file or secondary HTML page?

This is a difference in the security models and choices of Firefox (and Edge) vs the more strict Chrome. You could argue for the utility vs security of either approach.
To make this work with Chrome the way the other two browsers do, you'll need to disable that security measure with a command line flag when you start Chrome:
> chrome --allow-file-access-from-files
The other alternative is to run a local webserver (e.g. WAMP or XAMPP) and load your files via http://localhost/.

Ok, found a way that works.
In the two webpages that function as my databases, I added code to read the tables into a 2-dimensional arrays (row by cells).
These are then JSON.stringified and "var variableName = " is tacked on the beginning of the returned strings. All this is then added to a p tag (.innerText since there is also HTML code in the JSON data, rendering is not desired).
The presented data is then copied and saved using a plain text editor in a JSON.variableName.js file (the JSON in the name is to remind me what's in the file). Loading it is as easy as loading javascript code using a script tag with src="".
Also, now everything works in Firefox, Edge, and Chrome. I don't have Safari or other browsers. Bonus for me, it works in Android Firefox as well.
The two database webpages can be easily updated and they will generate the new JSON data output.
All in all, there are 6 base files: the main webpage, functions.js, two JSON variable js files, and the 2 database/JSON generator webpages. All local, and no additional webserver needed.

read value from txt file in javascript

I have a simple html file in which there's javascript code referring to google charts.
The code I use is this (I'll show the important part):
function drawChart(){
var data = google.visualization
.arrayToDataTable([ ['Label', 'Value'],['Temp', 22.75],]);
// etc...
}
I use a bash command (sed) to replace that 22.75 value with a new one from the last line of a .txt file. However, this throws some errors which I haven't been able to neither correct nor ever identify.
So is there any javascript code that takes that file, extracts the last value and simply displays it on the right place of the code?
UPDATE:
Sorry for the lack of info in this question, I really appreciate all the people that took the time on reading my question. I'll try to fill with more information in the next minutes.
I am able to extract the last line of the .txt file, extract the value on the right part of the '-' symbol and store it in a variable. Then that value is taken to update the html file with a sed command. The error comes when the value is updated but with no value. I guess that happends due to a failed record of temperature in the txt file, then the extracted value is a null. Finally is the html fiel with javascrit code happens to be like this:
(...)['Temp', ],]);
Then the updater can't update the value since due to the way that sed command is written I guess there's no way that it can detect a no-number-value in there. So the html remains without a value all the time.
TXT File structure:
(...)
20:25:03-23.312
20:26:02-23.312
20:27:03-23.375
20:28:03-23.375
20:29:02-23.375
20:30:02-23.312
Bash script:
# (...code...)
lastRecord=`cat /home/pi/scripts/temp_control/logs/"$today".log | awk 'END{print}'`
function rightNow {
lastTemp=`echo $lastRecord | cut -d'-' -f2`
timeOfTemp=`echo $lastRecord | cut -d'-' -f1` # Not used yet
#Command used to update
sed -i "s/['Temp', [0-9]\{1,2\}.[0-9]\{1,3\}]/$lastTemp]/" /var/www/rightnow.html
}
rightNow

You cud get your file just like any other ajax request.
Using javascript
var request = new XMLHttpRequest();
request.open('GET', 'public_path_to_file.txt', false);
request.send();
var textFileContent = request.responseText
Using jQuery
var textFileContent;
$.get('public_path_to_file.txt', function(data) {
textFileContent = data;
});
Whats left is to get the right part from textFileContent. Dependent of the structure of the file we can do this in different ways. Without an example file you are on your own but here is some examples.
If you need the last line
var lines = textFileContent.split("\n");
var lastLine = lines[lines.length - 1];
If you need to use regex
var regex = //* some regex to get your content*//gm;
var result = regex.exec(textFileContent);
// result should now the content who matches your regex

First I'll assume that you ultimately want to read a local file with your browser and your current workflow is something like a local 'bash-script' that
first updates/modifies an inline piece of javascript (inside a locally stored html
file) with the last occurring value retrieved from a local txt-file (using sed)
opens the (just modified html-) file (via commandline) inside a common browser.
Then I assume the sed-route once worked but now doesn't work anymore (probably because the html file has changed?) and now you'd like the inline javascript (in the html file) to fetch that value from the textfile itself and subsequently use it (thus without the need for the 'bash-script'/sed solution.
Thus, the answer (based on above assumptions) to your final question: 'is there any javascript code that takes that file, extracts the last value and simply displays it on the right place of the code?', depends on your final requirement:
are you ok with a file-input where you select the text-file every time you view the html-file?
If your answer is YES, then, (depending on the browser you use) you can read a local file (and work your magic on it's contents).
In modern browsers, using the File API (which was added to the DOM in HTML5) it's now possible for web content to ask the user to select local files, then read the contents of those files.
For example, using FireFox's 'FileReader' you could do:
html:
<input type="file" id="fileinput" multiple />
javascript:
function readAllFiles(evt){
var files = evt.target.files, i = 0, r, f;
if(files){
for(; f = files[i++]; ){
r = new FileReader();
r.onload = (function(f){
return function(e){
alert(e.target.result);
};
})(f);
r.readAsText(f);
}
} else {
alert("Error loading files");
}
}
document.getElementById('fileinput')
.addEventListener('change', readAllFiles, false);
Note that for accessing local files in Chrome you must start Chrome with this switch: chrome --disable-web-security
However,
if the answer is NO (so you want to specify the file, and more importantly it's path, inside the 'code', so you don't have to select the text-file every time your local app runs) then you (usually) can't (because you can't get/set the path, thank the great maker)...
Unless you choose a specific older/unpatched browser (specifically for this task) where you know of a (hack) way to do this anyway (like the IE xml vulnerability or the XMLHTTP vulnerability or etc... you get the picture..).
Some alternative solutions (that don't require you to select the correct textfile over and over again)
Setup a fullblown web (LAMP) server (to use the XMLHttpRequest way as used in aross answer, but this might feel like shooting at a mosquito with a cannon..)
Explore different script languages (but effectively still do the same as your now broken sed-solution)
Combine 1 and 2, choosing from php (the latest version has a small webserver included, you might start/stop it when needed (even in the bash-script workflow) OR using node.js (which is 'javascript' and where you can program/control a small task-specific server in just a couple of lines).
Hope this helps!
Update:
Based on your updated question, comments and request for recommendation, I'd like to suggest to use PHP to dynamically fetch the value from your log txt file and have it generate your html code with inline javascript on the fly (every time you visit the page).
The browser will never see the php code, only what php inserted to your page (in this example the last found value or 0).
You'd rename the rightnow.html file to rightnow.php and modify it (something like) this:
<!DOCTYPE html>
<html><head>
<!-- your header code -->
<script type="text/javascript">
//parts of your javascript
<?php // start php script
$logFile= '/pathToYour/logFile.log'; // <-Modify
if( $fp= #fopen($logFile, "r") ){ // if logfile succesfully opened,
fseek($fp, -30, SEEK_END); // set pointer 30 chars from EOF
$val= array_pop(explode("-", rtrim(fread($fp, 30)))); //get last value
fclose($fp); // close file
}
?> // end php script
function drawChart(){
var data=google.visualization
.arrayToDataTable([ ['Label', 'Value'],
['Temp', <?php echo $val? $val : "0"; ?>],
]); // ^php above inserts value or 0
// etc...
}
//parts of your javascript
</script>
</head><body>
<!-- your body code -->
</body></html>
Note that fopen in combination with setting the filepointer via fseek and sequentially fread-ing from the pointer to EOF does not load the complete logfile (60min * 24hour=1440 lines of 16 bytes=22.5kB at the end of the day) into memory (good for this purpose), but only the last 30 chars (as in this example).
The variable to your logfile and path must still be modified to your situation (I don't know the format of your $today variable).
Depending on your further needs you might want to perform some extra checks/logic on the array of values that explode returns (instead of popping the last value). Or what about modifying the html a little so you could also include the last temperature's time reading, etc. (But this tested piece of code should get you started and explains the procedure of going the php way).
Update:
Since you have chosen to place the last known value of your logfile as in textfile placed inside your public www-root (with a bash script I assume, every minute of the day?), you can now indeed go the 'ajax' way, as answered by aross!
However I want to hint that the code/solutions in all current answers here could be mixed (since you now also have ajax working): instead of ajax-ing (loading) a txt file, you could have php fetch and send this value to the browser on-the-fly/on-demand!
So, instead of requesting http://url_to_my_rpi/file_to_download.txt, you could request http://url_to_my_rpi/read_last_temperature.PHP which should fetch the last known value out of the log-file (set proper security/access) and send it to the browser (set proper headers), just like your text-file did. You wouldn't have to change anything in the html/javascript except the url you request.
The advantage would be (depending on how your current bash-scripts works) that your PI now only does this 'work' (of getting the last value of your logfile) when you are viewing your monitor-page. And that you are not writing that file in your www-root every minute of every day (as I suspect).

The solution achieved, finally, was like this:
I did it with a jQuery statement and reusing the javascript code of Google Charts.
First I added javascript and jQuery tags in the html file:
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script type='text/javascript' src='https://www.google.com/jsapi'></script>
Then I merged jquery code and javascript code that I had in one script:
<script type='text/javascript'>
// Needed this var so that I could use it in other places of the code
var t;
jQuery.get('http://url_to_my_rpi/file_to_download.txt',function(data){
console.log(data)
t=data;
},'text');
google.load('visualization', '1', {packages:['gauge']});
google.setOnLoadCallback(drawChart);
function drawChart() {
t=eval(t);
var data = google.visualization.arrayToDataTable([
['Label', 'Value'],
['Temp', t],]);
// (... more javascript with Google Charts options, display parameters..)
</script>
Finally, and even if it's not listed as the main question, be sure to enable *mod_headers* on your apache and add Header set to apache2.conf file (In my case: /etc/apache2/apache2.conf)
1) Enable the module:
a2enmod headers
2) Add the line on your configuration file
Header set Access-Control-Allow-Origin "*"
3) Restart apache 2
4) In case the 3 steps above didn't work, follow the instrcutions by entering in this website or reinstall apache2.

Auto Save a file in Firefox

I am trying to find a way where by we can auto save a file in Firefox using JS. The way I have done till yet using FireShot on a Windows Desktop:
var element = content.document.createElement("FireShotDataElement");
element.setAttribute("Entire", EntirePage);
element.setAttribute("Action", Action);
element.setAttribute("Key", Key);
element.setAttribute("BASE64Content", "");
element.setAttribute("Data", Data);
element.setAttribute("Document", content.document);
if (typeof(CapturedFrameId) != "undefined")
element.setAttribute("CapturedFrameId", CapturedFrameId);
content.document.documentElement.appendChild(element);
var evt = content.document.createEvent("Events");
evt.initEvent("capturePageEvt", true, false);
element.dispatchEvent(evt);
But the issue is that it opens a dialog box to confirm the local drive location details. Is there a way I can hard code the local drive storage location and auto save the file?

If you are creating a Firefox add-on then FileUtils and NetUtil.asyncCopy are your friends:
Components.utils.import("resource://gre/modules/FileUtils.jsm");
Components.utils.import("resource://gre/modules/NetUtil.jsm");
var TEST_DATA = "this is a test string";
var source = Components.classes["#mozilla.org/io/string-input-stream;1"].
createInstance(Components.interfaces.nsIStringInputStream);
source.setData(TEST_DATA, TEST_DATA.length);
var file = new FileUtils.File("c:\\foo\\bar.txt");
var sink = file.openSafeFileOutputStream(file, FileUtils.MODE_WRONLY |
FileUtils.MODE_CREATE);
NetUtil.asyncCopy(source, sink);
This will asynchronously write the string this is a test string into the file c:\foo\bar.txt. Note that NetUtil.asyncCopy closes both streams automatically, you don't need to do it. However, you might want to pass a function as third parameter to this method - it will be called when the write operation is finished.
See also: Code snippets, writing to a file

Every computer has a different file structure. But still, there is a way. You can save it to cookie / session, depends on how "permanent" your data wants to be.
Do not consider writing a physical file as it requires extra permission.

Mozilla (Firefox, Thunderbird) Extension: How to get extension id (from install.rdf)?

If you are developing an extension for one of the mozilla applications (e.g. Firefox, Thunderbird, etc.) you define a extension id in the install.rdf.
If for some reason you need to know the extension id e.g. to retrieve the extension dir in local file system (1) or if you want to send it to a webservice (useage statistic) etc. it would be nice to get it from the install.rdf in favour to have it hardcoded in your javascript code.
But how to access the extension id from within my extension?
1) example code:
var extId = "myspecialthunderbirdextid#mydomain.com";
var filename = "install.rdf";
var file = extManager.getInstallLocation(extId).getItemFile(extId, filename);
var fullPathToFile = file.path;

I'm fairly sure the 'hard-coded ID' should never change throughout the lifetime of an extension. That's the entire purpose of the ID: it's unique to that extension, permanently. Just store it as a constant and use that constant in your libraries. There's nothing wrong with that.
What IS bad practice is using the install.rdf, which exists for the sole purpose of... well, installing. Once the extension is developed, the install.rdf file's state is irrelevant and could well be inconsistent.
"An Install Manifest is the file an Add-on Manager-enabled XUL application uses to determine information about an add-on as it is being installed" [1]
To give it an analogy, it's like accessing the memory of a deleted object from an overflow. That object still exists in memory but it's not logically longer relevant and using its data is a really, really bad idea.
[1] https://developer.mozilla.org/en/install_manifests

Like lwburk, I don't think its available through Mozilla's API's, but I have an idea which works, but it seems like a complex hack. The basic steps are:
Set up a custom resource url to point to your extension's base directory
Read the file and parse it into XML
Pull the id out using XPath
Add the following line to your chrome.manifest file
resource packagename-base-dir chrome/../
Then we can grab and parse the file with the following code:
function myId(){
var req = new XMLHttpRequest();
// synchronous request
req.open('GET', "resource://packagename-base-dir/install.rdf", false);
req.send(null);
if( req.status !== 0){
throw("file not found");
}
var data = req.responseText;
// this is so that we can query xpath with namespaces
var nsResolver = function(prefix){
var ns = {
"rdf" : "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"em" : "http://www.mozilla.org/2004/em-rdf#"
};
return ns[prefix] || null;
};
var parser = CCIN("#mozilla.org/xmlextras/domparser;1", Ci.nsIDOMParser);
var doc = parser.parseFromString(data, "text/xml");
// you might have to change this xpath expression a bit to fit your setup
var myExtId = doc.evaluate("//em:targetApplication//em:id", doc, nsResolver,
Ci.nsIDOMXPathResult.FIRST_ORDERED_NODE_TYPE, null);
return myExtId.singleNodeValue.textContent;
}
I chose to use a XMLHttpRequest(as opposed to simply reading from a file) to retrieve the contents since in Firefox 4, extensions aren't necessarily unzipped. However, XMLHttpRequest will still work if the extension remains packed (haven't tested this, but have read about it).
Please note that resource URL's are shared by all installed extensions, so if packagename-base-dir isn't unique, you'll run into problems. You might be able to leverage Programmatically adding aliases to solve this problem.
This question prompted me to join StackOverflow tonight, and I'm looking forward participating more... I'll be seeing you guys around!

As Firefox now just uses Chrome's WebExtension API, you can use #serg's answer at How to get my extension's id from JavaScript?:
You can get it like this (no extra permissions required) in two
different ways:
Using runtime api: var myid = chrome.runtime.id;
Using i18n api: var myid = chrome.i18n.getMessage("##extension_id");

I can't prove a negative, but I've done some research and I don't think this is possible. Evidence:
This question, which shows that
the nsIExtensionManager interface
expects you to retrieve extension
information by ID
The full nsIExtensionManager interface
description, which shows no
method that helps
The interface does allow you to retrieve a full list of installed extensions, so it's possible to retrieve information about your extension using something other than the ID. See this code, for example:
var em = Cc['#mozilla.org/extensions/manager;1']
.getService(Ci.nsIExtensionManager);
const nsIUpdateItem = Ci.nsIUpdateItem;
var extension_type = nsIUpdateItem.TYPE_EXTENSION;
items = em.getItemList(extension_type, {});
items.forEach(function(item, index, array) {
alert(item.name + " / " + item.id + " version: " + item.version);
});
But you'd still be relying on hardcoded properties, of which the ID is the only one guaranteed to be unique.

Take a look on this add-on, maybe its author could help you, or yourself can figure out:
[Extension Manager] Extended is very
simple to use. After installing, just
open the extension manager by going to
Tools and the clicking Extensions. You
will now see next to each extension
the id of that extension.
(Not compatible yet with Firefox 4.0)
https://addons.mozilla.org/firefox/addon/2195