Develop Reference

How Discord checks if a Javascript File is a folder on client side? [duplicate]

I haven't seen any examples that do this. Is this not allowed in the API spec?
I am searching for an easy drag-drop solution for uploading an entire folder tree of photos.

It's now possible, thanks to Chrome >= 21.
function traverseFileTree(item, path) {
path = path || "";
if (item.isFile) {
// Get file
item.file(function(file) {
console.log("File:", path + file.name);
});
} else if (item.isDirectory) {
// Get folder contents
var dirReader = item.createReader();
dirReader.readEntries(function(entries) {
for (var i=0; i<entries.length; i++) {
traverseFileTree(entries[i], path + item.name + "/");
}
});
}
}
dropArea.addEventListener("drop", function(event) {
event.preventDefault();
var items = event.dataTransfer.items;
for (var i=0; i<items.length; i++) {
// webkitGetAsEntry is where the magic happens
var item = items[i].webkitGetAsEntry();
if (item) {
traverseFileTree(item);
}
}
}, false);
More info: https://protonet.info/blog/html5-experiment-drag-drop-of-folders/
As a note (from the comments) this code is not complete if more than 100 entries are returned, some iteration is required, see https://stackoverflow.com/a/53058574/885922

Unfortunately none of the existing answers are completely correct because readEntries will not necessarily return ALL the (file or directory) entries for a given directory. This is part of the API specification (see Documentation section below).
To actually get all the files, we'll need to call readEntries repeatedly (for each directory we encounter) until it returns an empty array. If we don't, we will miss some files/sub-directories in a directory e.g. in Chrome, readEntries will only return at most 100 entries at a time.
Using Promises (await/ async) to more clearly demonstrate the correct usage of readEntries (since it's asynchronous), and breadth-first search (BFS) to traverse the directory structure:
// Drop handler function to get all files
async function getAllFileEntries(dataTransferItemList) {
let fileEntries = [];
// Use BFS to traverse entire directory/file structure
let queue = [];
// Unfortunately dataTransferItemList is not iterable i.e. no forEach
for (let i = 0; i < dataTransferItemList.length; i++) {
// Note webkitGetAsEntry a non-standard feature and may change
// Usage is necessary for handling directories
queue.push(dataTransferItemList[i].webkitGetAsEntry());
}
while (queue.length > 0) {
let entry = queue.shift();
if (entry.isFile) {
fileEntries.push(entry);
} else if (entry.isDirectory) {
queue.push(...await readAllDirectoryEntries(entry.createReader()));
}
}
return fileEntries;
}
// Get all the entries (files or sub-directories) in a directory
// by calling readEntries until it returns empty array
async function readAllDirectoryEntries(directoryReader) {
let entries = [];
let readEntries = await readEntriesPromise(directoryReader);
while (readEntries.length > 0) {
entries.push(...readEntries);
readEntries = await readEntriesPromise(directoryReader);
}
return entries;
}
// Wrap readEntries in a promise to make working with readEntries easier
// readEntries will return only some of the entries in a directory
// e.g. Chrome returns at most 100 entries at a time
async function readEntriesPromise(directoryReader) {
try {
return await new Promise((resolve, reject) => {
directoryReader.readEntries(resolve, reject);
});
} catch (err) {
console.log(err);
}
}
Complete working example on Codepen: https://codepen.io/pen/QWmvxwV
FWIW I only picked this up because I wasn't getting back all the files I expected in a directory containing 40,000 files (many directories containing well over 100 files/sub-directories) when using the accepted answer.
Documentation:
This behaviour is documented in FileSystemDirectoryReader. Excerpt with emphasis added:
readEntries()
Returns a an array containing some number of the
directory's entries. Each item in the array is an object based on
FileSystemEntry—typically either FileSystemFileEntry or
FileSystemDirectoryEntry.
But to be fair, the MDN documentation could make this clearer in other sections. The readEntries() documentation simply notes:
readEntries() method retrieves the directory entries within the directory being read and delivers them in an array to the provided callback function
And the only mention/hint that multiple calls are needed is in the description of successCallback parameter:
If there are no files left, or you've already called readEntries() on
this FileSystemDirectoryReader, the array is empty.
Arguably the API could be more intuitive as well.
It's also worth noting that DataTransferItem.webkitGetAsEntry() is a non-standard feature and may change e.g. renamed getAsEntry(). Its usage is necessary to handle uploading files nested within directories.
Related:
johnozbay comments that on Chrome, readEntries will return at most 100 entries for a directory (verified as of Chrome 64).
Xan explains the correct usage of readEntries quite well in this answer (albeit without code).
Pablo Barría Urenda's answer correctly calls readEntries in a asynchronous manner without BFS. He also notes that Firefox returns all the entries in a directory (unlike Chrome) but we can't rely on this given the specification.

This function will give you a promise for array of all dropped files, like <input type="file"/>.files:
function getFilesWebkitDataTransferItems(dataTransferItems) {
function traverseFileTreePromise(item, path='') {
return new Promise( resolve => {
if (item.isFile) {
item.file(file => {
file.filepath = path + file.name //save full path
files.push(file)
resolve(file)
})
} else if (item.isDirectory) {
let dirReader = item.createReader()
dirReader.readEntries(entries => {
let entriesPromises = []
for (let entr of entries)
entriesPromises.push(traverseFileTreePromise(entr, path + item.name + "/"))
resolve(Promise.all(entriesPromises))
})
}
})
}
let files = []
return new Promise((resolve, reject) => {
let entriesPromises = []
for (let it of dataTransferItems)
entriesPromises.push(traverseFileTreePromise(it.webkitGetAsEntry()))
Promise.all(entriesPromises)
.then(entries => {
//console.log(entries)
resolve(files)
})
})
}
Usage:
dropArea.addEventListener("drop", function(event) {
event.preventDefault();
var items = event.dataTransfer.items;
getFilesFromWebkitDataTransferItems(items)
.then(files => {
...
})
}, false);
NPM package:
https://www.npmjs.com/package/datatransfer-files-promise
Usage example:
https://github.com/grabantot/datatransfer-files-promise/blob/master/index.html

In this message to the HTML 5 mailing list Ian Hickson says:
HTML5 now has to upload many files at
once. Browsers could allow users to
pick multiple files at once, including
across multiple directories; that's a
bit out of scope of the spec.
(Also see the original feature proposal.)
So it's safe to assume he considers uploading folders using drag-and-drop also out of scope. Apparently it's up to the browser to serve individual files.
Uploading folders would also have some other difficulties, as described by Lars Gunther:
This […] proposal must have two
checks (if it is doable at all):
Max size, to stop someone from uploading a full directory of several
hundred uncompressed raw images...
Filtering even if the accept attribute is omitted. Mac OS metadata
and Windows thumbnails, etc should be
omitted. All hidden files and
directories should default to be
excluded.

Now you can upload directories with both drag and drop and input.
<input type='file' webkitdirectory >
and for drag and drop(For webkit browsers).
Handling drag and drop folders.
<div id="dropzone"></div>
<script>
var dropzone = document.getElementById('dropzone');
dropzone.ondrop = function(e) {
var length = e.dataTransfer.items.length;
for (var i = 0; i < length; i++) {
var entry = e.dataTransfer.items[i].webkitGetAsEntry();
if (entry.isFile) {
... // do whatever you want
} else if (entry.isDirectory) {
... // do whatever you want
}
}
};
</script>
Resources:
http://updates.html5rocks.com/2012/07/Drag-and-drop-a-folder-onto-Chrome-now-available

Firefox now supports folder upload, as of November 15, 2016, in v50.0: https://developer.mozilla.org/en-US/Firefox/Releases/50#Files_and_directories
You can drag and drop folders into Firefox or you can browse and select a local folder to upload. It also supports folders nested in subfolders.
That means you can now use either Chrome, Firefox, Edge or Opera to upload folders. You can't use Safari or Internet Explorer at present.

Here's a complete example of how to use the file and directory entries API:
var dropzone = document.getElementById("dropzone");
var listing = document.getElementById("listing");
function scanAndLogFiles(item, container) {
var elem = document.createElement("li");
elem.innerHTML = item.name;
container.appendChild(elem);
if (item.isDirectory) {
var directoryReader = item.createReader();
var directoryContainer = document.createElement("ul");
container.appendChild(directoryContainer);
directoryReader.readEntries(function(entries) {
entries.forEach(function(entry) {
scanAndLogFiles(entry, directoryContainer);
});
});
}
}
dropzone.addEventListener(
"dragover",
function(event) {
event.preventDefault();
},
false
);
dropzone.addEventListener(
"drop",
function(event) {
var items = event.dataTransfer.items;
event.preventDefault();
listing.innerHTML = "";
for (var i = 0; i < items.length; i++) {
var item = items[i].webkitGetAsEntry();
if (item) {
scanAndLogFiles(item, listing);
}
}
},
false
);
body {
font: 14px "Arial", sans-serif;
}
#dropzone {
text-align: center;
width: 300px;
height: 100px;
margin: 10px;
padding: 10px;
border: 4px dashed red;
border-radius: 10px;
}
#boxtitle {
display: table-cell;
vertical-align: middle;
text-align: center;
color: black;
font: bold 2em "Arial", sans-serif;
width: 300px;
height: 100px;
}
<p>Drag files and/or directories to the box below!</p>
<div id="dropzone">
<div id="boxtitle">
Drop Files Here
</div>
</div>
<h2>Directory tree:</h2>
<ul id="listing"></ul>
webkitGetAsEntry is supported by Chrome 13+, Firefox 50+ and Edge.
Source: https://developer.mozilla.org/en-US/docs/Web/API/DataTransferItem/webkitGetAsEntry

Does HTML5 allow drag-drop upload of folders or a folder tree?
Only Chrome supports this feature. It has failed to have any traction and is likely to be removed.
Ref : https://developer.mozilla.org/en/docs/Web/API/DirectoryReader#readEntries

UPDATE: Since 2012 a lot has changed, see answers above instead. I leave this answer here for the sake of archeology.
The HTML5 spec does NOT say that when selecting a folder for upload, the browser should upload all contained files recursively.
Actually, in Chrome/Chromium, you can upload a folder, but when you do it, it just uploads a meaningless 4KB file, which represents the directory. Some servers-side applications like Alfresco can detect this, and warn the user that folders can not be uploaded:

Recently stumbled upon the need to implement this in two of my projects so I created a bunch of utility functions to help with this.
One creates a data-structure representing all the folders, files and relationship between them, like so 👇
{
folders: [
{
name: string,
folders: Array,
files: Array
},
/* ... */
],
files: Array
}
While the other just returns an Array of all the files (in all folders and sub-folders).
Here's the link to the package: https://www.npmjs.com/package/file-system-utils

I had been happy copy/pasting #grabantot 's solution until I met the 100 file limit issue.
#xlm 's solution overcomes the 100-file-limit, and it returns an array of FileEntry objects.
However in my project I need to extract the file paths from fileEntry objects.
This works if you have access to the ChromeFileSystem api:
const getAllPaths = async (dataTransferItems) =>{
async function getAllFileEntries(dataTransferItemList) {
let fileEntries = [];
// Use BFS to traverse entire directory/file structure
let queue = [];
for (let i = 0; i < dataTransferItemList.length; i++) {
queue.push(dataTransferItemList[i].webkitGetAsEntry());
}
while (queue.length > 0) {
let entry = queue.shift();
if (entry.isFile) {
fileEntries.push(entry);
} else if (entry.isDirectory) {
queue.push(...await readAllDirectoryEntries(entry.createReader()));
}
}
return fileEntries;
}
// Get all the entries (files or sub-directories) in a directory
// by calling readEntries until it returns empty array
async function readAllDirectoryEntries(directoryReader) {
let entries = [];
let readEntries = await readEntriesPromise(directoryReader);
while (readEntries.length > 0) {
entries.push(...readEntries);
readEntries = await readEntriesPromise(directoryReader);
}
return entries;
}
// Wrap readEntries in a promise to make working with readEntries easier
// readEntries will return only some of the entries in a directory
// e.g. Chrome returns at most 100 entries at a time
async function readEntriesPromise(directoryReader) {
try {
return await new Promise((resolve, reject) => {
directoryReader.readEntries(resolve, reject);
});
} catch (err) {
console.log(err);
}
}
const getDisplayPath = (entry)=>{
return new Promise((resolve, reject) =>{
chrome.fileSystem.getDisplayPath(entry, (path)=>{
if(chrome.runtime.lastError) {
reject(chrome.runtime.lastError)
}else {
resolve(path);
}
})
})
}
const fileEnties = await getAllFileEntries(dataTransferItems);
const files = await Promise.all(fileEnties.map(async(x)=>{
return (await getDisplayPath(x))
}))
return files;
}

Update HTML object with node.js and javascript

I'm new to nodejs and jquery, and I'm trying to update one single html object using a script.
I am using a Raspberry pi 2 and a ultrasonic sensor, to measure distance. I want to measure continuous, and update the html document at the same time with the real time values.
When I try to run my code it behaves like a server and not a client. Everything that i console.log() prints in the cmd and not in the browesers' console. When I run my code now i do it with "sudo node surveyor.js", but nothing happens in the html-document. I have linked it properly to the script. I have also tried document.getElementsByTagName("h6").innerHTML = distance.toFixed(2), but the error is "document is not defiend".
Is there any easy way to fix this?
My code this far is:
var statistics = require('math-statistics');
var usonic = require('r-pi-usonic');
var fs = require("fs");
var path = require("path");
var jsdom = require("jsdom");
var htmlSource = fs.readFileSync("../index.html", "utf8");
var init = function(config) {
usonic.init(function (error) {
if (error) {
console.log('error');
} else {
var sensor = usonic.createSensor(config.echoPin, config.triggerPin, config.timeout);
//console.log(config);
var distances;
(function measure() {
if (!distances || distances.length === config.rate) {
if (distances) {
print(distances);
}
distances = [];
}
setTimeout(function() {
distances.push(sensor());
measure();
}, config.delay);
}());
}
});
};
var print = function(distances) {
var distance = statistics.median(distances);
process.stdout.clearLine();
process.stdout.cursorTo(0);
if (distance < 0) {
process.stdout.write('Error: Measurement timeout.\n');
} else {
process.stdout.write('Distance: ' + distance.toFixed(2) + ' cm');
call_jsdom(htmlSource, function (window) {
var $ = window.$;
$("h6").replaceWith(distance.toFixed(2));
console.log(documentToSource(window.document));
});
}
};
function documentToSource(doc) {
// The non-standard window.document.outerHTML also exists,
// but currently does not preserve source code structure as well
// The following two operations are non-standard
return doc.doctype.toString()+doc.innerHTML;
}
function call_jsdom(source, callback) {
jsdom.env(
source,
[ 'jquery-1.7.1.min.js' ],
function(errors, window) {
process.nextTick(
function () {
if (errors) {
throw new Error("There were errors: "+errors);
}
callback(window);
}
);
}
);
}
init({
echoPin: 15, //Echo pin
triggerPin: 14, //Trigger pin
timeout: 1000, //Measurement timeout in µs
delay: 60, //Measurement delay in ms
rate: 5 //Measurements per sample
});

Node.js is a server-side implementation of JavaScript. It's ok to do all the sensors operations and calculations on server-side, but you need some mechanism to provide the results to your clients. If they are going to use your application by using a web browser, you must run a HTTP server, like Express.js, and create a route (something like http://localhost/surveyor or just http://localhost/) that calls a method you have implemented on server-side and do something with the result. One possible way to return this resulting data to the clients is by rendering an HTML page that shows them. For that you should use a Template Engine.
Any DOM manipulation should be done on client-side (you could, for example, include a <script> tag inside your template HTML just to try and understand how it works, but it is not recommended to do this in production environments).
Try searching google for Node.js examples and tutorials and you will get it :)

A property not initialized... yet it should be?

I have a list containing folders, and I'm trying to get the count of the total number of files in these folders.
I manage to retrieve a ListItemCollection containing my folders. Then it starts being... picky.
ctx is my ClientContext, and collection my ListItemCollection.
function countFiles()
{
var enumCollection = collection.getEnumerator();
while(enumCollection.moveNext())
{
currentItem = enumCollection.get_current();
var folder = currentItem.get_folder();
if (folder === 'undefined')
return;
ctx.load(folder, 'ItemCount');
ctx.executeQueryAsync(Function.createDelegate(this, function()
{
totalCount += folder.get_itemCount();
}), Function.createDelegate(this, onQueryFailed));
}
}
So it works... half of the time. If I have 6 items in my collection, I get 3 or 4 "The property or field 'ItemCount' has not been initialized" exceptions, and obviously my totalCount is wrong. I just can't seem to understand why, since the executeQueryAsync should not happen before the folder is actually loaded.
I'm very new to Javascript, so it may look horrid and be missing some essential code I didn't consider worthy of interest, feel free to ask if it is so.

Referencing closure variables (like folder in this case) from an asynchronous callback is generally a big problem. Thankfully it's easy to fix:
function countFiles()
{
function itemCounter(folder) {
return function() { totalCount += folder.get_itemCount(); };
}
var enumCollection = collection.getEnumerator();
while(enumCollection.moveNext())
{
var folder = enumCollection.getCurrent().get_folder();
if (folder === undefined) // not a string!
return;
ctx.load(folder, 'ItemCount');
ctx.executeQueryAsync(itemCounter(folder), Function.createDelegate(this, onQueryFailed));
}
}
(You don't need that .createDelegate() call because the function doesn't need this.)
Now, after that, you face the problem of knowing when that counter has been finally updated. Those asynchronous callbacks will eventually finish, but when? You could keep a separate counter, one for each query you start, and then decrement that in the callback. When it drops back to zero, then you'll know you're done.

Since SP.ClientContext.executeQueryAsync is an async function it is likely that the loop could be terminated before the first call to callback function completes, so the behavior of specified code could be unexpected.
Instead, i would recommend another and more clean approach for counting files (including files located under nested folders) using SharePoint JSOM.
How to count the total number of files in List using JSOM
The following function allows to count the number of list items in List:
function getItemsCount(listTitle, complete){
var ctx = SP.ClientContext.get_current();
var list = ctx.get_web().get_lists().getByTitle(listTitle);
var items = list.getItems(createQuery());
ctx.load(items);
ctx.executeQueryAsync(
function() {
complete(items.get_count());
},
function() {
complete(-1);
}
);
function createQuery()
{
var query = new SP.CamlQuery();
query.set_viewXml('<View Scope="RecursiveAll"><Query><Where><Eq><FieldRef Name="FSObjType" /><Value Type="Integer">0</Value></Eq></Where></Query></View>');
return query;
}
}
Usage
getItemsCount('Documents', function(itemsCount){
console.log(String.format('Total files count in Documents library: {0}',itemsCount));
});

Loading csv files content into dictionary of objects

I'm trying to load the content of the several CSV files into a new array. CSV files have a typical structure, with a label in the first row, and values (both string and real numbers) separated by commas. This part of code is responsible for loading the data for future use with Google Maps Api (not a problem for now, since I'm stuck on just loading the data). I would like to have a structure, in which I could call an element by it's name, that's why the var nodedata = {}; is created.
So the thing I totally don't get is why some part of the code is not being executed at all? console.log(nodedata); is empty, at least not in my Firefox console.
That's my attempt to the problem - links to the csv files are in the code.
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.js"></script>
<script src="http://jquery-csv.googlecode.com/files/jquery.csv-0.71.js"></script>
<script type="text/javascript">
var nodes = {};
var generation = {};
var nodedata = {};
$.get('https://dl.dropboxusercontent.com/u/25575808/energy/nodes.csv', function (response) {
nodes = $.csv.toObjects(response);
console.log(nodes);
});
$.get('https://dl.dropboxusercontent.com/u/25575808/energy/generation.csv', function (response) {
generation = $.csv.toObjects(response);
console.log(generation);
});
function getGeneration (nodename){
gen = 0;
for (var i = 0; i < generation.length; i++) {
if (generation[i].datetime == "2013-01-01 01:00"){
if (generation[i].node == nodename){
gen = gen + Number(generation[i]["output (MW)"])
}
}
}
return gen;
}
for (var i = 0; i < nodes.length; i++) {
nodedata[nodes[i].Node] = {
center: new google.maps.LatLng(nodes[i].Latitude,nodes[i].Longitude),
nodegen : getGeneration(nodes[i].Node)
}
}
console.log(nodedata);

I believe the problem you're having is unrelated to the usage of CSV data, rather it is the fact that the data is being loaded asynchronously.
You are executing 2 $.get() requests to load the files, which will take some time to download the files. The browser does not wait for them to finish before continuing through the rest of the code.
Therefore, it is possible for console.log(nodedate) to be executed before any data exists inside the nodes array.
An easy way to handle this is to stack your callback functions so that the first GET request completes -> run the 2nd GET request -> finally, run the processing code.
Check out this reorganization of the code: http://jsfiddle.net/Vr7sw/
(I removed the Google Maps line since I don't have the library loaded)

the problem is, the $.get requests are asynchronous (see jquery documentation), try to call to a function, into your callback body like this :
function nodesToJson(nodes) {
for (var i = 0; i < nodes.length; i++) {
nodedata[nodes[i].Node] = {
center: new google.maps.LatLng(nodes[node].Latitude,nodes[node].Longitude),
nodegen : getGeneration(nodes[i].Node)
}
}
console.log(nodedata);
}
$.get('https://dl.dropboxusercontent.com/u/25575808/energy/nodes.csv', function (response) {
nodes = $.csv.toObjects(response);
//when the request are ready, process the nodes
nodesToJson(nodes);
});

How to using javascript to read files into an array in order

I have a bunch of text files on server side with file names 0.txt, 1.txt, 2.txt, 3.txt and so forth. I want to read the content of all files and store them in an array A, such that A[0] has 0.txt's content, A[1] has 1.txt's, ...
How can I do it in Javascript / jquery?
Originally, I used $.ajax({}) in jQuery to load those text files. But it didn't work, because of the asynchronous nature of ajax. I tried to set $.ajax({...async=false...}), but it was very slow -- I have ~1000 10KB files to read in total.

from your question, you want to load txt file from server to local:
var done = 0, resultArr = [], numberOfFiles = 1000;
function getHandler(idx) {
return function(data) {
resultArr[idx] = data;
done++;
if (done === numberOfFiles) {
// tell your other part all files are loaded
}
}
}
for (var i = 0; i < numberOfFiles; i++) {
$.ajax(i + ".txt").done(getHandler(i));
}
jsFiddle: http://jsfiddle.net/LtQYF/1/

What you're looking for is File API introduced in HTML5 (working draft).
The examples in this article will point you in the right direction. Remember that the end user will have to initiate the action and manually select the files - otherwise it would have been a terrible idea privacy- and security-wise.
Update:
I found (yet again) the mozilla docos to be more readable! Quick html mockup:
<input type="file" id="files" name="files[]" onchange="loadTextFile();" multiple/>
<button id="test"onclick="test();">What have we read?</button>
...and the JavaScript:
var testArray = []; //your array
function loadTextFile() {
//this would be tidier with jQuery, but whatever
var _filesContainer = document.getElementById("files");
//check how many files have been selected and iterate over them
var _filesCount = _filesContainer.files.length;
for (var i = 0; i < _filesCount; i++) {
//create new FileReader instance; I have to read more into it
//but I was unable to just recycle one
var oFReader = new FileReader();
//when the file has been "read" by the FileReader locally
//log its contents and push them into an array
oFReader.onload = function(oFREvent) {
console.log(oFREvent.target.result);
testArray.push(oFREvent.target.result);
};
//actually initiate the read
oFReader.readAsText(_filesContainer.files[i]);
}
}
//sanity check
function test() {
for (var i = 0; i < testArray.length; i++) {
console.warn(testArray[i]);
}
}​
Fiddled

You don't give much information to give a specific answer. However, it is my opinion that "it doesn't work because of the asynchronous nature of ajax" is not correct. You should be able to allocate an array of the correct size and use a callback for each file. You might try other options such as bundling the files on the server and unbundling them on the client, etc. The designs, that address the problem well, depend on specifics that you have not provided.