Develop Reference

How to cache a file loaded through Ajax - JavaScript (Node JS)

I am working on an application where I am loading a pdf file through https.get() in node js, sending the data in chunks to the front end. It works pretty well, The reason I did it like this is that I sometimes load the pdf from a remote server, which causes a CORS issue, so I have written a simple proxy to handle it for me. It works well, but the problem is, every time I make a request from the front-end, the pdf is re-loaded and each time it loads it. What I want to do is, when the pdf file is loaded for the first time, I should cache it may be on the front end or backend and then load it from the cache each time the page is refreshed.
The proxy written in Node JS
The reading the file stream on front end
const url = "http://localhost:8003/pdf-cors";
const file = "https://veraviafit.com/wp-content/uploads/2018/11/N2RPlan-3-1.pdf";
fetch(`${url}?file=${file}`).then(async(response) => {
const reader = response!.body!.getReader();
const stream = new ReadableStream({
start(controller) {
// The following function handles each data chunk
function push() {
// "done" is a Boolean and value a "Uint8Array"
reader.read().then((r) => {
// console.log(r);
const {
done,
value
} = r;
console.log("VALUE ---> ", value);
// Is there no more data to read?
if (done) {
// Tell the browser that we have finished sending data
controller.close();
return;
}
// console.log(value);
// Get the data and send it to the browser via the controller
controller.enqueue(value);
push();
});
};
push();
}
});
return new Response(stream, {
headers: {
"Content-Type": "text/html"
}
});
});

Check out the memoize function from lodash. You could implement a function like this yourself quite easily if you don't want to include lodash.
https://lodash.com/docs/4.17.15#memoize

I found a feasible solution that dropped the loading time perfectly. So the first time it took 2.5 minutes to load this pdf (really depends on the internet speed), and the second time when I reloaded it, it took only 54 seconds to load the pdf as it was cached, all I had to do is to set response cache headers, which will actually cache the loaded data. You can visit this question to see how it works, I set it like this in my proxy code.
In order to load cache the dynamic files, I let the file load for the first time and save it with a specified cache 'key', and then write the whole chunks to that file. After the file is loaded, when I reload the page, and a request is made for the same file, I am sending back data from the file and not loading external resources on each call. It loads the file in milliseconds.

Blob name issue with new tab in chrome and firefox [duplicate]

In my Vue app I receive a PDF as a blob, and want to display it using the browser's PDF viewer.
I convert it to a file, and generate an object url:
const blobFile = new File([blob], `my-file-name.pdf`, { type: 'application/pdf' })
this.invoiceUrl = window.URL.createObjectURL(blobFile)
Then I display it by setting that URL as the data attribute of an object element.
<object
:data="invoiceUrl"
type="application/pdf"
width="100%"
style="height: 100vh;">
</object>
The browser then displays the PDF using the PDF viewer. However, in Chrome, the file name that I provide (here, my-file-name.pdf) is not used: I see a hash in the title bar of the PDF viewer, and when I download the file using either 'right click -> Save as...' or the viewer's controls, it saves the file with the blob's hash (cda675a6-10af-42f3-aa68-8795aa8c377d or similar).
The viewer and file name work as I'd hoped in Firefox; it's only Chrome in which the file name is not used.
Is there any way, using native Javascript (including ES6, but no 3rd party dependencies other than Vue), to set the filename for a blob / object element in Chrome?
[edit] If it helps, the response has the following relevant headers:
Content-Type: application/pdf; charset=utf-8
Transfer-Encoding: chunked
Content-Disposition: attachment; filename*=utf-8''Invoice%2016246.pdf;
Content-Description: File Transfer
Content-Encoding: gzip

Chrome's extension seems to rely on the resource name set in the URI, i.e the file.ext in protocol://domain/path/file.ext.
So if your original URI contains that filename, the easiest might be to simply make your <object>'s data to the URI you fetched the pdf from directly, instead of going the Blob's way.
Now, there are cases it can't be done, and for these, there is a convoluted way, which might not work in future versions of Chrome, and probably not in other browsers, requiring to set up a Service Worker.
As we first said, Chrome parses the URI in search of a filename, so what we have to do, is to have an URI, with this filename, pointing to our blob:// URI.
To do so, we can use the Cache API, store our File as Request in there using our URL, and then retrieve that File from the Cache in the ServiceWorker.
Or in code,
From the main page
// register our ServiceWorker
navigator.serviceWorker.register('/sw.js')
.then(...
...
async function displayRenamedPDF(file, filename) {
// we use an hard-coded fake path
// to not interfere with legit requests
const reg_path = "/name-forcer/";
const url = reg_path + filename;
// store our File in the Cache
const store = await caches.open( "name-forcer" );
await store.put( url, new Response( file ) );
const frame = document.createElement( "iframe" );
frame.width = 400
frame.height = 500;
document.body.append( frame );
// makes the request to the File we just cached
frame.src = url;
// not needed anymore
frame.onload = (evt) => store.delete( url );
}
In the ServiceWorker sw.js
self.addEventListener('fetch', (event) => {
event.respondWith( (async () => {
const store = await caches.open("name-forcer");
const req = event.request;
const cached = await store.match( req );
return cached || fetch( req );
})() );
});
Live example (source)
Edit: This actually doesn't work in Chrome...
While it does set correctly the filename in the dialog, they seem to be unable to retrieve the file when saving it to the disk...
They don't seem to perform a Network request (and thus our SW isn't catching anything), and I don't really know where to look now.
Still this may be a good ground for future work on this.
And an other solution, I didn't took the time to check by myself, would be to run your own pdf viewer.
Mozilla has made its js based plugin pdf.js available, so from there we should be able to set the filename (even though once again I didn't dug there yet).
And as final note, Firefox is able to use the name property of a File Object a blobURI points to.
So even though it's not what OP asked for, in FF all it requires is
const file = new File([blob], filename);
const url = URL.createObjectURL(file);
object.data = url;

In Chrome, the filename is derived from the URL, so as long as you are using a blob URL, the short answer is "No, you cannot set the filename of a PDF object displayed in Chrome." You have no control over the UUID assigned to the blob URL and no way to override that as the name of the page using the object element. It is possible that inside the PDF a title is specified, and that will appear in the PDF viewer as the document name, but you still get the hash name when downloading.
This appears to be a security precaution, but I cannot say for sure.
Of course, if you have control over the URL, you can easily set the PDF filename by changing the URL.

I believe Kaiido's answer expresses, briefly, the best solution here:
"if your original URI contains that filename, the easiest might be to simply make your object's data to the URI you fetched the pdf from directly"
Especially for those coming from this similar question, it would have helped me to have more description of a specific implementation (working for pdfs) that allows the best user experience, especially when serving files that are generated on the fly.
The trick here is using a two-step process that perfectly mimics a normal link or button click. The client must (step 1) request the file be generated and stored server-side long enough for the client to (step 2) request the file itself. This requires you have some mechanism supporting unique identification of the file on disk or in a cache.
Without this process, the user will just see a blank tab while file-generation is in-progress and if it fails, then they'll just get the browser's ERR_TIMED_OUT page. Even if it succeeds, they'll have a hash in the title bar of the PDF viewer tab, and the save dialog will have the same hash as the suggested filename.
Here's the play-by-play to do better:
You can use an anchor tag or a button for the "download" or "view in browser" elements
Step 1 of 2 on the client: that element's click event can make a request for the file to be generated only (not transmitted).
Step 1 of 2 on the server: generate the file and hold on to it. Return only the filename to the client.
Step 2 of 2 on the client:
If viewing the file in the browser, use the filename returned from the generate request to then invoke window.open('view_file/<filename>?fileId=1'). That is the only way to indirectly control the name of the file as shown in the tab title and in any subsequent save dialog.
If downloading, just invoke window.open('download_file?fileId=1').
Step 2 of 2 on the server:
view_file(filename, fileId) handler just needs to serve the file using the fileId and ignore the filename parameter. In .NET, you can use a FileContentResult like File(bytes, contentType);
download_file(fileId) must set the filename via the Content-Disposition header as shown here. In .NET, that's return File(bytes, contentType, desiredFilename);
client-side download example:
download_link_clicked() {
// show spinner
ajaxGet(generate_file_url,
{},
(response) => {
// success!
// the server-side is responsible for setting the name
// of the file when it is being downloaded
window.open('download_file?fileId=1', "_blank");
// hide spinner
},
() => { // failure
// hide spinner
// proglem, notify pattern
},
null
);
client-side view example:
view_link_clicked() {
// show spinner
ajaxGet(generate_file_url,
{},
(response) => {
// success!
let filename = response.filename;
// simplest, reliable method I know of for controlling
// the filename of the PDF when viewed in the browser
window.open('view_file/'+filename+'?fileId=1')
// hide spinner
},
() => { // failure
// hide spinner
// proglem, notify pattern
},
null
);

I'm using the library pdf-lib, you can click here to learn more about the library.
I solved part of this problem by using api Document.setTitle("Some title text you want"),
Browser displayed my title correctly, but when click the download button, file name is still previous UUID. Perhaps there is other api in the library that allows you to modify download file name.

How to execute javascript function after another function completes [duplicate]

I want to redirect the user to a different webpage after they click a hyperlink which allows them to download a file. However since they need to make a choice in the open/save file dialog, I don't want to redirect them until they accept the download.
How can I detect that they performed this action?

As i've found from years of maintaining download.js, there simply is no way to tell from JS (or likely in general, see below) what a user chooses to do with the download Open/Save dialog. It's a common feature request, and i've looked into it repeatedly over the years. I can say with confidence that it's impossible; I'll joyfully pay 10 times this bounty if someone can demo a mechanical way to determine the post-prompt user action on any file!
Further, it's not really just a matter of JS rules, the problem is complicated by the way browsers download and prompt such files. This means that even servers can't always tell what happened. There might be some specific work-arounds for a few specific cases, but they are not pretty or simple.
You could force your users to "re-upload" a downloaded file with an <input type=file> to validate it, but that's cumbersome at best, and the local file browse dialog could be alarming to some. It's the only sure-fire method to ensure a download, but for non-sensitive applications its very draconian, and it won't work on some "mobile" platforms that lack file support.
You might also try watching from the server side, pushing a message to the client that the file was hit on the server. The problem here is that downloads start downloading as soon as the Open/Save dialog appears, though invisibly in the background. That's why if you wait a few moments to "accept" a large file, it seems to download quickly at first. From the server's perspective, the activity is the same regardless of what the user does.
For a huge file, you could probably detect that the whole file was not transferred, which implies the user clicked "cancel", but it's a complicated syncing procedure pushing the status from backend to client. It would require a lot of custom programming with sockets, PHP message passing, EventSource, etc for little gain. It's also a race against time, and an uncertain amount of time at that; and slowing down the download is not recommended for user satisfaction.
If it's a small file, it will physically download before the user even sees the dialog, so the server will be useless. Also consider that some download manager extensions take over the job, and they are not guaranteed to behave the same as a vanilla browser. Forcing a wait can be treacherous to someone with a slow hard drive that takes "forever" to "finish" a download; we've all experienced this, and not being able to continue while the "spinny" winds down would lower user satisfaction, to put it mildly.
In short, there's no simple way, and really no way in general, except for huge files you know will take a long time to download. I've spent a lot of blood sweat and tears trying to provide my download.js users the ability, but there are simply no good options. Ryan dahl initially wrote node.js so he could provide his users an upload progress bar, maybe someone will make a server/client package to make it easy to do the same for downloads.

Here is a hacky solution.
My StreamSaver lib don't use blobs to download a file with a[download]. It uses service worker to stream something to the disc by emulating how the server handles download with content-disposition attachment header.
evt.respondWith(
new Response(
new ReadableStream({...})
)
)
Now you don't have any exact way of knowing what the user pressed in the dialog but you have some information about the stream. If the user press cancel in the dialog or abort the ongoing download then the stream gets aborted too.
The save button is trickier. But lets begin with what a stream bucket highWaterMark can tell us.
In my torrent example I log the writer.desiredSize. It's the correlation to how much data it is willing to receive. When you write something to the stream it will lower the desired size (whether it be a count or byte strategy). If it never increases then it means that user probably have paused the download. When it goes down below 0 then you are writing more data than what the user is asking for.
And every chunk write you do returns a promise
writer.getWriter().write(uint8).then(() => {
// Chunk have been sent to the destination bucket
// and desiredSize increase again
})
That promise will resolve when the bucket isn't full. But it dose not mean that the chunk have been written to the disc yet, it only means that the chunk has been passed from the one stream to another stream (from write -> readable -> respondWith) and will often do so in the beginning of the stream and when another earlier chunk have been written to the disc.
It's a possibility that the write stream can finishes even before the user makes a choice if the hole data can fit within the bucket (memory)
Tweaking the bucket size to be lower then the data can help
So you can make assumption on when the
download starts
finish
and pauses
but you won't know for sure since you don't get any events (apart from the abort that closes the stream)
Note that the torrent example don't show correct size if you don't have support for Transferable streams but you could get around this if you do everything inside a service worker. (instead of doing it in the main thread)
Detecting when the stream finish is as easy as
readableStream.pipeTo(fileStream).then(done)
And for future references WICG/native-file-system might give you access to write files to disc but it has to resolve a prompt dialog promise before you can continue and might be just what the user is asking for.
There are examples of saving a blob as a stream, and even more multiple blob's as a zip too if you are interested.

Given that user is, or should be aware that file should be downloaded before next step in process, user should expect some form of confirmation that file has been downloaded to occur.
You can create a unique idenfifier or timestamp to include within downloaded file name by utilizing <a> element with download attribute set to a the modified file name.
At click event of <button> element call .click() on <a> element with href set to a Blob URL of file. At a element click handler call .click() on an <input type="file"> element, where at attached change event user should select same file which was downloaded at the user action which started download of file.
Note the chaining of calls to .click() beginning with user action. See Trigger click on input=file on asynchronous ajax done().
If the file selected from user filesystem is equal to modified downloaded file name, call function, else notify user that file download has not been confirmed.
window.addEventListener("load", function() {
let id, filename, url, file;
let confirmed = false;
const a = document.querySelector("a");
const button = document.querySelector("button");
const confirm = document.querySelector("input[type=file]");
const label = document.querySelector("label");
function confirmDownload(filename) {
if (confirmed) {
filename = filename.replace(/(-\d+)/, "");
label.innerHTML = "download of " + filename + " confirmed";
} else {
confirmed = false;
label.innerHTML = "download not confirmed";
}
URL.revokeObjectURL(url);
id = url = filename = void 0;
if (!file.isClosed) {
file.close()
}
}
function handleAnchor(event) {
confirm.click();
label.innerHTML = "";
confirm.value = "";
window.addEventListener("focus", handleCancelledDownloadConfirmation);
}
function handleFile(event) {
if (confirm.files.length && confirm.files[0].name === filename) {
confirmed = true;
} else {
confirmed = false;
}
confirmDownload(filename);
}
function handleDownload(event) {
// file
file = new File(["abc"], "file.txt", {
type: "text/plain",
lastModified: new Date().getTime()
});
id = new Date().getTime();
filename = file.name.match(/[^.]+/g);
filename = filename.slice(0, filename.length - 1).join("")
.concat("-", id, ".", filename[filename.length - 1]);
file = new File([file], filename, {
type: file.type,
lastModified: id
});
a.download = filename;
url = URL.createObjectURL(file);
a.href = url;
alert("confirm download after saving file");
a.click();
}
function handleCancelledDownloadConfirmation(event) {
if (confirmed === false && !confirm.files.length) {
confirmDownload(filename);
}
window.removeEventListener("focus", handleCancelledDownloadConfirmation);
}
a.addEventListener("click", handleAnchor);
confirm.addEventListener("change", handleFile);
button.addEventListener("click", handleDownload);
});
<button>download file</button>
<a hidden>download file</a>
<input type="file" hidden/>
<label></label>
plnkr http://plnkr.co/edit/9NmyiiQu2xthIva7IA3v?p=preview

jquery.fileDownload allows you to do this:
$(document).on("click", "a.fileDownloadPromise", function () {
$.fileDownload($(this).prop('href'))
.done(function () { alert('File download a success!'); })
.fail(function () { alert('File download failed!'); });
return false;
});
Take a look at Github:
https://github.com/johnculviner/jquery.fileDownload

I had a project that I dabbled in recently that required me to specify whether a user could upload a particular kind of file, i.e. (a user can upload a png but not a pdf).
I may not used the most efficient method, but ultimately what I did was to code a small, built in "webapp" that functioned as a file browser, for upload or download.
I suppose the closest example without releasing my "secret project" would be https://encodable.com/filechucker/
Maybe you could write a simple integrated filebrowser such as that that cloud services use sometimes (i.e. dropbox) and have some functions that detect input with custom boxes and stuff.
Just a few thoughts.

window.showSaveFilePicker from the File System Access API does what you want, but unfortunately it's currently supported only by Chrome and Edge. It returns a promise -- if the user chooses to download the file the promise is resolved; if they cancel, an AbortError is raised.

try this:
<html>
<head>
<script type="text/javascript">
function Confirmation(pg) {
var res = confirm("Do you want to download?");
if(res){
window.open(pg,"_blank");
}
return res;
}
</script>
</head>
<body>
download
</body>
</html>

How can I get multiple files to upload to the server from a Javascript page without skipping?

I'm working on a research experiment which uses getUserMedia, implemented in recorder.js, to record .wav files from the user's microphone and XMLHttpRequest to upload them to the server. Each file is about 3 seconds long and there are 36 files in total. The files are recorded one after another and sent to the server as soon as they are recorded.
The problem I'm experiencing is that not all of the files end up on the server. Apparently the script or the php script are unable to catch up with all the requests in a row. How can I make sure that I get all the files? These are important research data, so I need every recording.
Here's the code that sends the files to the server. The audio data is a blob:
var filename = subjectID + item__number;
xhr.onload=function(e) {
if(this.readyState === 4) {
console.log("Server returned: ",e.target.responseText);
}
};
var fd=new FormData();
fd.append("audio_data",blob, filename);
xhr.open("POST","upload_wav.php",true);
xhr.send(fd);
And this is the php file on the server side:
print_r($_FILES);
$input = $_FILES['audio_data']['tmp_name'];
$output = "audio/".$_FILES['audio_data']['name'].".wav";
move_uploaded_file($input, $output)
This way of doing things is basically copied from this website:
Using Recorder.js to capture WAV audio in HTML5 and upload it to your server or download locally
I have already tried making the XMLHttpRequest wait by using
while (xhr.readyState != 4)
{
console.log("Waiting for server...")
}
It just caused the page to hang.
Would it be better to use ajax than XMLHttp Request? Is there something I can do to make sure that all the files get uploaded? I'm pretty new to Javascript so code examples are appreciated.

I have no idea what your architecture looks like, but here is a potential solution that will work to solve your problem.
The solution uses the Web Worker API to off load the file uploading to a sub-process. This is done with the Worker Interface of that API. This approach will work because there is no contention of the single thread of the main process - web workers work in their own processes.
Using this approach, we do three basic things:
create a new worker passing a script to execute
pass messages to the worker for the worker to deal with
pass messages back to the main process for status updates/replies/resolved data transformation/etc.
The code is heavily commented below to help you understand what is happening and where.
This is the main JavaScript file (script.js)
// Create a sub process to handle the file uploads
///// STEP 1: create a worker and execute the worker.js file immediately
let worker = new Worker('worker.js');
// Ficticious upload count for demonstration
let uploadCount = 12;
// repeatedly build and send files every 700ms
// This is repeated until uplaodCount == 0
let builder = setInterval(buildDetails, 700);
// Recieve message from the sub-process and pipe them to the view
///// STEP 2: listen for messages from the worker and do something with them
worker.onmessage = e => {
let p = document.createElement('pre');
// e.data represents the message data sent from the sub-process
p.innerText = e.data;
document.body.appendChild(p);
};
/**
* Sort of a mock to build up your BLOB (fake here of-course)
*
* Post the data needed for the FormData() to the worker to handle.
*/
function buildDetails() {
let filename = 'subject1234';
let blob = new Blob(['1234']);
///// STEP 3: Send a message to the worker with file details
worker.postMessage({
name: "audio_data",
blob: blob,
filename: filename
});
// Decrease the count
uploadCount--;
// if count is zero (== false) stop the fake process
if (!uploadCount) clearInterval(builder);
}
This is the sub-process JavaScript file (worker.js)
// IGNORE the 'fetch_mock.js' import that is only here to avoid having to stand up a server
// FormDataPolyFill.js is needed in browsers that don't yet support FormData() in workers
importScripts('FormDataPolyFill.js', 'fetch_mock.js');
// RXJS provides a full suite of asynchronous capabilities based around Reactive Programming (nothing to do with ReactJS);
// The need for your use case is that there are guarantees that the stream of inputs will all be processed
importScripts('https://cdnjs.cloudflare.com/ajax/libs/rxjs/6.3.3/rxjs.umd.js');
// We create a "Subject" that acts as a vessel for our files to upload
let forms = new rxjs.Subject();
// This says "every time the forms Subject is updated, run the postfile function and send the next item from the stream"
forms.subscribe(postFile);
// Listen for messages from the main process and run doIt each time a message is recieved
onmessage = doIt;
/**
* Takes an event object containing the message
*
* The message is presumably the file details
*/
function doIt(e) {
var fd = new FormData();
// e.data represents our details object with three properties
fd.append(e.data.name, e.data.blob, e.data.filename);
// Now, place this FormData object into our stream of them so it can be processed
forms.next(fd);
}
// Instead of using XHR, this uses the newer fetch() API based upon Promises
// https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API
function postFile(fd) {
// Post the file to the server (This is blocked in fetch_mock.js and doesn't go anywhere)
fetch('fake', {
method: 'post',
body: fd,
})
.then((fd) => {
// After the XHR request is complete, 'Then' post a message back to the main thread (If there is a need);
postMessage("sent: " + JSON.stringify(fd));
});
}
Since this will not run in stackoverflow, I've created a plunker so that you can run this example:
http://plnkr.co/edit/kFY6gcYq627PZOATXOnk
If all this seems complicated, you've presented a complicated problem to solve. :-)
Hope this helps.

Access JSON property value before loading the contents of the file

I have an angularjs project which retrieves JSON files from a server and uses the contents to display the data in the screen.
I'm using a service to load the data, and this service calls the server for a new JSON file every 2 seconds (I removed that from the code below for simplicity).
var data = $resource(:file.json', {}, {
query: {method: 'GET', params: {file: '#file'}}
});
this.load = function(file, myFunction) {
data.query({file:file}, function(data) {
myFunction(data);
}
}
Now, these files can be really big and sometimes there's no need to process the file because there are no changes from the previous one received. I have a property in the JSON file with the version number, and I should not process the file unless that version number is higher than the one in the previous file.
I can do that by calling the query service, which loads the file contents into a js object and then check the version, if the file is really big it might take a while to load it. Is there a way to access that property value (version) ONLY and then, depending on it, load the file into a js object?
EDIT: The thing that I'm guessing is that loading a 1MB JSON file to check a version number inside it might take a while (or maybe no and that $resource action is really fast, anyone knows?), but I'm not really sure that it can be done any other way, as I'm checking a specific property inside the file.
Many thanks in advance.

HTML5 and Javascript now provides a File API which can be used to read the file line by line. You can find information regarding this feature here:
http://www.html5rocks.com/en/tutorials/file/dndfiles/
This will slice the full file into string and take just the first line(asuming the version is in there)
data.substr(0, data.indexOf("\n"));
--
Bonus:
Also in this answer you will find out how to read the first line of a file:
https://stackoverflow.com/a/12227851/2552259
var XHR = new XMLHttpRequest();
XHR.open("GET", "http://hunpony.hu/today/changelog-en.txt", true);
XHR.send();
XHR.onload = function (){
console.log( XHR.responseText.slice(0, XHR.responseText.indexOf("\n")) );
};
Another question with the same topic:
https://stackoverflow.com/a/6861246/2552259
var txtFile = new XMLHttpRequest();
txtFile.open("GET", "http://website.com/file.txt", true);
txtFile.onreadystatechange = function()
{
if (txtFile.readyState === 4) { // document is ready to parse.
if (txtFile.status === 200) { // file is found
allText = txtFile.responseText;
lines = txtFile.responseText.split("\n");
}
}
}
txtFile.send(null);

Do you have access to the json files?
I'm not sure how you generate your json files but you could try adding the version number in the filename and check if a newer filename exists. I have not tested this but maybe it's worth a try.