Develop Reference

How to download entire website from inside the website

I'm making a website, in which I want to offer the user to download the whole website (CSS and images included) for them to modify. I know I can download individual resources with
Click Me
but like I said, this only downloads one file, whereas I would like to download the entire website.
If it helps you visualise what I mean: in chrome, IE and Firefox you can press ctrl+s to download the entire website (make sure you save it as Web page, Complete.
Edit: I know I can create a .zip file that it will download, however doing so requires me to update it every time I make a change, which is something I'd rather not do, as I could potentially be making a lot of changes.

As I mention, it is better that you will have a cron job or something like this that once in a while will create you a zip file of all the desired static content.
If you insist doing it in javascript at the client side have a look at JSZip .
You still have to find a way to get the list of static files of the server to save.
For instance, you can create a txt file with each line is a link to a webpage static file.
you will have to iterate over this file and use $.get to get it's content.
something like this:
// Get list of files to save (either by GET request or hardcoded)
filesList = ["f1.json /echo/jsonp?name=1", "inner/f2.json /echo/jsonp?name=2"];
function createZip() {
zip = new JSZip();
// make bunch of requests to get files content
var requests = [];
// for scoping the fileName
_then = (fname) => data => ({ fileName: fname, data });
for (var file of filesList) {
[fileName, fileUrl] = file.split(" ");
requests.push($.get(fileUrl).then(_then(fileName)));
}
// When all finished
$.when(...requests).then(function () {
// Add each result to the zip
for (var arg of arguments) {
zip.file(arg.fileName, JSON.stringify(arg.data));
}
// Save
zip.generateAsync({ type: "blob" })
.then(function (blob) {
saveAs(blob, "site.zip");
});
});
}
$("#saver").click(() => {
createZip();
});
JSFiddle
Personally, I don't like this approach. But do as you prefer.

Why is it taking so long to download a zip file through AJAX?

On my webpage, there is a feature to let user downloading a zip file. The way it works as after user clicked the download button, a GET request would be sent to the api, and then server sends back the file data as arraybuffer. Then the UI converts the data into a blob object and save it as a zip file. Here is the sample code for that:
function download(){
var req = {
method: 'GET',
url: "api/download",
contentType: "application/zip",
responseType: "arraybuffer",
headers: {"Accept": "application/zip"}
};
$http(req).then(function (res) {
if(res.status === 200){
var file = new Blob([res.data], { type: "application/zip" });
var link = document.createElement('a');
link.href = URL.createObjectURL(file);
link.download = "MyFile.zip";
document.body.appendChild(link);
link.click();
document.body.removeChild(link);
}
}, function (err) {});
}
The code works great except for one issue: the overall process seems taking too long to complete. Even for a zip file that only 100kb, it still gonna cost at least 2 minutes to download. When I check the dev tool, I noticed the data transfer process finished in almost 3-5 seconds, and then the program was like just halting there for another 2 minutes before proceeding to the next step. (meanwhile the dev tool keep showing the request is still going on). What make it seems more peculiar is that I have some test files that are in different sizes, say from 100KB to 150MB, and the downloading time are roughly the same (in less than 5 seconds different).
Can someone elaborate on what could be the cause, and how can I improve this? Or if there is a better way to achieve the goal through a different workflow/model?
Thank you in advance!

The issue seems to be that you're temporarily storing the ZIP file in the clients browser memory via the Blob where it's getting stored twice (once as the resultant data, once as a Blob). This looks like something that would work a lot better if you just have the server generate the zip as a temporary file and send the client a link to that.

How to display a PDF in iframe with ajax after calling a servlet

I have a problem when I want to display a PDF in an iframe (using the attribute src).The PDF is never downloaded from the browser.
I need to use an ajax call to display a message in a popup in some cases.
The code of the servlet that generate the PDF works because currently we use this directly in the src attribute of the iframe.
<iframe src="myServlet" />
But when I use the servlet with a ajax call, and I want to add the content of the PDF in the iframe, it doesn't work.
I don't know why.
This is an example of the js :
$.ajax(
{
url: "myServlet",
...
complete: function(data) {
var blob = new Blob([data], { type: 'application/pdf' });
var downloadUrl = URL.createObjectURL(blob);
$('#myframe').attr("src", downloadUrl);
}
}
);
I try with chrome and I have the error:
jquery-2.1.3.js?ts=13012017:3 GET data:application/pdf;base64,blob:http://localhost:8180/d21d82b6-8254-4a38-907e-129b3ac037fa net::ERR_INVALID_URL
I can see that I have the data as a binary file:
%PDF-1.4
%����
3 0 obj
<>stream
x��{PU��^A��e�(�......................
...................
startxref
38448
%%EOF
Sorry but I don't have the code here (at home), but I think that the explanations are fairly clear.
I don't know why the PDF cannot be downloaded if I use ajax with iframe and attribut src.
Do you have an idea?
Thanks.
Fred

This do the trick for me.
$.ajax(
{
url: "myServlet",
...
complete: function(data) {
$('#myframe').attr('src','data:application/pdf;base64,'+data);
}
}
);
Also, you should encode the ajax ans this way base64_encode($data).
I hope you find this helpful.

In Javascript, to encode the ajax answer you should do it this way:
var enc = btoa(data)
If you have problems, you should try this:
var enc = btoa(unescape(encodeURIComponent(data)))
Then
$('#myframe').attr('src','data:application/pdf;base64,'+ enc);

Pass PDF document to user-agent via XHR, using modern (HTML5) methods, without compromising document encoding

I am trying to accomplish exactly what is described at Handle file download from ajax post , but with a dynamically-generated PDF file, which I'm generating with PHP v5.6.10 (using the third-party PDFLib extension, v9.0.5).
Yet, when the PDF file is downloaded, it is corrupted; no PDF-reading implementation that I've tried is able to read the file, and every observation points to the fact that the file content is being butchered somewhere between printing the PDF content to the response body and saving the file via the user-agent (web-browser) with JavaScript.
I happen to be using jQuery v2.1.4, but I'm not sure that it matters, ultimately.
Important Provisos
I should mention that, like the other asker (cited above), I have an HTML form that users fill-out and submit via the POST verb. The form submission is performed with JavaScript, because there are actually 5 forms displayed in a tabbed layout that are submitted simultaneously, and any validation errors must be sent back via AJAX and displayed without refreshing the entire page). I mention this to make clear the fact that this is a POST request, which may return either a) a JSON object (that contains validation error strings, primarily), or b) a string that represents a PDF document, which should be presented to the user-agent as a file download.
My Code
The JavaScript
$('#submit-button').click(function() {
$.ajax({
url: $(this).data('action'),
type: 'post',
data: $($(this).data('forms')).serialize(),
processData: false,
statusCode: {
500: function() {
alert('An internal server error occurred. Go pound sand.');
}
}
}).done(function(data, status, xhr) {
processResponse(data, status, xhr);
}).fail(function(jqXHR, textStatus) {
if (textStatus === 'timeout') {
alert('The request timed-out. Please try again.');
}
});
});
function processResponse(response, status, xhr)
{
if (response !== null && typeof response === 'object') {
//The server will return either a JSON string (if the input was invalid)
//or the PDF file. We land here in the former case.
}
else {
//This doesn't change the behavior.
xhr.responseType = 'blob';
//This doesn't change the behavior, either.
//xhr.overrideMimeType('text\/plain; charset=x-user-defined');
//The remainder of this function taken verbatim from:
//https://stackoverflow.com/a/23797348
// check for a filename
var filename = "";
var disposition = xhr.getResponseHeader('Content-Disposition');
if (disposition && disposition.indexOf('attachment') !== -1) {
var filenameRegex = /filename[^;=\n]*=((['"]).*?\2|[^;\n]*)/;
var matches = filenameRegex.exec(disposition);
if (matches != null && matches[1]) filename = matches[1].replace(/['"]/g, '');
}
var type = xhr.getResponseHeader('Content-Type');
//Is logged to console as "application/pdf".
console.log(type);
var blob = new Blob([response], { type: type });
if (typeof window.navigator.msSaveBlob !== 'undefined') {
// IE workaround for "HTML7007: One or more blob URLs were revoked by closing the blob for which they were created. These URLs will no longer resolve as the data backing the URL has been freed."
window.navigator.msSaveBlob(blob, filename);
} else {
var URL = window.URL || window.webkitURL;
var downloadUrl = URL.createObjectURL(blob);
//Is logged to console as URL() (it's an object, not a string).
console.log(URL);
//Is logged to console as "blob:https://example.com/108eb066-645c-4859-a4d2-6f7a42f4f369"
console.log(downloadUrl);
//Is logged to console as "pdftest.pdf".
console.log(filename);
if (filename) {
// use HTML5 a[download] attribute to specify filename
var a = document.createElement("a");
// safari doesn't support this yet
if (typeof a.download === 'undefined') {
window.location = downloadUrl;
} else {
a.href = downloadUrl;
a.download = filename;
document.body.appendChild(a);
a.click();
}
} else {
window.location = downloadUrl;
}
setTimeout(function () { URL.revokeObjectURL(downloadUrl); }, 100); // cleanup
}
}
}
The PHP
<?php
use File;
use \PDFLib;
class Pdf {
protected $p;
protected $bufferedContent;
public function __construct()
{
$this->p = new PDFlib();
$this->p->set_option('errorpolicy=return');
$this->p->set_option('textformat=utf8');
$this->p->set_option('escapesequence=true');
}
//...
public function sendToBrowser()
{
$this->bufferedContent = $this->p->get_buffer();
header_remove();
header('Content-Type: application/pdf');
header('Content-Length: ' . strlen($this->bufferedContent));
header('Content-Disposition: attachment; filename=pdftest.pdf');
$bytesWritten = File::put(realpath(__DIR__ . '/../../public/assets/pdfs') . '/' . uniqid() . '.pdf', $this->bufferedContent);
echo $this->bufferedContent;
exit;
}
//...
}
Notice that in the PHP method I am writing the PDF file to disk prior to sending it in the response body. I added this bit to determine whether the PDF file written to disk is corrupted, too, and it is not; it opens perfectly well in every reader I've tried.
Observations and Theories
What I find so strange about this is that I've tried the download in three different browsers (the most recent versions of Chrome, Firefox, and IE 11) and the PDF size is drastically different with each browser. Following are the file sizes from each:
Written to disk (not corrupted): 105KB
Chrome: 193KB
Firefox: 188KB
IE 11: 141KB
At this point, I am convinced that the problem relates to the encoding used within the PDF. I discovered a discrepancy when using WinMerge to compare the copy of the PDF that I dump directly to disk before returning the HTTP response with the copy that is handled via AJAX.
The first clue was this error message, which appears when I attempt to compare the two PDF documents:
I click OK to dismiss the error, and the comparison resumes.
The functional/correct PDF (at right, in WinMerge) is encoded using Windows-1252 (CP1252); I assume that that encoding happens within PDFLib (despite running on a GNU/Linux system). One can see from the PHP snippet, above, that I am calling $this->p->set_option('textformat=utf8'); explicitly, but that seems to set the encoding for input text that is included in the PDF document (and not the document encoding).
Ultimately, I am left wondering if there is any means by which to get this PDF to be displayed correctly after download.
Change the PDF Encoding, Instead?
I wonder if there is a "good reason" for which PDFLib is using Windows-1252 encoding to generate the PDF document. Is there any chance that this is as simple as changing the encoding on the PDFLib side to match what jQuery's AJAX implementation requires (UTF-8)?
I've consulted the PDFLib manual for more information, and there is a section dedicated to this subject: 4.2 Unicode-capable Language Bindings. This section has two subsections: 4.2.1 Language Bindings with native Unicode Strings (PHP is not among them) and 4.2.2 Language Bindings with UTF-8 Support (PHP falls into this category). But everything discussed herein seems to pertain to the actual strings that are inserted into the PDF body, and not to the overall document encoding.
Then there is 4.4 Single-Byte (8-Bit) Encodings, with the following note:
Note The information in this section is unlikely to be required in
Unicode workflows.
How does one employ a Unicode workflow in this context?
The manual is available at http://www.pdflib.com/fileadmin/pdflib/pdf/manuals/PDFlib-9.0.5-tutorial.pdf for anyone who feels it may be useful.
Approaches That I'd Prefer to Avoid
I really hesitate to get into the business of re-encoding the PDF in JavaScript, client-side, once it has been downloaded. If that is the only means by which to achieve this, I will go another direction.
Initially, my primary aim was to avoid an approach that could leave abandoned PDF files laying-around on the server, in some temporary directory (thereby necessitating a clean-up cron-job or similar), but that may be the only viable option.
If necessary, I will implement an interstitial step whereby I write the PDF file to disk (on the web-server), pass it to the client using some unsightly hidden-iframe hack, and then delete the file once the user-agent receives it. Of course, if the user-agent never finishes the download, the user closes the browser, etc., the file will be abandoned and I'll be left to clean it up by some other means (the idea of which I hate, on principle).
Any assistance with this is hugely appreciated.

You tried this on iframe?
I have this same problem, but i resolve by iframe.
ugly code but works for me.
solution with iframe

Access JSON property value before loading the contents of the file

I have an angularjs project which retrieves JSON files from a server and uses the contents to display the data in the screen.
I'm using a service to load the data, and this service calls the server for a new JSON file every 2 seconds (I removed that from the code below for simplicity).
var data = $resource(:file.json', {}, {
query: {method: 'GET', params: {file: '#file'}}
});
this.load = function(file, myFunction) {
data.query({file:file}, function(data) {
myFunction(data);
}
}
Now, these files can be really big and sometimes there's no need to process the file because there are no changes from the previous one received. I have a property in the JSON file with the version number, and I should not process the file unless that version number is higher than the one in the previous file.
I can do that by calling the query service, which loads the file contents into a js object and then check the version, if the file is really big it might take a while to load it. Is there a way to access that property value (version) ONLY and then, depending on it, load the file into a js object?
EDIT: The thing that I'm guessing is that loading a 1MB JSON file to check a version number inside it might take a while (or maybe no and that $resource action is really fast, anyone knows?), but I'm not really sure that it can be done any other way, as I'm checking a specific property inside the file.
Many thanks in advance.

HTML5 and Javascript now provides a File API which can be used to read the file line by line. You can find information regarding this feature here:
http://www.html5rocks.com/en/tutorials/file/dndfiles/
This will slice the full file into string and take just the first line(asuming the version is in there)
data.substr(0, data.indexOf("\n"));
--
Bonus:
Also in this answer you will find out how to read the first line of a file:
https://stackoverflow.com/a/12227851/2552259
var XHR = new XMLHttpRequest();
XHR.open("GET", "http://hunpony.hu/today/changelog-en.txt", true);
XHR.send();
XHR.onload = function (){
console.log( XHR.responseText.slice(0, XHR.responseText.indexOf("\n")) );
};
Another question with the same topic:
https://stackoverflow.com/a/6861246/2552259
var txtFile = new XMLHttpRequest();
txtFile.open("GET", "http://website.com/file.txt", true);
txtFile.onreadystatechange = function()
{
if (txtFile.readyState === 4) { // document is ready to parse.
if (txtFile.status === 200) { // file is found
allText = txtFile.responseText;
lines = txtFile.responseText.split("\n");
}
}
}
txtFile.send(null);

Do you have access to the json files?
I'm not sure how you generate your json files but you could try adding the version number in the filename and check if a newer filename exists. I have not tested this but maybe it's worth a try.