I transfer a fair amount of XML from the server to the client, in my application (250K-500K each time).
How can I compress it on the server, and decompress it on the client using standard JavaScript? Is it possible?
First, a scolding and possible workarounds. Second, a possible crazy solution.
The Scolding
If you're sending 250k - 500k back and forth between the client and server, you're doing it wrong. Look into
Only requesting and sending back the information you need
If there's information in the XML that doesn't change often, configure your web-server for aggressive http caching for these requests. Also, consider sending the "doesn't change often information" down with the initial page request instead of grabbing it from the server
Consider sending down JSON instead of XML. JSON is less verbose than XML and will usually take up less space. If your client-side code NEEDS an XML object, reconstruct it from the JSON you just downloaded.
If you're using apache, mod_deflate will gzip your data for client that can accept it. Last time I looked into gzip encoding on IIS it did it's zipping and unzipping on disk, which quickly became a bottle-neck in high load situations (YMMV)
The Crazy Solution
If you're hell-bent on trying the compress-in-javascript route, the LZ77 compression algorithum has been ported to most languages, including javascript.
Find the analogous code for your server platform, compress the XML, and then feed it to the code found in the link above. To avoid encoding issues, I'd try wrapping the compressed data itself in either XML or JSON when sending it down.
Javascript compression performance is going to be slow.as.molasses, which is why I discourage this route.
Is this HTTP type transfers?
The easiest way would be to look into compression at the web server side.
XML Usually compresses really well.
Related
I want to display a static HTML page with expressjs. What's the fastest way to do so? Currently I have a html file stored in the file system and I'm using res.sendfile(home + "/public/file.xml")
Is sendfile the fastest way to do it or might storing the file in some kind of database be faster?
Thanks
Storing files in database? Well, some people try to do such evil things. But what is the point? I mean database engine has to read the file from disk as well. Additionally has to do many other things making the loading process longer.
Now you seem to confuse some things. Sending files is the same thing as sending any other response. At some point you have to send the binary data, so you cannot avoid using res.send (or any other form of it). This is of course if we are talking about Node.js, because you might consider serving files via nginx, which is probably the best static file server there is.
One last thing: if you still want to serve files via Node.JS, then you may consider caching them in memory. This of course depends on how many and how big files you want to serve. This is as easy as creating global variable which holds the data and sends it whenever there is a request. That's the only real performance boost you can make with Node.JS.
Express has support for static file serving: app.use(express.static(__dirname + '/public'));.
I am working on a file upload system which will store individual parts of large files on more than one server. So the distribution of a 1GB file will look something like this:
Server 1: 0-128MB
Server 2: 128MB-256MB
Server 2: 256MB-384MB
... etc
The intention of this is to allow for redundancy (each part will exist on more than one server), security (no one server has access to the entire file), and cost (bandwidth expenses are distributed).
I am curious if anyone has an opinion on how I might be able to "trick" web browsers into downloading the various parts all in one link.
What I had in mind was something like:
Browser is linked to Server 1, which provides a content-size of the full file
Once 128MB is served, Server 1 will intentionally close the connection
Hopefully, the browser will try to restart the download, requesting Server 1
Server 1 provides a 3XX redirect to Server 2
Browser continues downloading from Server 2
I don't know for certain that my example works, as I haven't tested it yet. I was curious if there were other solutions someone might have?
I'd like to make the whole process as easy as possible (ideally requiring no work beyond a simple download). I don't want the users to have to use another program (ie: cat'ing the files together). I'd also like to not use a proxy server, since it would incur extra bandwidth costs.
As far as I'm aware, there is no javascript solution for writing a file, if there was one, that would be great.
AFAIK this is not possible by using the HTTP protocol. You can probably use a custom browser extension but it would depend on the browser. Another alternative is to create a Java applet that would download the file from different servers. The applet can accept the URLs to the different servers as parameters.
To save the generated file:
https://stackoverflow.com/a/4551467/329062
That solution stores the file in memory though, so it won't work with very large files.
You can download the partial files into a JS variable using JSONP. That will also let you get around the same-origin policy.
Javascripts security model will only allow you to access data from the same origin where the Javascript came from - i.e. not multiple servers.
If you are going to have the file bits on multiple servers, you will need the user to load the web page, fetch the bit and then finally stick the bits together in the correct order. If you can manage to get all your users to do this (correctly), you are a better man than I.
It's possible to do in modern browsers over standard HTTP.
You can use XHR2 with CORS to download file chunks as ArrayBuffers and then merge them using Blob constructor and use createObjectURL to send merged file to the user.
However, I suspect that browsers will store these objects in RAM, so it's probably a bad idea to use it for large files.
I have a emailing system in a program. The person writes a email, sends it to a database, and several people will be allowed to retreive that email later on. However, these emails are sometimes short but are sometimes quite long.
Consequently I would like to compress these emails before sending them to the database (in the javascript on the client side), then uncompress them when they are retreived (again in the js on the client side).
Are there any libraries (free) that can do this? Is such a thing easy to write myself?
If you really want to do this at the client end, you can maybe look how JSend works - it's not designed to save space; it's designed to compress data for transportation, but it's an example of encoding/decoding data. You will probably need to write your own script that does a similar thing.
Remember, by making the client do the compression, you need it to work in every browser. A server side script is much more reliable (and efficient).
You may want to look at InnoDB Data Compression. If you can implement it, you might find that a native database process is much more efficient than anything you can do in Javascript.
I had a discussion with my colleague about Flash security. We're in the phase of planning some things for our web project that is using Flash plugin to display content. We need to dynamically pull settings for the Flash application from the server, using JSON.
Proposal that I offered was that we should save an extra HTTP request to pull the data file after the plugin is loaded and embed the JSON directly in the page containing the Flash plugin. Flash would fire a Javascript function that'd return the deserialized JSON data to it.
My colleague opposed this proposal with significant "security concerns".
I believe that there's literally zero difference between these two approaches besides the fact that his approach requires additional HTTP request. All of this is client/server and client should never be trusted. If I want to change the data that is in the JSON query, I can do that in both cases. File pull is little more difficult to hack though, but possible with custom HTTP proxy.
What are your thoughts?
There is no difference. Both can be fabricated.
if you really care that much about delivering original settings to the .swf:
don't use http - httpFox is a brilliant plugin - use a server that supports RTMP/RTMPE and NetConnection.call() to retrieve the data.
create an algorithm for validating original json so that your app won't work if the config doesn't pass the test.
after the config is loaded your swf might check the values with the server (not all at a time) and throw an error if something goes wrong
Why is it not a good idea to use SOAP for communicating with the front end? For example, a web browser using JavaScript.
Because it's bloated
Because JSON is natively understandable by the JavaScript
Because XML isn't fast to manipulate with JavaScript.
Because SOAP reinvents a lot of the HTTP wheel in its quest for protocol-independence. What's the point if you know you're going to serve the response over HTTP anyway (since your client is a web browser)?
UPDATE: I second gizmo's (implied) suggestion of JSON.
If the web browser is your only client then I would have to agree that SOAP is overkill.
However, if you are going to have multiple types of front end clients on running on different platforms then SOAP may be appropriate. The nice part about SOAP is that there are a lot of tools out there that will generate code for you to handle sending, receiving, and parsing of SOAP based on the WSDL file.
For example, if you wanted to develop a C++ front end client then all you need is the WSDL file and Microsoft's tools will generate all the C++ code to generate the SOAP request based on a data structure, send the request, receive the response, and parse the response into a return data structure.
There are tools to do this both on the client and server side.
It could be done. Just remember that SOAP is not the fastest way to exchange information as there is a big overhead (big XMLs have to be sent back and forth) - that's probably why you don't see it used that often