I am writing a php script-wrapper for my API encode/decode requests and responses. The php script works fine with forms but I can't send properly encoded requests using javascript (angular)
In details:
I have an encoded request (on client side):
{"ct":"2UExUFiKFiZeT73AbLWYrCzJZnYUuCaaLq6VKcaboj/VQxDIrRXa6b4FvbVJXgU9","iv":"5c37e23d740ceb7611cb707bacd66633","s":"11bd03dda58ffbbe"}
The form converts this request into
%7B"ct"%3A"2UExUFiKFiZeT73AbLWYrCzJZnYUuCaaLq6VKcaboj/VQxDIrRXa6b4FvbVJXgU9%3D%3D"%2C"iv"%3A"5c37e23d740ceb7611cb707bacd66633"%2C"s"%3A"11bd03dda58ffbbe"%7D
It works, but I need to replicate it using angular http.get()
http.get doesn't convert this request properly. I tried to use encodeURI() but it doesn't help, obtain:
"%7B%22ct%22:%222UExUFiKFiZeT73AbLWYrCzJZnYUuCaaLq6VKcaboj/VQxDIrRXa6b4FvbVJXgU9%22,%22iv%22:%225c37e23d740ceb7611cb707bacd66633%22,%22s%22:%2211bd03dda58ffbbe%22%7D"
It converts quotes, what based on the previous example I don't need
Do you have any ideas?
I'm trying to do client-side processing of some data sent in a server's HTTP response.
Here's what I'm working with: I have a web application that sends commands to a backend simulation engine, which then sends back a bunch of data/results in the response body. I want to be able to access this response using JavaScript (note..not making a new response, but simply accessing the data already sent from the server).
Right now, I am able to do this via a kludgy hack of sorts:
var responseText = "{{response}}";
This is using Django's template system, where I have already pre-formatted the template context variable "response" to contain a pre-formatted string representation of a csv file (i.e., proper unicode separators, etc).
This results in a huge string being transmitted to the page. Right now, this supports my immediate goal of making this data available for download as a csv, but it doesn't really support more sophisticated tasks. Also, I'm not sure if it will scale well when my string is, say, 2 MB as opposed to less than 1 KB.
I'd like to have this response data stored more elegantly, perhaps as part of the DOM or maybe in a cache (?) [not familiar with this].
The ideal way to do this is to not load the csv on document load, either as a javascript variable or as part of the DOM. Why would you want to load a 2MB data every time to the user when his intention may not be to download the csv everytime?
I suggest creating a controller/action for downloading the csv and get it on click of the download button.
I have a page in wich the user will write a code (in a programming language). the code can be very long (more than 2000 characters) and can contain any type of characters. And when he press the send button the code will be sent (using javascript ajax) to a script file on the server etc..
currently i am using this to send the code with ajax :
jxobj.open('POST', 'www.my-website.com/my_script.php', true);
jxobj.send('code=' + code);
but if the code contains some characters like & then the value of code will be broken into many pieces and i can't retrieve it in my script file.
Is there any way to send ajax request without a string concatenation? I would like to use only javascript if possible and thanks.
The data format is well documented and the standard JavaScript encodeURIComponent function will encode data for it.
jxobj.send('code=' + encodeURIComponent(code));
Is there any way to send ajax request without a string concatenation?
Not really. The send method takes a string, so you have to build a string using some mechanism or another. You could achieve it with (for instance) arrays and joins, but it still comes down to the same basic principles.
I'm looking for a way to transfer the raw file data of any file-type with any possible content (By that I mean files and file-content are all user generated) both ways using xhr/ajax calls in a Backbone front-end against a Django back-end.
EDIT: Maybe the question is still unclear...
If you open a file in an IDE (such as Sublime), you can view and edit the actual code that comprises that file. I'm trying to put THAT raw content into a JSON so I can send to the browser, it can be modified, and then sent back.
I posted this question because I was under the impression that because the contents of these files can effectively be in ANY coding language that just stringify-ing the contents and sending it seems like a brittle solution that would be easy to break or exploit. Content could contain any number of ', ", { and } chars that would seem to break JSON formatting, and escaping those characters would leave artifacts within the code that would effectively break them (wouldn't it?).
If that assumption is wrong, THAT would also be an acceptable answer (so long as you could point out whatever it is I'm overlooking).
The project I'm working on is a browser-based IDE that will receive a complete file-structure from the server. Users can add/remove files, edit the content of those files, then save their changes back to the server. The sending/receiving all has to be handled via ajax/xhr calls.
Within Backbone, each "file" is instantiated as a model and stored in a Collection. The contents of the file would be stored as an attribute on the model.
Ideally, file content would still reliably throw all the appropriate events when changes are made.
Fetching contents should not be broken out into a separate call from the rest of the file model. I'd like to just use a single save/fetch call for sending/receiving files including the raw content.
Solutions that require Underscore/jQuery are fine, and I am able to bring in additional libraries if there is something available that specializes in managing that raw file data.
Interesting question. The code required to implement this would be quite involved, sorry that I'm not providing examples, but you seem like a decent programmer and should be able to implement what's mentioned below.
Regarding the sending of raw data through JSON, all you would need to do to make it JSON-safe and not break your code is to escape the special characters by stringyfying using Python's json.dumps & JavaScript's JSON.stringyfy. [1]
If you are concerned about some form of basic tamper-proofing, then light encoding of your data will fit the purpose, in addition to having the client and server pass a per-session token back and forth with JSON transfers to ensure that the JSON isn't forged from a malicious address.
If you want to check the end-to-end integrity of the data, then generate an md5 checksum and send it inside your JSON and then generate another md5 on arrival and compare with the one inside your JSON.
Base64 encoding: The size of your data would grow by 33% as it encodes four characters to represent three bytes of data.
Base85: Encodes four bytes as five characters and will grow your data by 25%, but uses much more processing overhead than Base64 in Python. That's a 8% improvement in data size, but at the expense of processing overhead. Also it's not string safe as double & single quotation marks, angle brackets, and ampersands cannot be used unescaped inside JSON, as it uses all 95 printable ASCII characters. Needs to be stringyfied before JSON transport. [2]
yEnc has as little as 2-3% overhead (depending on the frequency of identical bytes in the data), but is ruled out by impractical flaws (see [3]).
ZeroMQ Base-85, aka Z85. It's a string-safe variant of Base85, with a data overhead of 25%, which is better than Base64. No stringyfying necessary for sticking it into JSON. I highly recommended this encoding algorithm. [4] [5] [6]
If you're sending only small files (say a few KB), then the overhead of binary-to-text conversion will be acceptable. With files as large as a few Mbs, it might not be acceptable to have them grow by 25-33%. In this case you can try to compress them before sending. [7]
You can also send data to the server using multipart/form-data, but I can't see how this will work bi-directionally.
UPDATE
In conclusion, here's my solution's algorithm:
Sending data
Generate a session token and store it for the associated user upon
login (server), or retrieve from the session cookie (client)
Generate MD5 hash for the data for integrity checking during transport.
Encode the raw data with Z85 to add some basic tamper-proofing and JSON-friendliness.
Place the above inside a JSON and send POST when requested.
Reception
Grab JSON from POST
Retrieve session token from storage for the associated user (server), or retrieve from the session cookie (client).
Generate MD5 hash for the received data and test against MD5 in received JSON, reject or accept conditionally.
Z85-decode the data in received JSON to get raw data and store in file or DB (server) or process/display in GUI/IDE (client) as required.
References
[1] How to escape special characters in building a JSON string?
[2] Binary Data in JSON String. Something better than Base64
[3] https://en.wikipedia.org/wiki/YEnc
[4] http://rfc.zeromq.org/spec:32
[5] Z85 implementation in C/C++ https://github.com/artemkin/z85
[6] Z85 Python implementation of https://gist.github.com/minrk/6357188
[7] JavaScript zip library http://stuk.github.io/jszip/
[8] JavaScript Gzip SO JavaScript implementation of Gzip
AFAI am concerned a simple Base64 conversion will do it. Stringify, convert to base64, then pass it to the server and decode it there. Then you won't have the raw file transfer and you will still maintain your code simple.
I know this solution could seem a bit too simple, but think about it: many cryptographics algorithms can be broken given the right hardware. One of the most secure means would be through a digital certificate and then encrypt data with the private key and then send it over to the server. But, to reach this level of security every user of your application would have to have a digital certificate, which I think would be an excessive demand to your users.
So ask yourself, if implementing a really safe solution adds a lot of hassle to your users, why do you need a safe transfer at all? Based on that I reaffirm what I said before. A simple Base64 conversion will do. You can also use some other algotithms like SHA256 ou something to make it a litter bit safer.
If the only concern here is that the raw content of your code files (the "data" your model is storing), will cause some type of issue when stored in JSON, this is easily availed by escaping your data.
Stringifying your raw code file contents can cause issues as anything resembling JavaScript or JSON will be parsed into an actual JSON object. Your code file data can and should be stored simply as an esacaped string. Your fear here is that said string may contain characters that could break being stored in JavaScript inside a string, this is alleviated by escaping the entire string, and thus double, triple, quadruple, etc. escaping anything already escaped in the code file.
In essence it is important to remember here that raw code in a file is nothing but a glorified string when stored in a database, unless you are adding in-line metadata dynamically. It's just text, and doing standard escaping will make it safe to store in whatever format as a string (inside "" or '') in JSON.
I recommend reading this SO answer, as I also referenced it to verify what I already thought was correct:
How To Escape a JSON string containing newline characters using JavaScript
I'm using JQuery '.serialize' + '.post' to send form data using ajax to a non-unicode, asp based website. The result data is url-encoded in unicode (characters are encoded in double values).
Can I post the form data encoded like it's sent using the form submit(), using JQuery ajax?
Ok, after hours and hours of trying...
Javascript uses UTF-8 for strings, and AJAX requests are always sent with UTF-8, period. There will always be a conversion on the server side if UTF-8 is not supported.
Description in the JQuery contentType parameter states:
Data will always be transmitted to the server using UTF-8 charset; you
must decode this appropriately on the server side.
So, eventhough there are QA's which claim some solutions as working, they are simply some conversions on the client & the server side. So firefox's firebug will always show UTF-8 characters on the net panel for AJAX requests, and it will never look like a submitted form with single byte characters.
You can use
var encStr = encodeURIComponent(str)