I am posting a large amount of data. I need this to be as small as possible for performance reasons.
My data starts out as JS objects. I then stringify it using json. I then send it in a post.
The thing is that I have a lot of objects:lists [] and dict {}, as well as short texts, which are placed in quotes "" by json.
These are then uri encoded before being posted. I do not do this; the browser does it. I can see the result when I look in the request body.
So, every [, {, and "" is now uri encoded, meaning that my string becomes much longer. In fact, if I compare
alert( JSON_local.stringify(myStuff).length);
alert(encodeURI(JSON_local.stringify(myStuff).length);
the uri encoded string is 50% larger. That's a lot bigger when the string starts out big.
Am I missing something here? json is standard, but it seems to have a negative side effect for me. is there an alternative to using json? Or am I doing something wrong here? Data alsways has to be send as uri encoded, no?
Data always has to be send as uri encoded, no?
Not true. This depends on the content type you're sending it.
If you use x-www-form-urlencoded content-type when sending it, you need to encode the data. If you use multipart/form-data, for example, you do not need to. This has been discussed in more length in here. For considerable amount of data, I don't see any real reason to use x-www-form-urlencoded.
Of course, there is more to it than just changing the content type, you need to supply the mime boundaries then. It does sound to me however that that'd be more efficient for you. From http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4:
The content type "application/x-www-form-urlencoded" is inefficient
for sending large quantities of binary data or text containing
non-ASCII characters. The content type "multipart/form-data" should be
used for submitting forms that contain files, non-ASCII data, and
binary data.
Related
I am receiving an ID that is encrypted from the backend and I can see in the response it is coming without any question marks. But when passing this value in a request, I see some values are being changed. We are simply taking this value and passing it. Why would this be?
Example:
Getting from backend... ID = 7$ĄrÂŬÛ,ŕ4Ŀ+
While passing to another service... ID = 7$?rÂ?Û,?4?+
Edit: a few things to note
this is all happening within an iFrame
on page load, initial value is held in redux as an empty string ('')
issue does not happen when characters from the extended ascii table are not there, but only some ascii characters are changing to ?
Well, without any code to see what's really happening, I have a few guesses on what could be going on.
If the text is still encrypted or it's partially encrypted, then there won't actually be any character to display because it's in an encrypted binary format.
Make sure the encrypted text is being fully decrypted.
Character Support
If you are viewing the encrypted text in a terminal, it may not be able to display the characters correctly, (as those specific characters are only supported in
Unicode,
and not in
ASCII).
Try outputting the text to a file.
If you are sending the request in a URL, the URL can only take certain characters without special formatting, only a subset (64) of the characters from the ASCII character set can be used.
Make sure it's being encoded (and decoded) with
base64
for
URLs
and
forms.
I have been using openKM for document management and after retrieving image from openkm using api it shows question marks rectangles.
I have already checked this question but did not help
.
my python code for making api request
any help will be much appreciated
url = "http://ipaddress/aa18be7a5/hhhhhggg.png"
payload={}
headers = {'Internal-Key':"gjffhddsgsgdfgkhkhggdgsfd"}
response = requests.request("GET", url, headers=headers, data=payload)
return response.text
You requested .PNG data, and that's what the server sent you.
It all looks good.
You did this
response = requests.request("GET", url, ...)
return response.text
The request is beautiful.
But then you looked at .text, hoping for some unicode.
That would make sense, on a text document.
But instead you obtained a binary PNG document.
Just look at the returned headers -- they will
explain that its "Content-type" is "image/png".
You want response.content instead,
which is uninterpreted binary bytes,
suitable for writing to some "foo.png" file for display.
(Remember to use "wb" as the open() mode, not just "w"!)
It's important to understand that some byte strings
do not form valid utf-8, and trying to treat
binary images as unicode will soon end in tears.
I'm looking for a way to transfer the raw file data of any file-type with any possible content (By that I mean files and file-content are all user generated) both ways using xhr/ajax calls in a Backbone front-end against a Django back-end.
EDIT: Maybe the question is still unclear...
If you open a file in an IDE (such as Sublime), you can view and edit the actual code that comprises that file. I'm trying to put THAT raw content into a JSON so I can send to the browser, it can be modified, and then sent back.
I posted this question because I was under the impression that because the contents of these files can effectively be in ANY coding language that just stringify-ing the contents and sending it seems like a brittle solution that would be easy to break or exploit. Content could contain any number of ', ", { and } chars that would seem to break JSON formatting, and escaping those characters would leave artifacts within the code that would effectively break them (wouldn't it?).
If that assumption is wrong, THAT would also be an acceptable answer (so long as you could point out whatever it is I'm overlooking).
The project I'm working on is a browser-based IDE that will receive a complete file-structure from the server. Users can add/remove files, edit the content of those files, then save their changes back to the server. The sending/receiving all has to be handled via ajax/xhr calls.
Within Backbone, each "file" is instantiated as a model and stored in a Collection. The contents of the file would be stored as an attribute on the model.
Ideally, file content would still reliably throw all the appropriate events when changes are made.
Fetching contents should not be broken out into a separate call from the rest of the file model. I'd like to just use a single save/fetch call for sending/receiving files including the raw content.
Solutions that require Underscore/jQuery are fine, and I am able to bring in additional libraries if there is something available that specializes in managing that raw file data.
Interesting question. The code required to implement this would be quite involved, sorry that I'm not providing examples, but you seem like a decent programmer and should be able to implement what's mentioned below.
Regarding the sending of raw data through JSON, all you would need to do to make it JSON-safe and not break your code is to escape the special characters by stringyfying using Python's json.dumps & JavaScript's JSON.stringyfy. [1]
If you are concerned about some form of basic tamper-proofing, then light encoding of your data will fit the purpose, in addition to having the client and server pass a per-session token back and forth with JSON transfers to ensure that the JSON isn't forged from a malicious address.
If you want to check the end-to-end integrity of the data, then generate an md5 checksum and send it inside your JSON and then generate another md5 on arrival and compare with the one inside your JSON.
Base64 encoding: The size of your data would grow by 33% as it encodes four characters to represent three bytes of data.
Base85: Encodes four bytes as five characters and will grow your data by 25%, but uses much more processing overhead than Base64 in Python. That's a 8% improvement in data size, but at the expense of processing overhead. Also it's not string safe as double & single quotation marks, angle brackets, and ampersands cannot be used unescaped inside JSON, as it uses all 95 printable ASCII characters. Needs to be stringyfied before JSON transport. [2]
yEnc has as little as 2-3% overhead (depending on the frequency of identical bytes in the data), but is ruled out by impractical flaws (see [3]).
ZeroMQ Base-85, aka Z85. It's a string-safe variant of Base85, with a data overhead of 25%, which is better than Base64. No stringyfying necessary for sticking it into JSON. I highly recommended this encoding algorithm. [4] [5] [6]
If you're sending only small files (say a few KB), then the overhead of binary-to-text conversion will be acceptable. With files as large as a few Mbs, it might not be acceptable to have them grow by 25-33%. In this case you can try to compress them before sending. [7]
You can also send data to the server using multipart/form-data, but I can't see how this will work bi-directionally.
UPDATE
In conclusion, here's my solution's algorithm:
Sending data
Generate a session token and store it for the associated user upon
login (server), or retrieve from the session cookie (client)
Generate MD5 hash for the data for integrity checking during transport.
Encode the raw data with Z85 to add some basic tamper-proofing and JSON-friendliness.
Place the above inside a JSON and send POST when requested.
Reception
Grab JSON from POST
Retrieve session token from storage for the associated user (server), or retrieve from the session cookie (client).
Generate MD5 hash for the received data and test against MD5 in received JSON, reject or accept conditionally.
Z85-decode the data in received JSON to get raw data and store in file or DB (server) or process/display in GUI/IDE (client) as required.
References
[1] How to escape special characters in building a JSON string?
[2] Binary Data in JSON String. Something better than Base64
[3] https://en.wikipedia.org/wiki/YEnc
[4] http://rfc.zeromq.org/spec:32
[5] Z85 implementation in C/C++ https://github.com/artemkin/z85
[6] Z85 Python implementation of https://gist.github.com/minrk/6357188
[7] JavaScript zip library http://stuk.github.io/jszip/
[8] JavaScript Gzip SO JavaScript implementation of Gzip
AFAI am concerned a simple Base64 conversion will do it. Stringify, convert to base64, then pass it to the server and decode it there. Then you won't have the raw file transfer and you will still maintain your code simple.
I know this solution could seem a bit too simple, but think about it: many cryptographics algorithms can be broken given the right hardware. One of the most secure means would be through a digital certificate and then encrypt data with the private key and then send it over to the server. But, to reach this level of security every user of your application would have to have a digital certificate, which I think would be an excessive demand to your users.
So ask yourself, if implementing a really safe solution adds a lot of hassle to your users, why do you need a safe transfer at all? Based on that I reaffirm what I said before. A simple Base64 conversion will do. You can also use some other algotithms like SHA256 ou something to make it a litter bit safer.
If the only concern here is that the raw content of your code files (the "data" your model is storing), will cause some type of issue when stored in JSON, this is easily availed by escaping your data.
Stringifying your raw code file contents can cause issues as anything resembling JavaScript or JSON will be parsed into an actual JSON object. Your code file data can and should be stored simply as an esacaped string. Your fear here is that said string may contain characters that could break being stored in JavaScript inside a string, this is alleviated by escaping the entire string, and thus double, triple, quadruple, etc. escaping anything already escaped in the code file.
In essence it is important to remember here that raw code in a file is nothing but a glorified string when stored in a database, unless you are adding in-line metadata dynamically. It's just text, and doing standard escaping will make it safe to store in whatever format as a string (inside "" or '') in JSON.
I recommend reading this SO answer, as I also referenced it to verify what I already thought was correct:
How To Escape a JSON string containing newline characters using JavaScript
I have been breaking my head for the last couple of days trying to save the screenshot from a ThreeJS on the server using .NET Web API.
I have gone through all the possible questions on this specific topic and related ones on StackOverflow and tried the suggestions as well.
The issue details are specified below:
I am successfully able to get the base64 encoded string from renderer.domElement.toDataUrl() which contains a valid threejs image.
When I pass this string as is to my .NET WebAPI, the Convert.FromBase64String() method fails saying invalid length of the Base64 string or invalid padding characters. I ensured to extract out the "data:image/png;base64," part before passing.
I tried a number of things to resolve this issue like adding padding characters to ensure the length is mod 4, using regular expression to extract out the right data. I was able to get through the Convert.FromBase64String() and saved the resulting byte array as a png on my server. It resulted in a blank image.
I also discovered that when I used the chrome extension Advanced Rest Client to hit my WebAPI and used the Encode Payload feature before posting the string, I was able to get the image saved on my server successfully and got back the desired image as well.
Seeing this, I used the encodeURIComponent() function in Javascript to pass my base64 string to the WebAPI from my web app, but failed, getting back the same behavior as my earlier attempts.
One important observation was that the whitespaces were getting eliminated in case of Encode Payload but not in case of encodeURIComponent.
I compared the strings between encodeURIComponent() and the Encode Payload from Advanced Rest Client. Although, on a high level, they do the same thing by replacing the special characters with their escape sequences, there is still a significant difference between them.
Request help on this issue.
I would like to know if there is any other way of getting the threejs base64 string passed to .NET successfully.
What might be the difference between the encoding of encodeURIComponent and Advanced Rest Client Encode Payload feature?
Thanks in advance!
Using &url='+encodeURIComponent(url); to pass a URL from browser to server will encode the url but when it is decoded at the server, the parameters of url are interpreted as seperate parameters and not as part of the single url parameter.
What is the recommended way to pass urls as url parameters ?
encodeURIComponent() should work. For example,
'&url=' + encodeURIComponent("http://a.com/?q=query&n=10")
produces
"&url=http%3A%2F%2Fa.com%2F%3Fq%3Dquery%26n%3D10"
(which doesn't have any & or ? in the value).
When your server gets this url, it should be able to decode that to get the original:
param["url"] = "http://a.com/?q=query&n=10"
I'm not sure what server you're using (e.g. Rails, Django, ...) but that should work "out of the box" on any normal system.
Using '&url='+encodeURIComponent(url); to pass a URL from browser to server will encode the url
Yes, that's what you should be doing. encodeURIComponent is the correct way to encode a text value for putting in part of a query string.
but when it is decoded at the server, the parameters of url are interpreted as seperate parameters and not as part of the single url parameter.
Then the server is very broken indeed. If that's really what's happening, you need to fix it at the server end.
Code?
I ran into this issue, personally I couldn't use any of the accepted answers, but it can also be done by just encoding the url into Base 64, passing it as a parameter, and then decoding it. With javascript, you can encode a string s to base 64 with btoa(s) and decode with atob(s).
Other languages have ways of doing the same thing.
Base 64 is just kinda like representing a larger series of characters with 64 character(For ex, all the capital letters, all the lowercase, and a couple symbols). Kinda like how we represent letters in Binary. But it's nice to use, because then we can just pass base64 strings as parameters, and then they won't interfere/get interpreted in a weird fashion, and then we can decode them at the next stage.
Use escape() to url encode it, it will encode the ampersands so that does not happen.
Honestly, go with Google URL Shortener. Then you can just use the URL code in the url query string: http://example.com/url/A7dh3
In your application, take that and prepend the Google URL Shortener domain name, and do the redirect. This adds tracking to the URL through Google Analytics also. Lots of advantages in this approach. Just a short code and added tracking data, too.