Develop Reference

JSON not transferring "+" character to CGI script

I am currently working on a personal project, where I have to deal with some chemical formulas;
I have a form with JavaScript where I enter these formulas; The formulas are entered in a LaTeX-like style for super- en subscript.
An example formula can be found below:
Fe^{3+}
When I use JavaScript to read the form and console.log(); the formula is working as expected.
However if I send the formula to the back-end (Python with CGI), the + character seems to have disappeared and been replaced with a space.
I thought it had something to do with escaping the character, since parts of the formula look a lot like regex's, but after looking around, I couldn't find anything that would suggest that I had to escape the + character.
And now I have absolutely no idea how to resolve this... I could use a different character and replace it on the back-end but that seems like it is not the optimal solution...

Most important question: How did you invoke the CGI script?
With HTTP GET or HTTP POST?
If you're using HTTP POST and the data was being transferred in the HTTP Data portion, then you don't need to escape the "+" sign.
But if you're using HTTP GET, then the "+" sign will first be translated according to URL encoding standard (thus, "+" becomes a space), before transferred to the CGI script.
So in the latter scenario, you need to escape the "+" sign (and other 'special' characters such as "%" and "?").

Ckeditor 4 does not encode accented chars

I am using the ck editor (version 4) and users enter greek text. Most characters are converted into html entities but the accented characters seem to be ignored.
On my page and within my database the text is stored an can be displayed but when I create an email with the entered text the characters are not shown correctly.
Is there a config setting that I can use so that all greek characters are converted? Or do I have to try and change these characters manually?

This sounds like an encoding issue rather than converting the characters into entities. Presumably the text is being stored as UTF-8 in the database, and the webs pages are also in the same encoding, so the characters get displayed correctly.
When it comes to the email, I suspect that it's being encoded in another encoding like ASCII or ISO-8859-1 (Latin characters only) so the Greek text will appear as a mix of letters and punctuation rather than as it should look.
You don't mention what you're using to send the email, but I'd probably look there first to see if there's a way of setting the character encoding to the same encoding as the data is stored in the database.

.NET removing all spaces when decoding URI

I am sending a parameter, from Javascript to .NET. The parameter could contain multiple spaces like 'John [3 spaces here, stackoverflow not showing them] Smith', I need the spaces to stay. However, it looks like the spaces disappear in .NET. In an attempt to fix this, I made sure to encode the URI on client side, and decode it on server side. The code (in VB.NET) looks like this:
<AjaxPro.AjaxMethod()> _
Public Function GetSearch(ByVal strValue As String) As String
strValue = HttpUtility.UrlDecode(strValue)
...
End Function
Before the UrlDecode, strValue looks like John%20%20%20Smith'. Afterwards it looks like John Smith. Can anyone tell me how to fix this?
I am using .NET 2.0 Web Forms.
EDIT: following one of the suggestions below I replaced all the spaces with (ampersand)nbsp;. I can see all the spaces when I debug, however, my database is SQL server, for some reason it views regular spaces to be different from (ampersand)nbsp; spaces, and as a result the query does not return the right values. It took me some time to figure this out because I could not see the difference with the naked eye.

Ok, I was able to resolve the issue in the front-end by replacing all spaces with Non-breaking spaces. I then had to replace the Non-breaking spaces back with regular spaces in my SQL stored procedure (otherwise the query wouldn't work).

JavaScript utf8 encoding or regex pattern with Unicode letters support?

I have been recently requested, to adapt an app's input, to support Unicode letters, on some of the inputs within the web app.
That app, already does some validation with regex, with the pattern html attribute. Like so:
<input required="true" pattern="[a-zA-Z0-9_\-]+" type="text" name="name">
Now, since I have to adapt some inputs to the new requirements, I was wondering what would be better to do?
Do Encoding / decoding UTF8 in javascript?
http://ecmanaut.blogspot.ca/2006/07/encoding-decoding-utf8-in-javascript.html
http://laffers.net/blog/2010/12/10/regex-match-unicode-characters/
or
Addapt the regex pattern just like suggested here: PHP Regex for Multiple Unicode Characters ?

Javascript is by definition completely in unicode (Except websites with non unicode encoding, but there the solution may still work), so just add letters you need to regexp. If you need to add them by charcode use \x0000

I had decided to go for editing the regex option, since when my view starts being processed, there is a module that will set the pattern attribute, with the defined regex, on specific inputs.
So, it is better to edit the regex pattern and then set in on inputs when the view is loaded, then doing:
Load view and set pattern attributes for each input
Write js to analyze specific inputs, to see if they contain decoded Unicode characters
If so, encode those characters while typing/before submit, since I have a regex pattern that doesn't allow such characters
Essentially, I'm saving me time on writing useless code, and most important, a lot of browser processing (it was going to be to much, for what it is needed).
I should've gone for this option since the beginning (duh!)

Character Encoding: â?

I am trying to piece together the mysterious string of characters â?? I am seeing quite a bit of in our database - I am fairly sure this is a result of conversion between character encodings, but I am not completely positive.
The users are able to enter text (or cut and paste) into a Ext-Js rich text editor. The data is posted to a severlet which persists it to the database, and when I view it in the database i see those strange characters...
is there any way to decode these back to their original meaning, if I was able to discover the correct encoding - or is there a loss of bits or bytes that has occured through the conversion process?
Users are cutting and pasting from multiple versions of MS Word and PDF. Does the encoding follow where the user copied from?
Thank you
website is UTF-8
We are using ms sql server 2005;
SELECT serverproperty('Collation') -- Server default collation.
Latin1_General_CI_AS
SELECT databasepropertyex('xxxx', 'Collation') -- Database default
SQL_Latin1_General_CP1_CI_AS
and the column:
Column_name Type Computed Length Prec Scale Nullable TrimTrailingBlanks FixedLenNullInSource Collation
text varchar no -1 yes no yes SQL_Latin1_General_CP1_CI_AS
The non-Unicode equivalents of the
nchar, nvarchar, and ntext data types
in SQL Server 2000 are listed below.
When Unicode data is inserted into one
of these non-Unicode data type columns
through a command string (otherwise
known as a "language event"), SQL
Server converts the data to the data
type using the code page associated
with the collation of the column. When
a character cannot be represented on a
code page, it is replaced by a
question mark (?), indicating the data
has been lost. Appearance of
unexpected characters or question
marks in your data indicates your data
has been converted from Unicode to
non-Unicode at some layer, and this
conversion resulted in lost
characters.
So this may be the root cause of the problem... and not an easy one to solve on our end.

â is encoded as 0xE2 in ISO-8859-1 and windows-1252. 0xE2 is also a lead byte for a three-byte sequence in UTF-8. (Specifically, for the range U+2000 to U+2FFF, which includes the windows-1252 characters –—‘’‚“”„†‡•…‰‹›€™).
So it looks like you have text encoded in UTF-8 that's getting misinterpreted as being in windows-1252, and displays as a â followed by two unprintable characters.

This is an something of an educated guess that you're just experiencing a naive conversion of Word/PDF documents to HTML. (windows-1252 to utf8 most likely) If that's the case probably 2/3 of the mysterious characters from Word documents are "smart quotes" and most of the rest are a result of their other "smart" editing features, elipsis, em dashes, etc. PDF's probably have similar features.
I would also guess that if the formatting after pasting into the ExtJS editor looks OK, then the encoding is getting passed along. Depending on the resulting use of the text, you may not need to convert.
If I'm still on base, and we're not talking about internationalization issues, then I can add that there are Word to HTML converters out there, but I don't know the details of how they operate, and I had mixed success when evaluating them. There is almost certainly some small information loss/error involved with such converters, since they need to make guesses about the original source of the "smart" characters. In my isolated case it was easier to just go back to the users and have them turn off the "smart" features.

The issue is clear: if the browser is good enough, a form in a web page can accept any Unicode character you can type or paste. If the character belongs to the HTML charset, it will be sent as is. If it doesn't, it'll get converted to an HTML entity. SQL Server will perform the appropriate conversion and silently corrupt your data when a character does not have an equivalent.
There's not much you can do to fully fix it but you can make a workaround: let your servlet perform the conversion. This way you have full control about it. You can, for instance, compile a list of the most common non-Latin1 characters users paste (smart quotes, unicode spaces...), which should be fairly easy to identify from context, and replace them with something else better that ?. Or you use a library that makes this for you.
Or you can switch your DB to Unicode :)

you're storing unicode data that uses 2 bytes per charcter into a varchar type columns that uses 1 byte per character. any text that uses 2 bytes per chars will have 1 byte lost when stored in the db.
all you need to do is change varchar column to nvarchar.
and then change sql parameters you're using in code of course.