Develop Reference

JS (no lookback) regex for replacing :bound SQL vars without replacing 'colons:in:literals' ...?

Messing around with node-mysql, I wrote some code that lets me use PDO-style :bound values (plus ::bound field names), and rewrites the query with ? and ?? respectively where they are found, and builds a linear array of the values when I execute the statement. I did this because when I look at a SQL statement with a ton of ? ?? all over it and have to count the number of params in my execution, it makes my eyes bleed. I want to just assign a standard object at execution time.
The trouble is, after writing this (it works) I realized my regex for finding those colons in the statement had one tiny little problem, namely, it looks like this:
/.?:(\w+)/g
It picks up the first colon if needed and we take it from there. The problem is, it also picks up colons in literals within the query. So if for some reason you wanted a non-bound string as part of your insert/update, it would be replaced by this engine.
Is there any standard regex for picking up every global instance of the word ":param{#}" in the following statement, without picking up the word "Hello:world", in JS, without lookbacks?
INSERT INTO test VALUES(:param1, :param2, 'Hello:world', :param3);

You're often much better off writing a parser than using regular expressions. It's much more flexible, gives you better error reporting and allows you to handle current & future edge cases much more easily.
The string parsing deals with MySQL string literals syntax & escape sequences described here and just skips over them.
I'm not dealing with valid/invalid binding boundaries, but you could add that if you wanted. You could also remove error reporting such as underterminated string literals and just be forgiving.
The lookahead === ':' && peek() !== '=' condition is to ignore the := MySQL operator.
const parseBindings = (() => {
const bindingCharRx = /\w/;
return function(sql) {
const bindings = [];
let i = 0,
lookahead = sql[i];
while (lookahead) {
if (isStringDelim(lookahead)) parseString();
else if (lookahead === ':' && peek() !== '=') parseBinding();
else consume();
}
return bindings;
function parseString() {
const start = i,
delim = lookahead;
consume();
while (lookahead) {
if (lookahead === '\\') {
consume();
consume();
continue;
}
if (lookahead === delim) {
consume();
if (lookahead !== delim) return;
}
consume();
}
throw new Error(`Underterminated string literal starting at index ${start}.`);
}
function isStringDelim(char) {
return char === "'" || char === '"';
}
function parseBinding() {
const start = i;
consume();
while (lookahead && bindingCharRx.test(lookahead)) consume();
const name = sql.slice(start + 1, i);
if (!name.length) {
throw new Error(`Invalid binding starting at index ${start}.`);
}
bindings.push({
start,
end: i,
name: name
});
}
function consume() {
lookahead = sql[++i]
}
function peek() {
return sql[i + 1]
}
}
})();
function replaceNamedBindings(values, sql) {
const bindings = parseBindings(sql);
const bindingNames = new Set(bindings.map(b => b.name));
const unknownBinding = Object.keys(values).find(k => !bindingNames.has(k));
if (unknownBinding) throw new Error(`Couldn't find a binding named '${unknownBinding}'.`);
let lastIndex = 0,
newSql = '';
for (const binding of bindings) {
if (binding.name in values) {
newSql += sql.slice(lastIndex, binding.start) + values[binding.name];
lastIndex = binding.end;
}
}
newSql += sql.slice(lastIndex);
return newSql;
}
const sql = `INSERT INTO test VALUES(:param1, :param2, 'Hello:world', :param3);`;
console.log(replaceNamedBindings({
param1: '(param1 value)',
param2: '(param2 value)',
param3: '(param3 value)'
}, sql));
console.log(parseBindings(sql));
console.log(parseBindings(`:pickup1 ":dontpickup1" ':dontpickup2' := """:dontpickup3" ''':dontpickup4' "\\":dontpickup5" :pickup2`));
//Will throw exception b/c :world is not a binding
console.log(replaceNamedBindings({
world: '(world value)'
}, sql));

How can I compare "M" and "Ｍ" (in UTF) using Javascript?

I have a situation where I have to search a grid if it contains a certain substring. I have a search bar where the user can type the string. The problem is that the grid contains mix of Japanese text and Unicode characters,
for example : ＭＡＧシンチ注　３３３ＭＢｑ .
How can I compare for content equality the letter 'M' that I type from the keyboard and the letter "Ｍ" as in the example above? I am trying to do this using plain Javascript and not Jquery or other library. And I have to do this in Internet Explorer.
Thanks,

As mentioned in an insightful comment from #Rhymoid on the question, modern JavaScript (ES2015) includes support for normalization of Unicode. One mode of normalization is to map "compatible" letterforms from higher code pages down to their most basic representatives in lower code pages (to summarize, it's kind-of involved). The .normalize("NFKD") method will map the "M" from the Japanese code page down to the Latin equivalent. Thus
"ＭＡＧシンチ注　３３３ＭＢｑ".normalize("NFKD")
will give
"MAGシンチ注 333MBq"
As of late 2016, .normalize() isn't supported by IE.
At a lower level, ES2015 also has .codePointAt() (mentioned in another good answer), which is like the older .charCodeAt() described below but which also understands UTF-16 pairs. However, .codePointAt() is (again, late 2016) not supported by Safari.
below is original answer for older browsers
You can use the .charCodeAt() method to examine the UTF-16 character codes in the string.
"M".charCodeAt(0)
is 77, while
"Ｍ".charCodeAt(0)
is 65325.
This approach is complicated by the fact that for some Unicode characters, the UTF-16 representation involves two separate character positions in the JavaScript string. The language does not provide native support for dealing with that, so you have to do it yourself. A character code between 55926 and 57343 (D800 and DFFF hex) indicates the start of a two-character pair. The UTF-16 Wikipedia page has more information, and there are various other sources.

Building a dictionary should work in any browser, find the charCodes at the start of ranges you want to transform then move the characters in your favourite way, for example
function shift65248(str) {
var dict = {}, characters = [],
character, i;
for (i = 0; i < 10; ++i) { // 0 - 9
character = String.fromCharCode(65296 + i);
dict[character] = String.fromCharCode(48 + i);
characters.push(character);
}
for (i = 0; i < 26; ++i) { // A - Z
character = String.fromCharCode(65313 + i);
dict[character] = String.fromCharCode(65 + i);
characters.push(character);
}
for (i = 0; i < 26; ++i) { // a - z
character = String.fromCharCode(65313 + i);
dict[character] = String.fromCharCode(97 + i);
characters.push(character);
}
return str.replace(
new RegExp(characters.join('|'), 'g'),
function (m) {return dict[m];}
);
}
shift65248('ＭＡＧシンチ注　３３３ＭＢｑ'); // "MAGシンチ注　333MBｑ"
I tried just moving the whole range 65248..65375 onto 0..127 but it conflicted with the other characters :(

I am assuming that you have access to those strings, by reading the DOM for some other way.
If so, codePointAt will be your friend.
console.log("Test of values");
console.log("M".codePointAt(0));
console.log("Ｍ".codePointAt(0));
console.log("Determining end of string");
console.log("Ｍ".codePointAt(10));
var str = "ＭＡＧシンチ注　３３３ＭＢｑ .";
var idx = 0;
do {
point = str.codePointAt(idx);
idx++;
console.log(point);
} while(point !== undefined);

You could try building your own dictionary and compare function as follows:
var compareDB = {
'm' : ['Ｍ'],
'b' : ['Ｂ']
};
function doCompare(inputChar, searchText){
inputCharLower = inputChar.toLowerCase();
searchTextLower = searchText.toLowerCase();
if(searchTextLower.indexOf(inputChar) > -1)
return true;
if(compareDB[inputCharLower] !== undefined)
{
for(i=0; i<compareDB[inputCharLower].length; i++){
if(searchTextLower.indexOf(compareDB[inputCharLower][i].toLowerCase()) > -1)
return true;
}
}
return false;
}
console.log("searching with m");
console.log(doCompare('m', "searching text with Ｍ"));
console.log("searching with m");
console.log(doCompare('m', "searching text with Ｂ"));
console.log("searching with B");
console.log(doCompare('B', "searching text with Ｂ"));

Split a CSV string by line skipping newlines contained between quotes

If the following regex can split a csv string by line.
var lines = csv.split(/\r|\r?\n/g);
How could this be adapted to skip newline chars that are contained within a CSV value (Ie between quotes/double-quotes)?
Example:
2,"Evans & Sutherland","230-132-111AA",,"Visual","P
CB",,1,"Offsite",
If you don't see it, here's a version with the newlines visible:
2,"Evans & Sutherland","230-132-111AA",,"Visual","P\r\nCB",,1,"Offsite",\r\n
The part I'm trying to skip over is the newline contained in the middle of the "PCB" entry.
Update:
I probably should've mentioned this before but this is a part of a dedicated CSV parsing library called jquery-csv. To provide a better context I have added the current parser implementation below.
Here's the code for validating and parsing an entry (ie one line):
$.csvEntry2Array = function(csv, meta) {
var meta = (meta !== undefined ? meta : {});
var separator = 'separator' in meta ? meta.separator : $.csvDefaults.separator;
var delimiter = 'delimiter' in meta ? meta.delimiter : $.csvDefaults.delimiter;
// build the CSV validator regex
var reValid = /^\s*(?:D[^D\\]*(?:\\[\S\s][^D\\]*)*D|[^SD\s\\]*(?:\s+[^SD\s\\]+)*)\s*(?:S\s*(?:D[^D\\]*(?:\\[\S\s][^D\\]*)*D|[^SD\s\\]*(?:\s+[^SD\s\\]+)*)\s*)*$/;
reValid = RegExp(reValid.source.replace(/S/g, separator));
reValid = RegExp(reValid.source.replace(/D/g, delimiter));
// build the CSV line parser regex
var reValue = /(?!\s*$)\s*(?:D([^D\\]*(?:\\[\S\s][^D\\]*)*)D|([^SD\s\\]*(?:\s+[^SD\s\\]+)*))\s*(?:S|$)/g;
reValue = RegExp(reValue.source.replace(/S/g, separator), 'g');
reValue = RegExp(reValue.source.replace(/D/g, delimiter), 'g');
// Return NULL if input string is not well formed CSV string.
if (!reValid.test(csv)) {
return null;
}
// "Walk" the string using replace with callback.
var output = [];
csv.replace(reValue, function(m0, m1, m2) {
// Remove backslash from any delimiters in the value
if (m1 !== undefined) {
var reDelimiterUnescape = /\\D/g;
reDelimiterUnescape = RegExp(reDelimiterUnescape.source.replace(/D/, delimiter), 'g');
output.push(m1.replace(reDelimiterUnescape, delimiter));
} else if (m2 !== undefined) {
output.push(m2);
}
return '';
});
// Handle special case of empty last value.
var reEmptyLast = /S\s*$/;
reEmptyLast = RegExp(reEmptyLast.source.replace(/S/, separator));
if (reEmptyLast.test(csv)) {
output.push('');
}
return output;
};
Note: I haven't tested yet but I think I could probably incorporate the last match into the main split/callback.
This is the code that does the split-by-line part:
$.csv2Array = function(csv, meta) {
var meta = (meta !== undefined ? meta : {});
var separator = 'separator' in meta ? meta.separator : $.csvDefaults.separator;
var delimiter = 'delimiter' in meta ? meta.delimiter : $.csvDefaults.delimiter;
var skip = 'skip' in meta ? meta.skip : $.csvDefaults.skip;
// process by line
var lines = csv.split(/\r\n|\r|\n/g);
var output = [];
for(var i in lines) {
if(i < skip) {
continue;
}
// process each value
var line = $.csvEntry2Array(lines[i], {
delimiter: delimiter,
separator: separator
});
output.push(line);
}
return output;
};
For a breakdown on how that reges works take a look at this answer. Mine is a slightly adapted version. I consolidated the single and double quote matching to match just one text delimiter and made the delimiter/separators dynamic. It does a great job of validating entiries but the line-splitting solution I added on top is pretty frail and breaks on the edge case I described above.
I'm just looking for a solution that walks the string extracting valid entries (to pass on to the entry parser) or fails on bad data returning an error indicating the line the parsing failed on.
Update:
splitLines: function(csv, delimiter) {
var state = 0;
var value = "";
var line = "";
var lines = [];
function endOfRow() {
lines.push(value);
value = "";
state = 0;
};
csv.replace(/(\"|,|\n|\r|[^\",\r\n]+)/gm, function (m0){
switch (state) {
// the start of an entry
case 0:
if (m0 === "\"") {
state = 1;
} else if (m0 === "\n") {
endOfRow();
} else if (/^\r$/.test(m0)) {
// carriage returns are ignored
} else {
value += m0;
state = 3;
}
break;
// delimited input
case 1:
if (m0 === "\"") {
state = 2;
} else {
value += m0;
state = 1;
}
break;
// delimiter found in delimited input
case 2:
// is the delimiter escaped?
if (m0 === "\"" && value.substr(value.length - 1) === "\"") {
value += m0;
state = 1;
} else if (m0 === ",") {
value += m0;
state = 0;
} else if (m0 === "\n") {
endOfRow();
} else if (m0 === "\r") {
// Ignore
} else {
throw new Error("Illegal state");
}
break;
// un-delimited input
case 3:
if (m0 === ",") {
value += m0;
state = 0;
} else if (m0 === "\"") {
throw new Error("Unquoted delimiter found");
} else if (m0 === "\n") {
endOfRow();
} else if (m0 === "\r") {
// Ignore
} else {
throw new Error("Illegal data");
}
break;
default:
throw new Error("Unknown state");
}
return "";
});
if (state != 0) {
endOfRow();
}
return lines;
}
All it took is 4 states for a line splitter:
0: the start of an entry
1: the following is quoted
2: a second quote has been encountered
3: the following isn't quoted
It's almost a complete parser. For my use case, I just wanted a line splitter so I could provide a more granual approach to processing CSV data.
Note: Credit for this approach goes to another dev whom I won't name publicly without his permission. All I did was adapt it from a complete parser to a line-splitter.
Update:
Discovered a few broken edge cases in the previous lineSplitter implementation. The one provided should be fully RFC 4180 compliant.

As I have noted in a comment there is no complete solution just using single regex.
A novel method using several regexps by splitting on comma and joining back strings with embedded commas is described here:-
Personally I would use a simple finite state machine as described here
The state machine has more code, but the code is cleaner and its clear what each piece of code is doing. Longer term this will be much more reliable and maintainable.

It's not a good idea to use regex's to parse. Better to use it to detect the "bad" splits and then merge them back:
var lines = csv.split(/\r?\n/g);
var bad = [];
for(var i=lines.length-1; i> 0; i--) {
// find all the unescaped quotes on the line:
var m = lines[i].match(/[^\\]?\"/g);
// if there are an odd number of them, this line, and the line after it is bad:
if((m ? m.length : 0) % 2 == 1) { bad.push(i--); }
}
// starting at the bottom of the list, merge lines back, using \r\n
for(var b=0,len=bad.length; b < len; b++) {
lines.splice(bad[b]-1, 2, lines[bad[b]-1]+"\r\n"+lines[bad[b]]);
}
(This answer is licensed under both CC0 and WTFPL.)

Be careful- That newline is PART of that value. It's not PCB, it's P\nCB.
However, why can't you just use string.split(',')? If need be, you can run through the list and cast to ints or remove the padded quotation marks.

JavaScript CSV Parser Library [closed]

Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
Is there a decent CSV Parser library for JavaScript? I've used this and that solution so far. In the first solution a new line is never created as a new sub-array, also the code tells so and the second solution does not work on text files formatted in Windows with <CR><LF> , respectively \r\n
Is it sufficient to apply
text = text.replace("\r","");
to the Windows CSV files? This actually works, but I think this is a little bit quirks. Are there csv parser which are more common than a random bloggers solution?

Here's the 'easy' solution
csv.split(/\r\n|\r|\n/g)
It handles:
\n
\r
\r\n
\n\r
Unfortunately, it breaks on values that contain newline chars between delimiters.
For example, the following line entry...
"this is some","valid CSV data","with a \r\nnewline char"
Will break it because the '\r\n' will be mistakenly interpreted as the end of an entry.
For a complete solution, your best bet is to create a ND-FSM (Non-Deterministic Finite State Machine) lexer/parser. If you have ever heard of the Chomsky Hierarchy, CSV can be parsed as a Type III grammar. That means char-by-char or token-by-token processing with state tracking.
I have a fully RFC 4180 compliant client-side library available but somehow I attracted the attention of a delete-happy mod for external linking. There's a link in my profile if you're interested; otherwise, good luck.
I'll give you fair warning from experience, CSV looks deceptively easy on the surface. After studying tens/hundreds of implementations, I have only seen 3 javascript parsers that did a reasonable job of meeting the spec and none of them were completely RFC compliant. I managed to write one but only with the help of the community and lots and lots of pain.

If you're working in Node, there's an excellent CSV parser that can handle extremely large amounts of data (>GB files) and supports escape characters.
If you're working in browser JS, you could still extract the processing logic from the code so that it operates on a string (instead of a Node Stream).

Here is one way to do it:
// based on json_parse from JavaScript The Good Part by D. Crockford
var csv_parse = function () {
var at,
ch,
text,
error = function (m) {
throw {
name: 'SyntaxError',
message: m,
at: at,
text: text
};
},
next = function (c) {
if (c && c !== ch) {
error("Expected '" + c + "' instead of '" + ch + "'");
}
ch = text.charAt(at);
at += 1;
return ch;
},
//needed to handle "" which indicates escaped quote
peek = function () {
return text.charAt(at);
},
white = function () {
while (ch && ch <= ' ' && ch !== '\n') {
next();
}
},
// if numeric, then return number
number = function () {
var number,
string = word();
number = +string;
if (isNaN(number)) {
return string;
} else {
return number;
}
},
word = function () {
var string = '';
while (ch !== ',' && ch !== '\n') {
string += ch;
next();
}
return string;
},
// the matching " is the end of word not ,
// need to worry about "", which is escaped quote
quoted = function () {
var string ='';
if (ch === '"') {
while (next()) {
if (ch === '"') {
//print('need to know ending quote or escaped quote');
// need to know ending quote or escaped quote ("")
if (peek() === '"') {
//print('maybe double quote near '+string);
next('"');
string += ch;
} else {
next('"')
return string;
}
} else {
string += ch;
}
}
return string;
}
error("Bad string");
},
value = function () {
white();
switch(ch) {
case '-':
return number();
case '"':
return quoted();
default:
return ch >= '0' && ch <= '9' ? number() : word();
}
return number();
},
line = function () {
var array = [];
white();
if (ch === '\n') {
next('\n');
return array;//empty []
}
while (ch) {
array.push( value() );
white();
if (ch === '\n') {
next('\n');
return array;//got something
}
next(',');// not very liberal with delimiter
white();
}
};
return function (_line) {
var result;
text = _line;
at = 0;
ch = ' ';
result = line();
white();
if (ch) {
error("Syntax error");
}
return result;
};
}();

My function is solid, just drop in and use, I hope it is of help to you.
csvToArray v1.3
A compact (508 bytes) but compliant function to convert a CSV string into a 2D array, conforming to the RFC4180 standard.
http://code.google.com/p/csv-to-array/
Common Usage: jQuery
$.ajax({
url: "test.csv",
dataType: 'text',
cache: false
}).done(function(csvAsString){
csvAsArray=csvAsString.csvToArray();
});
Common usage: Javascript
csvAsArray = csvAsString.csvToArray();
Override field separator
csvAsArray = csvAsString.csvToArray("|");
Override record separator
csvAsArray = csvAsString.csvToArray("", "#");
Override Skip Header
csvAsArray = csvAsString.csvToArray("", "", 1);
Override all
csvAsArray = csvAsString.csvToArray("|", "#", 1);

problem in fetching a particular cookie

This is the script that i am using to fetch a particular cookie lastvisit :
AFTER THE EDIT
// This document writes a cookie
// called from index.php
window.onload = makeLastVisitCookie;
function makeLastVisitCookie() {
var now = new Date();
var last = new Date();
now.setFullYear(2020);
// set the cookie
document.cookie = "lastvisit=" + last.toDateString() + ";path=/;expires=" + now.toGMTString();
var allCookies = document.cookie.split(";");
for( var i=0 ; i < allCookies.length ; i++ ) {
if(allCookies[i].split("=")[0]== "lastvisit") {
document.getElementById("last_visit").innerHTML = "You visited this site on" + allCookies[i].split("=")[1];
} else {
alert("testing..testing..");
}
}
}
From this script the if part never works though there are 5 cookies stored from my website. (including the cookie that i am saving from this script) What is the mistake that i am making while fetching the cookie named lastvisit ?

You're splitting the cookie by ; an comparing those tokens with lastvisit. You need to split such a token by = first. allCookies[i] looks like key=val and will never equal lastvisit. Een if allCookies[i] == "lastvisit" is true, the result will still not be as expected since you're showing the value of allCookies[i + 1] which would be this=the_cookie_after_lastvisit.
if(allCookies[i].split("=") == "lastvisit") { should be:
var pair = allCookies[i].split("=", 2);
if (pair[0].replace(/^ +/, "") == "lastvisit") {
"You visited this site on" + allCookies[i+1]; should be:
"You visited this site on" + pair[1];
The 2 argument of split makes cookies like sum=1+1=2 be read correctly. When splitting cookies by ;, the key may contain a leading space which much be removed before comparing. (/^ +/ is a regular expression where ^ matches the beginning of a string and + one or more spaces.)
Alternatively, compare it directly against a RE for matching the optional spaces as well (* matches zero or more occurences of a space character, $ matches the end of a string):
if (/^ *lastvisit$/.test(pair[0])) {
I've tested several ways to get a cookie including using regular expressions and the below was the most correct one with best performance:
function getCookie(name) {
var cookie = "; " + document.cookie + ";";
var search = "; " + encodeURIComponent(name) + "=";
var value_start = cookie.indexOf(search);
if (value_start == -1) return "";
value_start += search.length;
var value_end = cookie.indexOf(';', value_start);
return decodeURIComponent(cookie.substring(value_start, value_end))
}

You need to remove possible white space around the cookie key before comparing to the string "lastvisit". This is done conveniently using regular expressions. /^\s+/ matches all white space at the beginning, /\s+$/ matches all white space at the end. The matches are replaced by the empty string, i.e. removed:
for( var i = 0 ; i < allCookies.length ; i++ ) {
var c = allCookies[i].split("="); // split only once
var key = c[0].replace(/^\s+/, '').replace (/\s+$/, ''); // remove blanks around key
if (key == "lastvisit") {
document.getElementById("last_visit").innerHTML = "You visited on " + c[1];
}
//...
}