I'm working with giant text files that have more than one
document inside. These documents have a very similar interface, with fixed fields
and dynamic values. I need to separate these documents in arrays.
Example:
[
[] <- Doc1
[] <- Doc2
[] <- Doc3
[] <- Doc4
...
...
...
]
For this, I need to create a regular expression that defines the delimiter, where the doc starts and where ends.
Example:
DOC_START
TEXT
TEXT
TEXT
TEXT
DOC_FINAL
DOC_START
TEXT
TEXT
TEXT
TEXT
DOC_FINAL
REGEX: ((?:DOC_START)(?:[\S\S]+)(?:DOC_FINAL)?)
The question is: Some documents may have peculiarities, starting or ending with a something a bit different, so I need to be able to pass start and end options.
My question: how can I do this? And how can I also improve the regex?
Just to be clear, sometimes, the document may have the beginning or the ending a bit different. Example:
DOC_START
TEXT
TEXT
TEXT
TEXT
DOC_FINAL
DOC_START
TEXT
TEXT
TEXT
TEXT
DOC_FINAL
OTHER_START
TEXT
TEXT
TEXT
TEXT
DOC_FINAL
DOC_START
TEXT
TEXT
TEXT
TEXT
OTHER_FINAL
OTHER_START
TEXT
TEXT
TEXT
TEXT
OTHER_FINAL
It would be better not to use regex, especially with large documents. Use indexOf():
var hugeDoc = 'DOC_STARTxxDOC_ENDOTHER_STARTyyOTHER_END';
var result = [];
var start =0;
var possibleDelimiters = [
{'start': 'OTHER_START', 'end':'OTHER_END'},
{'start': 'DOC_START', 'end':'DOC_END'}
];
function parseDoc(delimiter) {
var end = hugeDoc.indexOf(delimiter.end, start);
if(!end) return false;
result.push(hugeDoc.slice(start+delimiter.start.length, end));
//add +1 here, if you have a new line after DOC_END
start = end+delimiter.end.length;
return true;
}
do {
var found = false;
for(ix in possibleDelimiters) {
var delimiter = possibleDelimiters[ix];
if(hugeDoc.indexOf(delimiter.start, start) === start) {
found = parseDoc(delimiter) || found;
}
}
} while(found);
var node = document.getElementById('result');
node.innerHTML = JSON.stringify(result);
<html>
<body>
<div id="result"></div>
</body>
</html>
First I believe you have a typo in your regex it should be [\s\S] instead of [\S\S] notice the lower-case s. This correctly matches accross lines.
This regex could accomplish what you need for matching such a document, someone could probably make a more optimized version:
/(?:DOC_START|OTHER_START)([\s\S]*?)(?:DOC_FINAL|OTHER_FINAL)/g
On the other hand I would rather suggest you do this with a different approach if possible. For example if you're doing this within NodeJS I'd strongly suggest you do a check per line for the DOC_START or DOC_END delimiters. Then fill the array with lines until the ending delimiter.
Assuming that you want an array of lines in each document, loose pseudo code following:
create resulting object ({ doc1: null })
read line
if start delimiter
if current object property is null
create array (doc#: [])
else if end delimiter
create new doc property (doc2: null)
else
add line to array
Another note if you're doing this with HTML I'd strongly suggest not to use regex at all as HTML is not a regular language :) you'll find many links on SO pointing to evil.
Related
Making a tiny helper for authors to add/remove empty lines in textarea. The one that removes extra lines seems to work:
if(jQuery("#remove_line").prop("checked")){
conver = conver.replace(/^\s*[\r\n]/gm,'');
}
But there has to be a reverse one that makes any new line into double new line (no matter if there's just 1 or even 3 new lines in a row). E.g. this:
text
text
text
text
Should be processed into this:
text
text
text
text
Anyone can help with that? Thanks in advance!
You can match one or more newlines [\r\n]+ and replace with 2 newlines \n\n
const regex = /[\r\n]+/gm;
const str = `text
text
text
text`;
console.log(str.replace(/[\r\n]+/g, `\n\n`));
didn't test but i think this could work : replace(/[\n]+?/g,'\n\n')
\n+ will match 1 or more new lines, the question mark will force to take the shortest amount possible (this will stop the matching block at the first non new line character)
The reverse?
addEventListener('load', ()=>{
const ta = document.querySelector('textarea');
function doubleSpaceAfter(fullText, text, insensitive = true){
const s = insensitive ? 'i' : '';
return fullText.replace(new RegExp('('+text+')\\s+', 'g'+s), '\n\n$1\n\n');
}
ta.value = doubleSpaceAfter(ta.value, 'text here');
}); // end load
<textarea>other stuff text here is this what you want? text here text here</textarea>
I was pointed out to this post, which does not seem to follow the criteria I have:
Replace a Regex capture group with uppercase in Javascript
I am trying to make a regex that will:
format a string by adding uppercase for the first letter of each word and lower case for the rest of the characters
ignore HTML markup
Accept swedish characters (åäöÅÄÖ)
Say I've got this string:
<b>app</b>le store östersund
Then I want it to be (changes marked by uppercase characters)
<b>App</b>le Store Östersund
I've been playing around with it and the closest I've got is the following:
(?!([^<])*?>)[åäöÅÄÖ]|\s\b\w
Resulted in
<b>app</b>le Store Östersund
Or this
/(?!([^<])*?>)[åäöÅÄÖ]|\S\b\w/g
Resulted in
<B>App</B>Le store Östersund
Here's a fiddle:
http://refiddle.com/refiddles/598aabef75622d4a531b0000
Any help or advice is much appreciated.
It is not possible to do this with regexp alone, since regexp doesn't understand HTML structure. [*] Instead, we need to process each text node, and carry through our logic for what is the beginning of the word in case a word continues across different text nodes. A character is at start of the word if it is preceded by a whitespace, or if it is at the start of the string and it is either the first text node, or the previous text node ended in whitespace.
function htmlToTitlecase(html, letters) {
let div = document.createElement('div');
let re = new RegExp("(^|\\s)([" + letters + "])", "gi");
div.innerHTML = html;
let treeWalker = document.createTreeWalker(div, NodeFilter.SHOW_TEXT);
let startOfWord = true;
while (treeWalker.nextNode()) {
let node = treeWalker.currentNode;
node.data = node.data.replace(re, function(match, space, letter) {
if (space || startOfWord) {
return space + letter.toUpperCase();
} else {
return match;
}
});
startOfWord = node.data.match(/\s$/);
}
return div.innerHTML;
}
console.log(htmlToTitlecase("<b>app</b>le store östersund", "a-zåäö"));
// <b>App</b>le Store Östersund
[*] Maybe possible, but even if so, it would be horribly ugly, since it would need to cover an awful amount of corner cases. Also might need a stronger RegExp engine than JavaScript's, like Ruby's or Perl's.
EDIT:
Even if just specifying really simple html tags? The only ones I am actually in need of covering is <b> and </b> at the moment.
This was not specified in the question. The solution is general enough to work for any markup (including simple tags). But...
function simpleHtmlToTitlecaseSwedish(html) {
return html.replace(/(^|\s)(<\/?b>|)([a-zåäö])/gi, function(match, space, tag, letter) {
return space + tag + letter.toUpperCase();
});
}
console.log(simpleHtmlToTitlecaseSwedish("<b>app</b>le store östersund", "a-zåäö"));
I have a solution which use almost only regex. It may be not the most intuitive way to do it, but it should be effective and I find it funny :)
You have to append at the end of your string every lowercase character followed by their uppercase counterpart, like this (it must also be preceded by a space for my regex) :
aAbBcCdDeEfFgGhHiIjJkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZåÅäÄöÖ
(I don't know which letters are missing, I know nothing about swedish alphabet, sorry... I'm counting on you to correct that !)
Then you can use the following regex :
(?![^<]*>)(\s<[^/]*?>|\s|^)([\wåäö])(?=.*\2(.)\S*$)|[\wåÅäÄöÖ]+$
Replace by :
$1$3
Test it here
Here is a working javascript code :
// Initialization
var regex = /(?![^<]*>)(\s<[^/]*?>|\s|^)([\wåäö])(?=.*\2(.)\S*$)|[\wåÅäÄöÖ]+$/g;
var string = "test <b when=\"2>1\">ap<i>p</i></b>le store östersund";
// Processing
result = string + " aAbBcCdDeEfFgGhHiIjJkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZåÅäÄöÖ";
result = result.replace(regex, "$1$3");
// Display result
console.log(result);
Edit : I forgot to handle first word of the string, it's corrected :)
Hoping someone might help. I have a string formatted like the example below:
Lipsum text as part of a paragraph here, yada. |EMBED|{"content":"foo"}|/EMBED|. Yada and the text continues...
What I am looking for is a Javascript RegEx to capture the content between the |EMBED||/EMBED| 'tags', run a function on that content, and then to replace the entire |EMBED|...|/EMBED| string with the return of that function.
The catch is that I may have multiple |EMBED| blocks within a larger string. For example:
Yabba...|EMBED|{"content":"foo"}|/EMBED|. Dabba-do...|EMBED|{"content":"yo"}|/EMBED|.
I need the RegEx to capture and process each |EMBED| block separately, since the content contained within will be similar, but unique.
My initial thought is that I could just have a RegEx that captures the first iteration of the |EMBED| block, and the function which replaces this |EMBED| block is either part of a loop or recursion to continuously find the next block and replace it, until no more blocks are found in the string.
...but this seems expensive. Is there a more eloquent way?
You can use String.prototype.replace to replace a substring found via a regular expression with a modified version of the match using a mapping function, e.g.:
var input = 'Yabba...|EMBED|{"content":"foo"}|/EMBED|. Dabba-do...|EMBED|{"content":"yo"}|/EMBED|.'
var output = input.replace(/\|EMBED\|(.*?)\|\/EMBED\|/g, function(match, p1) {
return p1.toUpperCase()
})
console.log(output) // "Yabba...{"CONTENT":"FOO"}. Dabba-do...{"CONTENT":"YO"}."
Make sure that you use a non-greedy selector .*? to select the content between the delimiters to allow multiple replacements per string.
This is the cod which iterate through the matches of the regex:
var str = 'Lipsum text as part of a paragraph here, yada. |EMBED|{"content":"foo"}|/EMBED|. Yada and the text continues...';
var rx = /\|EMBED\|(.*)\|\/EMBED\|/gi;
var match;
while (true)
{
match = rx.exec(str);
if (!match)
break;
console.log(match[1]); //match[1] is the content between "the tags"
}
I have a strings where some html tags could present, like
this is a nice day for bowling <b>bbbb</b>
how can I replace with RegExp all b symbols, for example, with :blablabla: (for example) but ONLY outside html tags?
So in that case the resulting string should become
this is a nice day for :blablabla:owling <b>bbbb</b>
EDIT: I would like to be more specific, based on the answers I have received. So first of all I have just a string, not DOM element, or anything else. The string may or may not contain tags (opening and closing). The main idea is to be able to replace anywhere in the text except inside tags. For example if I have a string like
not feeling well today :/ check out this link http://example.com
the regexp should replace only first :/ with real smiley image, but should not replace second and third, because they are inside (and part of) tag. Here's an example snippet using the regexp from one of the answer.
var s = 'not feeling well today :/ check out this link http://example.com';
var replaced = s.replace(/(?:<[^\/]*?.*?<\/.*?>)|(:\/)/g, "smiley_image_here");
document.querySelector("pre").textContent = replaced;
<pre></pre>
It is strange but the DEMO shows that it captured the correct group, but the same regexp in replace function seem not to be working.
The regex itself to replace all bs with :blablabla: is not that hard:
.replace(/b/g, ":blablabla:")
It is a bit tricky to get the text nodes where we need to perform search and replace.
Here is a DOM-based example:
function replaceTextOutsideTags(input) {
var doc = document.createDocumentFragment();
var wrapper = document.createElement('myelt');
wrapper.innerHTML = input;
doc.appendChild( wrapper );
return textNodesUnder(doc);
}
function textNodesUnder(el){
var n, walk=document.createTreeWalker(el,NodeFilter.SHOW_TEXT,null,false);
while(n=walk.nextNode())
{
if (n.parentNode.nodeName.toLowerCase() === 'myelt')
n.nodeValue = n.nodeValue.replace(/:\/(?!\/)/g, "smiley_here");
}
return el.firstChild.innerHTML;
}
var s = 'not feeling well today :/ check out this link http://example.com';
console.log(replaceTextOutsideTags(s));
Here, we only modify the text nodes that are direct children of the custom-created element named myelt.
Result:
not feeling well today smiley_here check out this link http://example.com
var input = "this is a nice day for bowling <b>bbbb</b>";
var result = input.replace(/(^|>)([^<]*)(<|$)/g, function(_,a,b,c){
return a
+ b.replace(/b/g, ':blablabla:')
+ c;
});
document.querySelector("pre").textContent = result;
<pre></pre>
You can do this:
var result = input.replace(/(^|>)([^<]*)(<|$)/g, function(_,a,b,c){
return a
+ b.replace(/b/g, ':blablabla:') // you may do something else here
+ c;
});
Note that in most (no all but most) real complex use cases, it's much more convenient to manipulate a parsed DOM rather than just a string. If you're starting with a HTML page, you might use a library (some, like my one, accept regexes to do so).
I think you can use a regex like this : (Just for a simple data not a nested one)
/<[^\/]*?b.*?<\/.*?>|(b)/ig
[Regex Demo]
If you wanna use a regex I can suggest you use below regex to remove all tags recursively until all tags removed:
/<[^\/][^<]*>[^<]*<\/.*?>/g
then use a replace for finding any b.
Any ideas why this simple RegEx doesn't seem to be supported in a Google Docs script?
foo(?!bar)
I'm assuming that Google Apps Script uses the same RegEx as JavaScript. Is this not so?
I'm using the RegEx as such:
DocumentApp.getActiveDocument().getBody().replaceText('foo(?!bar)', 'hello');
This generates the error:
ScriptError: Invalid regular expression pattern foo(?!bar)
As discussed in comments on this question, this is a documented limitation; the replaceText() method doesn't support reverse-lookaheads or any other capture group.
A subset of the JavaScript regular expression features are not fully supported, such as capture groups and mode modifiers.ref
Serge suggested a work-around, "it should be possible to manipulate your document at a lower level (extracting text from paragraph etc) but it could rapidly become quite cumbersome."
Here's what that could look like. If you don't mind losing all formatting, this example will apply capture groups, RegExp flags (i for case-insensitivity) and reverse-lookaheads to change:
Little rabbit Foo Foo, running through the foobar.
to:
Little rabbit Fred Fred, running through the foobar.
Code:
function myFunction() {
var body = DocumentApp.getActiveDocument().getBody();
var paragraphs = body.getParagraphs();
for (var i=0; i<paragraphs.length; i++) {
var text = paragraphs[i].getText();
paragraphs[i].replaceText(".*", text.replace(/(f)oo(?!bar)/gi, '$1red') );
}
}
You have a sequence which you can match with a regular expression, but that regular expression will also match one, or more, things which you do not desire to change. The generalized solution to this situation is to:
Change the text such that you have known sequences of characters which are definitely not used. Effectively, this gives you sequences of characters which you use as variables to hold the values you don't want to change. Personally, I would use:
body.replaceText('Q','Qz');
Which will make it such that there is no sequence in your document which matches /Q[^z]/. This results in you being able to use sequences like Qa to represent some text you don't want to change. I use Q because it has a low frequency of use in English. You can use any character. For efficiency, choose a character which results in a low number of changes within the text you are affecting.
Change the things you don't want to end up changing to one of the character sequences you now know are unused. For example:
body.replaceText('foobar','Qa');
Repeat this for whatever additional items you don't want to end up changing.
Change the text you are really wanting to change. In this example:
body.replaceText('foo','hello'.replace(/Q/g,'Qz'));
Note that you need to apply to the new replacement text the first substitution which you used to open up known unused sequences.
Restore all of the things you did not want to change to their original state:
body.replaceText('Qa','foobar');
Restore the text you used to open up unused character sequences:
body.replaceText('Qz','Q');
All together that would be:
var body = DocumentApp.getActiveDocument().getBody();
body.replaceText('Q','Qz'); //Open up unused character sequences
body.replaceText('foobar','Qa'); //Save the things you don't want to change.
//In the general case, you need to apply to the new text the same substitution
// which you used to open up unused character sequences. If you don't you
// may end up with those sequences being changed in the new text.
body.replaceText('foo','hello'.replace(/Q/g,'Qz')); //Make the change you desire.
body.replaceText('Qa','foobar'); //Restore the things you saved.
body.replaceText('Qz','Q'); //Restore the original sequence.
While solving the problem this way does not allow you to use all the features of JavaScript RegExp (e.g. capture groups, look-ahead assertions, and flags), it should preserve the formatting within your document.
You can choose not to perform steps 1 and 5 above by picking a longer sequence of characters to use to represent the text which you do not want to match (e.g. kNoWn1UnUsEd). However, such a longer sequence is something that must be selected based on your knowledge of what already exists in the document. Doing that can save a couple of steps, but you either have to search for an unused string or accept that there is some probability that the string you use is already in the document, which would result in an undesired substitution.
I figured out a way to obtain most of JS's str.replace() functionalities including capture groups and smart replacers in Apps Script without messing up the style. The trick is to use Javascript's regex.exec() function and Apps Script's text.deleteText() and text.insertText() functions.
function replaceText(body, regex, replacer, attribute){
var content = body.getText();
const text = body.editAsText();
var match = "";
while (true){
content = body.getText();
var oldLength = content.length;
match = regex.exec(content);
if (match === null){
break;
}
var start = match.index;
var end = regex.lastIndex - 1;
text.deleteText(start, end);
text.insertText(start, replacer(match, regex));
var newLength = body.getText().length;
var replacedLength = oldLength - newLength;
var newEnd = end - replacedLength;
text.setAttributes(start, newEnd, attribute);
regex.lastIndex -= replacedLength;
}
}
Argument explanations:
body: the body of the document you want to operate on
regex: the normal JS regular expression object used as a search pattern
replacer: the replacer function used to return the string you want to replace with, replacer automatically receive two arguments:
I. match: match object generated by regex.exec() and
II. regex: the regular expression object used as a search pattern
attribute: An Apps Script attribute object
For example, if you want to apply bold style to new strings replacing the old ones, you can create a boldStyle attribute object:
var boldStyle = {};
boldStyle[DocumentApp.Attribute.BOLD] = true;
Tips:
How can I use capture groups in replaceText()?
You can access all capture groups from the replacer function, match[0] is the whole string matched, match[1] is the first capture group, match[2] is the second, etc.
How can I access the index and position of the match in replaceText()?
You can access the start index of the match (match.index) and end index of the match (regex.lastIndex) from the replacer function.
For more in-depth reference of JS RegExp, see this excellent tutorial from Javascript.info.
Example:
Here's a example use case of the replaceText() function. It's simple implementation of a markdown to google docs conversion script:
function markdownToDocs() {
const body = DocumentApp.getActiveDocument().getBody();
// Use editAsText to obtain a single text element containing
// all the characters in the document.
const text = body.editAsText();
// e.g. replace "**string**" with "string" (bolded)
var boldStyle = {};
boldStyle[DocumentApp.Attribute.BOLD] = true;
replaceDeliminaters(body, "\\*\\*", boldStyle, false);
// e.g. replace multiline "```line 1\nline 2\nline 3```" with "line 1\nline 2\nline 3" (with gray background highlight)
var blockHighlightStyle = {};
blockHighlightStyle[DocumentApp.Attribute.BACKGROUND_COLOR] = "#EEEEEE";
replaceDeliminaters(body, "```", blockHighlightStyle, true);
// e.g. replace inline "`console.log("hello world")`" with "console.log("hello world")" (in "Times New Roman" font and italic)
var inlineStyle = {};
inlineStyle[DocumentApp.Attribute.FONT_FAMILY] = "Times New Roman";
inlineStyle[DocumentApp.Attribute.ITALIC] = true;
replaceDeliminaters(body, "`", inlineStyle, false);
// feel free to change all the styling and markdown deliminaters as you wish.
}
// replace markdown deliminaters like "**", "`", and "```"
function replaceDeliminaters(body, deliminator, attributes, multiline){
var capture;
if (multiline){
capture = "([\\s\\S]+?)"; // capture newline characters as well
} else{
capture = "(.+?)"; // do not capture newline characters
}
const regex = new RegExp(deliminator + capture + deliminator, "g");
const replacer = function(match, regex){
return match[1]; // return the first capture group
}
replaceText(body, regex, replacer, attributes);
}