Develop Reference

Retry until response.results[-1].trans.id.includes('TransactionID')

Apologies if this has been answered before, I spent most of today looking and found some similar issues but nothing that answered my specific problem.
I am posting to a TransactionAPI with 'TransactionID'-> sleeping thread for 10 minutes (because this is ensures that the transaction posts in the next step) -> calling read api with
Then match response.results[*].trans.id contains TransactionID
this currently works, but sleep isn't great.. and I wanted to take full advantage of karate.
The [*] is because the new TransactionID appears in the final index of the returned array response.
so the first time it calls the readAPI it gets an array of [5], and the new TransactionID will appear in [6] (and I couldn't figure out how to make it get array.length + 1 and wait for that one to appear)
Reading the responses here Karate framework retry until not working as expected and the linked ones inside i tried a few things:
And retry until response.results[-1].trans.id.includes('TransactionID')
And retry until response.results[(#.length-1)].trans.id.includes('TransactionID')
These return an error:
org.graalvm.polyglot.PolyglotException: SyntaxError: Unnamed:1:18 Expected an operand but found error
I tried the in-line javascript function mentioned in the above link, but could not get it to work either.
Javascript function mentioned:
* def isValid = function(x){ return karate.match(x, { tokens: '##[_ > 0]' }).pass }
# ...
And retry until isValid(response)

So indexing to -1 will not work because that is JsonPath and not JavaScript.
Try this:
# set this by doing a call before if needed
* def prevLength = response.length
* def isValid = function(x){ return x.length > prevLength }
# ...
* retry until isValid(response)
My guess is that it is more than enough to achieve what you want.

NodeJS Schema virtual function if clause malfunctioning

I am getting a false truth from this script, and it's baffling. Using virtuals in nodejs. 'XFactor' is essentially a scale from 0.0 to 1.0 which represents the relationship between variable 1 and variable 2.
Schema.virtual('XFactor').get(function() {
if (this.variable1 > this.variable2) {
var x = 1/2 * (this.variable2 / this.variable1);
} else {
var x = 1- 1 / 2 * this.variable1 / this.variable2;
}
return x.toFixed(2);
});
For a while this code has been working. Yet it is unreliable for some reason. The if clause if (this.variable1 > this.variable2) seems to sometimes evaluate as true even when it isn't, and thereby the wrong portion of code is run. In case it is relevant, Variable 1 and Variable 2 are also virtuals, calculated in preceding sections of code. They are integers.

I'm delighted to be able to answer my own question! For some reason, the other virtuals I was using in this script, despite being numbers, were not numbers according to Javascript. To carry out accurate numerical functions, therefore, I had to convert them. I did this using the Number() function. Once this was done then everything worked as expected.

Get a return value from arbitrary eval'd code

I have a requirement that the user can provide arbitrary statements which can be stored in a function and called later to get a return value. A simple example of this is that userInput might be
var x = 10;
x;
I would store this via
var callback = function() {
return eval(userInput);
}
and then running callback() returns 10 as expected.
However, I also need to support the case with an explicit return statement, ie userInput might be
var x = 10;
return x;
In this case the eval method above will fail with SyntaxError: return not in function. Instead I could store callback as
var callback = new Function(userInput);
My issue is that I would like to combine these two approaches according the rule 'get explicit return value otherwise get the result of the last executed statement'. In particular this can't be done with analysis of the code at callback creation time as the user could potentially do something odd like
if(envVar < 10)
return a;
b * 0.5;
which combines both.
Any thoughts on how to structure the creation of the callback function to accommodate these possible inputs? Unfortunately it is not possible to enforce one style or the other on the user.
UPDATE to answer some of the comments below.
One solution suggested is to search for a return token and choose between new Function and eval. This doesn't work for my last example, see http://jsfiddle.net/ZGb6z/2/ - out4 should be "no" but ends up being undefined as the last executed statement is not returned.
Another suggestion is to modify the user input to add an explicit return on the last line if one is not found. Unfortunately it's not possible to know which statement will be executed last. Consider
var x = 10;
switch(x) {
case 10:
100;
break;
default:
200;
break;
}
When called it should return 100, but it would take some serious analysis to determine where to put returns for arbitrary code.

Just use a try catch, manipulating the input will be very painful for you, and try catch can't really make your code any more obtuse at this point.
var failback = function () {
try {
return eval(userInput);
} catch (e) {
return Function(userInput);
}
};
What I would really suggest is investing in a parser, kind of like how Angular does it. That kind of thing would prevent your users from doing whatever the hell they want, introducing attack vectors, yadda, yadda, yadda.

Either manage your expectations or manage your user's expectations. eval and new Function() are not suitable for your requirements if you require mixed usage of explicit and non-explicit return statements in the same user-input. You will continue to find issues following either of these routes.
Simply searching for the word return is not sufficient either... var returning = true; or var a = 'return'; or /* return true */ true; will all throw false positives.
Managing your expectations: To do such a thing you require a form of lexer and parser, at which point you can do away with eval entirely and execute your own safe functions based on the parsed input. This is the best approach when execution of user input has to occur anyway as you can ensure that nothing gets executed you do not wish to permit. If you want to cover these sort of edge cases and permit strange user input then you must be prepared to increase the size and development time of your application. I have built a few applications executing user generated code and have always come to the conclusion this is the correct route to go down.
Managing your user's expectations: Provide a guide, tell them not to mix explicit returns with non-explicit returns, these are strange coding practices anyway. Better yet explicitly tell them to include or omit the return statement. There is no shame in asking your users to follow them, especially if it allows you to improve their experience elsewhere.

There I was thinking I'd only see problems like this at the code golf stack exchange :)
My solution is here: http://jsfiddle.net/hKq87/1
It essentially replaces the 'return' statement with an exception that has a special string prefixed to it. If we see that string, we know we are actually returning a value, and return it rather than re-raising the exception.
The reason I chose to throw an exception rather than replace the return statement with a function call was because it is hard to know where the JS code evaluated for the return really ends. It could be split across multiple lines, contain several special characters and may not even have the optional semi-colon at the end. So I concatenate a string to whatever the value being returned is and throw it, as the throw keyword doesn't require it's argument to be wrapped in parentheses.
In addition, throwing exceptions provides me a convenient way to immediately terminate execution of the code block, without halting other JS execution.
Here is the callback method:
var callback = function(userInput) {
var returned = undefined;
userInput = userInput.replace(/(^|[\(\\)[\]{};,\s])return(\s*[^\s;])?/gi, function(m, b, e){
return b + " throw '__RETURNED_VALUE'" +
(e !== undefined ? "+" + e : "");
});
try {
returned = eval(userInput);
} catch (e) {
if (e.indexOf("__RETURNED_VALUE") == 0) {
returned = e.substring("__RETURNED_VALUE".length) || undefined;
}
else {
throw e;
}
}
return returned;
}
The regex above accounts for variables that may end with the string "return", that we would not want to replace as it is not a return statement. It also allows for return statements within braces, without trailing semi-colons or at the very beginning/end.
One issue with the current method is that you can not use the (rare) comma operator in a return statement, or expect numerical values to be returned correctly. The last test case in the jsfiddle demonstrates this. An example from here:
//original
return 5 * 2 + 3, 22;
//modified
throw '__RETURNED_VALUE='+ 5 * 2 + 3, 22;
//apply * operator
throw '__RETURNED_VALUE='+ 10 + 3, 22;
//apply + operators
throw '__RETURNED_VALUE=103', 22;
//apply , operator
throw 22;
This problem can be avoided by completely eliminating the prefix '__RETURNED_VALUE=' and just replacing 'return' with 'throw'. However, this means that the code provided must run without throwing exceptions, which I thought to be a harder constraint than just crafting return statements to be simple (avoiding comma operators, non-parenthesized arithmetic, etc.). In addition, if a user ever creates a return statement that we can't handle with the current code, we conveniently throw an exception for it so it easily comes to our attention.

jsFiddle Demo
Lets assume your user can be a little smarter than the average bear. We are going to ask them to specifically provide an initial value and then a callback with an expression for that value.
The main benefit of doing this is that you avoid eval, and actually have a nice implementation that is re-usable as opposed to being subject to later refactoring.
This way also provides a separation of where the input comes from and where the examination comes from. Although the provided example only shows integer input, really it could be another call with absolutely no knowledge of the value aside that it needs to conform to the callback logic.
function expression(x,callback){
return callback(x);
}
out1.innerHTML = expression(8,function(x){
return x;
});
out2.innerHTML = expression(10,function(x){
return x;
});
out3.innerHTML = expression(10,function(x){
if(x === 10) return "yes"; "no";
});
out4.innerHTML = expression(8,function(x){
return x === 10 ? "yes" : "no";
});

parsing out data segment from a TCP stream

Im having some trouble with receiving packet data from a TCP stream. I think this is due in part to not understanding the servers responses.
my code (objective c):
unsigned type=0;
unsigned bufferFirstByte=0;
unsigned bufferSecondByte=0;
unsigned bufferThirdByte=0;
NSScanner *hexToInt = [NSScanner scannerWithString:[mutableBuffer objectAtIndex:0]];
[hexToInt scanHexInt:&bufferFirstByte];
hexToInt = [NSScanner scannerWithString:[mutableBuffer objectAtIndex:1]];
[hexToInt scanHexInt:&bufferSecondByte];
hexToInt = [NSScanner scannerWithString:[mutableBuffer objectAtIndex:2]];
[hexToInt scanHexInt:&bufferThirdByte];
hexToInt = [NSScanner scannerWithString:[mutableBuffer objectAtIndex:0]];
[hexToInt scanHexInt:&type];
int len = (bufferSecondByte<<8)+bufferSecondByte;
if (![mutableBuffer count]<(3+len)) {
NSArray *payload = [mutableBuffer subarrayWithRange:NSMakeRange(2,([mutableBuffer count] - 2))];
NSLog(#"length %d",len);
[self processReceive:type length:len payload:payload];
}
is some what modelled from this javascript code:
self.receive = function (itemName, data) {
self.log("Receiving: " + self.toHex(data));
self.ourData += data;
while (self.ourData.length >= 3) {
var type = self.ourData.charCodeAt(0);
var len = (self.ourData.charCodeAt(1) << 8) + self.ourData.charCodeAt(2);
if (self.ourData.length < (3 + len)) { // sanity check: buffer doesn't contain all the data advertised in the packet
break;
}
var payload = self.ourData.substr(3,len);
self.ourData = self.ourData.substr(3 + len);
self.processMessage(type, len, payload); // process payload
}
};
The reason for the modeling is that the command fusion javascript project is talking to the same server I am (a crestron controller).
However I could never get the len thing to work and I think thats whats causing my problem. When looking at a sample packet (05:00:06:00:00:03:00:52:00) the len would equal 1280 (see math above) even though the data portion is only 9bytes.
Currently my code will work but it misses certain data. This happens because of the streaming that TCP does (some packets are conjoined while others are fragmented). But without knowing the data segment size I cannot fix the issue and I believe the answer to that is the len variable. But I dont see how to properly implement it.
My question comes down to this. How can I determine the size of the data segment from this len variable or control my receive method to only except one data segment at a time (which from my research is not possible since TCP is made as a stream)?
I have a feeling there will be questions so Im going to attempt to answer a few of them here.
A. How do you come up with 1280: look at the math in the method ((self.ourData.charCodeAt(1) << 8) + self.ourData.charCodeAt(2);) (5<<8)+0=1280d
B. Why are you using different indexes:
You will notice that the index for what data goes where (payload, len, type). This is merely because they have their payload/data bytes as a string and myn is an array. in the end it is the same data being referenced

Use the following logic:
1) Do you have enough bytes to determine the length? If not, receive some more bytes and try again.
2) Do you have enough bytes to have one complete unit? If not, receive some more bytes and try again.
3) Extract one complete unit using the decoded length from step 1 and process it.
4) If we have no leftover bytes, we are done.
5) Return to step 1 to process the leftover bytes.

ok so i got some help from (this group) which you might not be able to see without a login for the group. in any case there is a 3 byte header. so my len which is 6 and not 1280 like i thought is actually 9 once the 3 is added for the header. and this gets me the value i was loooking for (9) since the data segment is 9bytes.
Thanks for the suggestions david, one up for some good basic knowledge.

How to to extract a javascript function from a javascript file

I need to extract an entire javascript function from a script file. I know the name of the function, but I don't know what the contents of the function may be. This function may be embedded within any number of closures.
I need to have two output values:
The entire body of the named function that I'm finding in the input script.
The full input script with the found named function removed.
So, assume I'm looking for the findMe function in this input script:
function() {
function something(x,y) {
if (x == true) {
console.log ("Something says X is true");
// The regex should not find this:
console.log ("function findMe(z) { var a; }");
}
}
function findMe(z) {
if (z == true) {
console.log ("Something says Z is true");
}
}
findMe(true);
something(false,"hello");
}();
From this, I need the following two result values:
The extracted findMe script
function findMe(z) {
if (z == true) {
console.log ("Something says Z is true");
}
}
The input script with the findMe function removed
function() {
function something(x,y) {
if (x == true) {
console.log ("Something says X is true");
// The regex should not find this:
console.log ("function findMe(z) { var a; }");
}
}
findMe(true);
something(false,"hello");
}();
The problems I'm dealing with:
The body of the script to find could have any valid javascript code within it. The code or regex to find this script must be able to ignore values in strings, multiple nested block levels, and so forth.
If the function definition to find is specified inside of a string, it should be ignored.
Any advice on how to accomplish something like this?
Update:
It looks like regex is not the right way to do this. I'm open to pointers to parsers that could help me accomplish this. I'm looking at Jison, but would love to hear about anything else.

A regex can't do this. What you need is a tool that parses JavaScript in a compiler-accurate way, builds up a structure representing the shape of the JavaScript code, enables you to find the function you want and print it out, and enables you to remove the function definition from that structure and regenerate the remaining javascript text.
Our DMS Software Reengineering Toolkit can do this, using its JavaScript front end. DMS provides general parsing, abstract syntax tree building/navigating/manipulation, and prettyprinting of (valid!) source text from a modified AST. The JavaScript front end provides DMS with compiler-accurate definition of JavaScript. You can point DMS/JavaScript at a JavaScript file (or even various kinds of dynamic HTML with embedded script tags containing JavaScript), have it produce the AST.
A DMS pattern can be used to find your function:
pattern find_my_function(r:type,a: arguments, b:body): declaration
" \r my_function_name(\a) { \b } ";
DMS can search the AST for a matching tree with the specified structure; because this is an AST match and not a string match, line breaks, whitespace, comments and other trivial differences won't fool it. [What you didn't say is what to if you have more than one
function in different scopes: which one do you want?]
Having found the match, you can ask DMS to print just that matched code which acts as your extraction step. You can also ask DMS to remove the function using a rewrite rule:
rule remove_my_function((r:type,a: arguments, b:body): declaration->declaration
" \r my_function_name(\a) { \b } " -> ";";
and then prettyprint the resulting AST. DMS will preserve all the comments properly.
What this does not do, is check that removing the function doesn't break your code. After all, it may be in a scope where it directly accesses variables defined locally in the scope. Removing it to another scope now means it can't reference its variables.
To detect this problem, you not only need a parser, but you need a symbol table with maps identifiers in the code to definitions and uses. The removal rule then has to add a semantic condition to check for this. DMS provides the machinery to build such a symbol table from the AST using an attribute grammar.
To fix this problem, when removing the function, it may be necessary to modify the function to accept additional arguments replacing the local variables it accesses, and modify the call sites to pass in what amounts to references to the local variables. This can be implemented with a modest sized set of DMS rewrite rules, that check the symbol tables.
So removing such a function can be a lot more complex than just moving the code.

If the script is included in your page (something you weren't clear about) and the function is publicly accessible, then you can just get the source to the function with:
functionXX.toString();
https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Function/toString
Other ideas:
1) Look at the open source code that does either JS minification or JS pretty indent. In both cases, those pieces of code have to "understand" the JS language in order to do their work in a fault tolerant way. I doubt it's going to be pure regex as the language is just a bit more complicated than that.
2) If you control the source at the server and are wanted to modify a particular function in it, then just insert some new JS that replaces that function at runtime with your own function. That way, you let the JS compiler identify the function for you and you just replace it with your own version.
3) For regex, here's what I've done which is not foolproof, but worked for me for some build tools I use:
I run multiple passes (using regex in python):
Remove all comments delineated with /* and */.
Remove all quoted strings
Now, all that's left is non-string, non-comment javascript so you should be able to regex directly on your function declaration
If you need the function source with strings and comments back in, you'll have to reconstitute that from the original, now that you know the begin end of the function
Here are the regexes I use (expressed in python's multi-line format):
reStr = r"""
( # capture the non-comment portion
"(?:\\.|[^"\\])*" # capture double quoted strings
|
'(?:\\.|[^'\\])*' # capture single quoted strings
|
(?:[^/\n"']|/[^/*\n"'])+ # any code besides newlines or string literals
|
\n # newline
)
|
(/\* (?:[^*]|\*[^/])* \*/) # /* comment */
|
(?://(.*)$) # // single line comment
$"""
reMultiStart = r""" # start of a multiline comment that doesn't terminate on this line
(
/\* # /*
(
[^\*] # any character that is not a *
| # or
\*[^/] # * followed by something that is not a /
)* # any number of these
)
$"""
reMultiEnd = r""" # end of a multiline comment that didn't start on this line
(
^ # start of the line
(
[^\*] # any character that is not a *
| # or
\*+[^/] # * followed by something that is not a /
)* # any number of these
\*/ # followed by a */
)
"""
regExSingleKeep = re.compile("// /") # lines that have single lines comments that start with "// /" are single line comments we should keep
regExMain = re.compile(reStr, re.VERBOSE)
regExMultiStart = re.compile(reMultiStart, re.VERBOSE)
regExMultiEnd = re.compile(reMultiEnd, re.VERBOSE)
This all sounds messy to me. You might be better off explaining what problem you're really trying to solve so folks can help find a more elegant solution to the real problem.

I built a solution in C# using plain old string methods (no regex) and it works for me with nested functions as well. The underlying principle is in counting braces and checking for unbalanced closing braces. Caveat: This won't work for cases where braces are part of a comment but you can easily enhance this solution by first stripping out comments from the code before parsing function boundaries.
I first added this extension method to extract all indices of matches in a string (Source: More efficient way to get all indexes of a character in a string)
/// <summary>
/// Source: https://stackoverflow.com/questions/12765819/more-efficient-way-to-get-all-indexes-of-a-character-in-a-string
/// </summary>
public static List<int> AllIndexesOf(this string str, string value)
{
if (String.IsNullOrEmpty(value))
throw new ArgumentException("the string to find may not be empty", "value");
List<int> indexes = new List<int>();
for (int index = 0; ; index += value.Length)
{
index = str.IndexOf(value, index);
if (index == -1)
return indexes;
indexes.Add(index);
}
}
I defined this struct for easy referencing of function boundaries:
private struct FuncLimits
{
public int StartIndex;
public int EndIndex;
}
Here's the main function where I parse the boundaries:
public void Parse(string file)
{
List<FuncLimits> funcLimits = new List<FuncLimits>();
List<int> allFuncIndices = file.AllIndexesOf("function ");
List<int> allOpeningBraceIndices = file.AllIndexesOf("{");
List<int> allClosingBraceIndices = file.AllIndexesOf("}");
for (int i = 0; i < allFuncIndices.Count; i++)
{
int thisIndex = allFuncIndices[i];
bool functionBoundaryFound = false;
int testFuncIndex = i;
int lastIndex = file.Length - 1;
while (!functionBoundaryFound)
{
//find the next function index or last position if this is the last function definition
int nextIndex = (testFuncIndex < (allFuncIndices.Count - 1)) ? allFuncIndices[testFuncIndex + 1] : lastIndex;
var q1 = from c in allOpeningBraceIndices where c > thisIndex && c <= nextIndex select c;
var qTemp = q1.Skip<int>(1); //skip the first element as it is the opening brace for this function
var q2 = from c in allClosingBraceIndices where c > thisIndex && c <= nextIndex select c;
int q1Count = qTemp.Count<int>();
int q2Count = q2.Count<int>();
if (q1Count == q2Count && nextIndex < lastIndex)
functionBoundaryFound = false; //next function is a nested function, move on to the one after this
else if (q2Count > q1Count)
{
//we found the function boundary... just need to find the closest unbalanced closing brace
FuncLimits funcLim = new FuncLimits();
funcLim.StartIndex = q1.ElementAt<int>(0);
funcLim.EndIndex = q2.ElementAt<int>(q1Count);
funcLimits.Add(funcLim);
functionBoundaryFound = true;
}
testFuncIndex++;
}
}
}

I am almost afraid that regex cannot do this job. I think it is the same as trying to parse XML or HTML with regex, a topic that has already caused various religious debates on this forum.
EDIT: Please correct me if this is NOT the same as trying to parse XML.

I guess you would have to use and construct a String-Tokenizer for this job.
function tokenizer(str){
var stack = array(); // stack of opening-tokens
var last = ""; // last opening-token
// token pairs: subblocks, strings, regex
var matches = {
"}":"{",
"'":"'",
'"':'"',
"/":"/"
};
// start with function declaration
var needle = str.match(/function[ ]+findme\([^\)]*\)[^\{]*\{/);
// move everything before needle to result
var result += str.slice(0,str.indexOf(needle));
// everithing after needle goes to the stream that will be parsed
var stream = str.slice(str.indexOf(needle)+needle.length);
// init stack
stack.push("{");
last = "{";
// while still in this function
while(stack.length > 0){
// determine next token
needle = stream.match(/(?:\{|\}|"|'|\/|\\)/);
if(needle == "\\"){
// if this is an escape character => remove escaped character
stream = stream.slice(stream.indexOf(needle)+2);
continue;
}else if(last == matches[needle]){
// if this ends something pop stack and set last
stack.pop();
last = stack[stack.length-1];
}else if(last == "{"){
// if we are not inside a string (last either " or ' or /)
// push needle to stack
stack.push(needle);
last = needle;
}
// cut away including token
stream = stream.slice(stream.indexOf(needle)+1);
}
return result + stream;
}
oh, I forgot tokens for comments... but i guess you got an idea now of how it works...