Develop Reference

Date Formatting with Marklogic Optic API

My question is about getting the MarkLogic query console javascript API to format a column of strings to dates.
Working on a string directly works as expected:
var d = new Date("3/12/2019");
xdmp.monthNameFromDate(xs.date(d))
>>> March
Working with the optic api however:
const op = require('/MarkLogic/optic');
const ind = op.fromView('schema', 'money');
//get non null dates, stored as strings, [MM-DD-YYYY]
const ind2 = ind.where(op.ne(op.col('completed date'), ""))
const testMonth = op.as('testDate', fn.formatDate(xs.date(op.col('completed date')), "[M01]-[D01]-[Y0001]"))
Returns the following error:
[javascript] XDMP-CAST: function bound ()() -- Invalid cast: {_expr:"¿\"completed date\"", _preplans:null, _schemaName:null, ...} cast as xs.date
I believe this is different than the other questions on this topic because those didn't involve the OPTIC API as far as I can tell, and were resolved by just operating on single strings.How to convert string to date type in MarkLogic?
I need to take an optic "column" and convert its type to a date object so I can call the https://docs.marklogic.com/xdmp.monthNameFromDate and other related tools on it.
I feel missing something very straightforward about applying functions to row sets and selecting specific columns.
What I naturally want to do is apply a function to each property of the resulting row set:
let formatted = resulting_rows.map(x=>Date(x['completed date'])
or whatever. This is basically what I do client side, but it feels incorrect to just throw away so much of the built-in javascript functionality and do this all in the browser, especially when I need to do groups on years and months from these views.
It doesn't help that some links about operating on objects are broken:
https://docs.marklogic.com/map.keys

The op.as() call defines a dynamic column based on an expression that's applied to each row when the query is executed.
The expression can only use calls to functions provided by the Optic API. In particular, where xs.date() executes when called, op.xs.date() executes when each row is processed. Similarly fn.formatDate() executes immediately while op.fn.formatDate() executes during row processing.
To use the dynamic column, provide it as an argument to op.select(), similar to the following sketch:
op.fromView('schema', 'money');
.where(op.ne(op.col('completed date'), ""))
.select([
op.col('completed date'),
op.as('testDate', op.fn.formatDate(
xdmp.parseDateTime(
op.col('completed date'),
"[M01]/[D01]/[Y0001]"),
"[M01]-[D01]-[Y0001]"))
])
.result();
The call to .result() executes the query pipeline.
The map is an XQuery equivalent to JavaScript literal that's not used in sever-side JavaScript. Optic does support a map() pipeline step, which takes a lambda and appears in the pipeline step immediately before the call to result() as documented in:
http://docs.marklogic.com/AccessPlan.prototype.map
Belated footnote: One alternative for this case to parsing and formatting the date would be to use op.fn.translate() to transform the column value by turning every instance of "/" into "-"
Hoping that helps,

Quotationmark problem with Knex and SQL-database

I'm trying to save a query to a database using Knex. But if this query I'm trying to save includes quotation marks I'm getting an error.
Here's an example of how the code might look like:
db.query(`INSERT INTO test.searches VALUES ('TestUser', 'testqueryname', ''SELECT * FROM table WHERE team='rocket'')`, info, () =>{}
Of course in the real case I'm not sending in hardcoded values but variables as strings.
Trying to save this gets me the error
error: syntax error at or near "rocket"

To start with looks like your code has javascript syntax error at least closing parenthesis is missing.
Secondly knex does not have .query() method (maybe you are using database driver directly?). You should probably be using knex.raw() even that query above could be easily written with normal knex methods.
If you are using knex.raw you can do quoting with ?? replacements and value bindings wit ?.
If you are writing your code like in the question you have no advantage at all from using knex and you should be using database driver directly instead.

If you are running MySQL, you need to slash escape quotation marks:
db.query(`INSERT INTO test.searches VALUES ('TestUser', 'testqueryname', 'SELECT * FROM table WHERE team=\'rocket\'')`, info, () =>{}
If you are running PostgreSQL, you need to double escape quotation marks:
db.query(`INSERT INTO test.searches VALUES ('TestUser', 'testqueryname', 'SELECT * FROM table WHERE team=''rocket''')`, info, () =>{}
Please consider using parameterised SQL queries or at least appropriately escaping input to avoid SQL injection attacks.

Parsing and constructing filtering queries similiar to SQL WHERE clause in Python/JavaScript

I am building a query engine for a database which is pulling data from SQL and other sources. For normal use cases the users can use a web form where the use can specify filtering parameters with select and ranged inputs. But for advanced use cases, I'd like to to specify a filtering equation box where the users could type
AND, OR
Nested parenthesis
variable names
, <, =, != operators
So the filtering equation could look something like:
((age > 50) or (weight > 100)) and diabetes='yes'
Then this input would be parsed, input errors detected (non-existing variable name, etc) and SQL Alchemy queries built based on it.
I saw an earlier post about the similar problem https://stackoverflow.com/a/1395854/315168
There seem to exist several language and mini-language parsers for Python http://navarra.ca/?p=538
However, does there exist any package which would be out of the box solution or near solution for my problem? If not what would be the simplest way to construct such query parser and constructor in Python?

Have a look at https://github.com/dfilatov/jspath
It's similar to xpath, so the syntax isn't as familiar as SQL, but it's powerful over hierarchical data.

I don't know if this is still relevant to you, but here is my answer:
Firstly I have created a class that does exactly what you need. You may find it here:
https://github.com/snow884/filter_expression_parser/
It takes a list of dictionaries as an input + filter query and returns the filtered results. You just have to define the list of fields that are allowed plus functions for checking the format of the constants passed as a part of filter expression.
The filter expression it ingests has to have the following format:
(time > 45.34) OR (((user_id eq 1) OR (date gt '2019-01-04')) AND (username ne 'john.doe'))
or just
username ne 'john123'
Secondly it was foolish of me to even create this code because dataframe.query(...) from pandas already does almost exactly what you need: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.query.html

How to cast a Javascript String to SQL LongText

I am facing a problem for an hour or so with a LongText column in SQL.
I am currently working on a web project where at some point we have a search form which allows user to search through numerous fields. Due to the customer's users being forced to use IE 9, we had to create a javascript... script, that parse the whole form and only takes what's needed in order not to exceed the 2k characters limit IE has.
So far so good, the script was working fine until we added a textarea field which represents a LongText column in our DB. The problem seems to come from the fact SQL won't take a simple String in an SQL statement as a LongText.
Here is what the query looks like:
SELECT * FROM SONORA.CF_CPR_CASE WHERE CASEFOLDERID != 2
and CMF_VLM_APPLICATION like '%CPR%' and CPR_DESCRIPTION='rererere'
and CMF_TYPE_CASE_FK like '%LC_130125_074927_000001_60%'
But I get this error:
EJBException:; nested exception is: javax.ejb.EJBException: The data types ntext
and varchar are incompatible in the equal to operator.S_EXCP(204); nested
exception is: javax.ejb.EJBException: The data types ntext and varchar
are incompatible in the equal to operator.S_EXCP(204)
Found in FmsQuery: SELECT * FROM {{DBTABLEPREFIX}}CF_CPR_CASE WHERE
CASEFOLDERID != 2 and CMF_VLM_APPLICATION like '%CPR%' and
CPR_DESCRIPTION='ztztztztz' and CMF_TYPE_CASE_FK like
'%LC_130125_074927_000001_60%'
We are using CASE360 which is not so known but still doesn't matter much, we are building the "where clause" of the SQL statement through javascript and then send it to the CASE360 processor which will take this all and execute the query. This part works fine, it's just the LongText column which is giving me a hard time.
If you guys have any idea what this SQL Query should look like to be successfully interpreted then I'd be infinitely happy!
Thank you for your help in advance.

You can try using a CAST on the column which is ntext. It's not clear to me which column that would be from the error above. As an example, let's assume it's the CPR_DESCRIPTION column:
SELECT *
FROM SONORA.CF_CPR_CASE
WHERE CASEFOLDERID != 2
and CMF_VLM_APPLICATION like '%CPR%'
and CAST(CPR_DESCRIPTION as varchar(2000)) ='rererere'
and CMF_TYPE_CASE_FK like '%LC_130125_074927_000001_60%'
;
Keep in mind that you need to pick an appropriately large varchar size for the cast, and that even with the maximum size there is a potential for data loss.

How Do You Fix A Parameter Names Mismatch - DOJO and PL/SQL

How do you fix a names mismatch problem, if the client-side names are keywords or reserved words in the server-side language you are using?
The DOJO JavaScript toolkit has a QueryReadStore class that you can subclass to submit REST patterned queries to the server. I'm using this in conjunction w/ the FilteringSelect Dijit.
I can subclass the QueryReadStore and specify the parameters and arguments getting passed to the server. But somewhere along the way, a "start" and "count" parameter are being passed from the client to the server. I went into the API and discovered that the QueryReadStore.js is sending those parameter names.
I'm using Fiddler to confirm what's actually being sent and brought back. The server response is telling me I have a parameter names mismatch, because of the "start" and "count" parameters. The problem is, I can't use "start" and "count" in PL/SQL.
Workaround or correct implementation advice would be appreciated...thx.
//I tried putting the code snippet in here, but since it's largely HTML, that didn't work so well.

While it feels like the wrong thing to do, because I'm hacking at a well tested, nicely written JavaScript toolkit, this is how I fixed the problem:
I went into the DOJOX QueryReadStore.js and replaced the "start" and "count" references with acceptable (to the server-side language) parameter names.
I would have like to handled the issue via my PL/SQL (but I don't know how to get around reserved words) or client-side code (subclassing did not do the trick)...without getting into the internals of the library. But it works, and I can move on.

As opposed to removing it from the API, as you mentioned, you can actually create a subclass with your own fetch, and remove start/count parameters (theoretically). Have a look at this URL for guidance:
http://www.sitepen.com/blog/2008/06/25/web-service-data-store/
Start and count are actually very useful because they allow you to pass params for the query that you can use to filter massive data sets, and it helps to manage client-side paging. I would try to subclass instead, intercept, and remove.

Is your pl/sql program accessed via a URL and mod_plsql? If so, then you can use "flexible parameter passing" and the variables get assigned to an array of name/value pairs.
Define your package spec like this...
create or replace package pkg_name
TYPE plsqltable
IS
TABLE OF VARCHAR2 (32000)
INDEX BY BINARY_INTEGER;
empty plsqltable;
PROCEDURE api (name_array IN plsqltable DEFAULT empty ,
value_array IN plsqltable DEFAULT empty
);
END pkg_name;
Then the body:
CREATE OR REPLACE PACKAGE BODY pkg_name AS
l_count_value number;
l_start_value number;
PROCEDURE proc_name (name_array IN plsqltable DEFAULT empty,
value_array IN plsqltable DEFAULT empty) is
------------
FUNCTION get_value (p_name IN VARCHAR) RETURN VARCHAR2 IS
BEGIN
FOR i IN 1..name_array.COUNT LOOP
IF UPPER(name_array(i)) = UPPER(p_name) THEN
RETURN value_array(i);
END IF;
END LOOP;
RETURN NULL;
END get_value;
----------------------
begin
l_count_value := get_value('count');
l_start_value := get_value('start');
end api;
end pkg_name;
Then you can call pkg_name.api using
http://server/dad/!pkg_name.api?start=3&count=3