Develop Reference

Getting JSON data from snowflake stored procedure parameter and inserting it in target table

I have a requirement to receive JSON data in a Stored Proc parameter and insert the same in the snowflake target table (user_json_feedback). JSON Data has three key elements(User, EntityID, and Entity Type), whereas the target table has five columns (User, ID, Entity Type, Region, and Date). The region will have a default value of "NA," and the date will be the current date.
If the inserts are successful, it returns true; otherwise, it returns false.
I am struggling with the syntax and parsing issues here, as I am very new to writing procedures.
Here is what I have been trying to do, which is giving me errors obviously but serves the algorithm of my intent.
CREATE OR REPLACE SP_UPDATE_JSON_DATA (JSON_DATA VARIANT)
RETURNS BOOLEAN
LANGUAGE JAVASCRIPT
EXECUTE AS OWNER
AS
$$
//Declare variables
var REGION = 'NA'
var V_DATE = `select current_date;`;
var DATE_STMT= snowflake.createStatement({sqlText: V_DATE });
var curr_date = DATE_STMT.execute();
var src_json = JSON.parse(JSON_DATA);
var sql_command =
`INSERT INTO user_json_feedback (user,id,etype,region ,date)//
select src_json:USER,src_json:ENTITY_ID,src_json:ENTITY_TYPE,REGION,curr_date;`;
try {
snowflake.execute (
{sqlText: sql_command}
);
return "Succeeded."; // Return a success/error indicator.
}
catch (err) {
return "Failed: " + err; // Return a success/error indicator.
}
$$;
The function call with parameters will be like
call SP_UPDATE_JSON_DATA ('[{"USER":"XYZ","ENTITY_ID":"BMT0001","ENTITY_TYPE":"BMT"},{"USER":"ABC","ENTITY_ID":"BMT0002","ENTITY_TYPE":"BMT"}]');
Thanks in advance for the help!

theres a few things here.
Firstly the step to get current date. curr_date is a result set object. to extract the value and use it later, you need to read the first row with .next() then GetColumnValue to read the column content. to pass it later as a well formatted string you'll wanna convert with .toISOString().
Secondly the parsed json returns an array in this case so you'll need to iterate over the array to insert the individual records. As it's not known ahead of time if the variant will contain an array you're best checking if the parsed json is an array and handle it accordingly
Last tweak was altering the return type so you get the verbose feedback you're expecting from your return calls.
Updated code:
CREATE OR REPLACE TEMPORARY TABLE user_json_feedback
(
user VARCHAR(100)
,id VARCHAR(100)
,etype VARCHAR(100)
,region VARCHAR(100)
,date TIMESTAMP_NTZ
);
CREATE OR REPLACE TEMPORARY PROCEDURE SP_UPDATE_JSON_DATA(JSON_DATA VARIANT)
RETURNS STRING
LANGUAGE JAVASCRIPT
EXECUTE AS OWNER
AS
$$
//Declare variables
var REGION = 'NA'
var V_DATE = `select current_date;`;
var DATE_STMT= snowflake.createStatement({sqlText: V_DATE });
var DATE_STMT_RES = DATE_STMT.execute();
DATE_STMT_RES.next()
var curr_date = DATE_STMT_RES.getColumnValue(1).toISOString();
var src_json = JSON.parse(JSON_DATA);
try {
if (Array.isArray(src_json)){
for (key in src_json){
var sql_command =
`INSERT INTO user_json_feedback (user,id,etype,region,date)//
VALUES(:1,:2,:3,:4,:5)`;
snowflake.execute (
{
sqlText: sql_command,
binds: [src_json[key].USER,src_json[key].ENTITY_ID,src_json[key].ENTITY_TYPE,REGION,curr_date]
}
);
}
}
else {
var sql_command =
`INSERT INTO user_json_feedback (user,id,etype,region,date)//
VALUES(:1,:2,:3,:4,:5)`;
snowflake.execute (
{
sqlText: sql_command,
binds: [src_json.USER,src_json.ENTITY_ID,src_json.ENTITY_TYPE,REGION,curr_date]
}
);
}
return "Succeeded."; // Return a success/error indicator.
}
catch (err) {
return "Failed: " + err; // Return a success/error indicator.
}
$$;
--Need to cast variable string as variant.
--ARRAY example
call SP_UPDATE_JSON_DATA ('[{"USER":"XYZ","ENTITY_ID":"BMT0001","ENTITY_TYPE":"BMT"},{"USER":"ABC","ENTITY_ID":"BMT0002","ENTITY_TYPE":"BMT"}]'::VARIANT);
--Single object example
call SP_UPDATE_JSON_DATA ('{"USER":"JST","ENTITY_ID":"BMT0003","ENTITY_TYPE":"BMT"}'::VARIANT);
SELECT *
FROM user_json_feedback;
Result set:
While all this works, you may well be better served just inserting the whole variant into a table and relying on snowflake's significant semi-structured data querying capabilities. Certainly for large payloads you'll find much better performance from bulk loading to a variant column in a table then parsing in a view.

How to return a MAP like structure from bigquery javascript UDF so that I can generate a key-value column dynamically at runtime?

I'm trying to return a MAP like structure from Javascript UDF in bigquery. So that I can convert that structure directly into relational columns without knowing the column named prior.
In below approach, I'm trying to stringify JSON and then using json_extract_scaler function to create columns.
CREATE TEMP FUNCTION main(json_str STRING)
RETURNS STRING
LANGUAGE js AS
r"""
var row = JSON.parse(json_str);
return JSON.stringify(row);
""";
with temp_table as (
select "ram" name, "ram#gmail.com" email
),
Rule_result as (SELECT main(TO_JSON_STRING(STRUCT(t.name, t.email))) result FROM temp_table as t)
SELECT json_extract_scalar(result, '$.name') name,
json_extract_scalar(result, '$.email') email
FROM Rule_result as r;
In this approach, I'm returning the struct, knowing the column names beforehand.
CREATE TEMP FUNCTION main(json_str STRING)
RETURNS STRUCT<name STRING, email STRING>
LANGUAGE js AS
r"""
var row = JSON.parse(json_str);
return row;
""";
with temp_table as (
select "ram" name, "ram#gmail.com" email
),
Rule_result as (SELECT main(TO_JSON_STRING(STRUCT(t.name, t.email))) result FROM temp_table as t)
SELECT r.result.* FROM Rule_result as r;
Both the approaches work fine. But it doesn't solve the problem. Because I need to be aware of the column names.
Bigquery supports struct return type but that doesn't fit my usecase. Since I'm not aware about the column names beforehand.
How can I dynamically create columns from the data return by the javascript UDF without knowing the column names?
data is flat JSON object
{
"name":"ram",
"email":"ram#gmail.com"
}
I somehow need to convert this JSON object into table columns like
name
email
ram
ram#gmail.com

Consider below
create temp function extract_keys(input string) returns array<string> language js as """
return Object.keys(JSON.parse(input));
""";
create temp function extract_values(input string) returns array<string> language js as """
return Object.values(JSON.parse(input));
""";
create temp table tmp as
select id, key, value
from your_table,
unnest(extract_keys(json)) key with offset
join unnest(extract_values(json)) value with offset
using(offset);
execute immediate (select
'''select * from tmp
pivot (any_value(value) for key in (''' || string_agg(distinct "'" || key || "'") || '''))
'''
from tmp
);
if applied to sample data like in your question's example
output is

Snowflake JavaScript procedure how to update a field from an object that is not in a stage?

I am trying to do a JavaScript procedure on Snowflake to update a table with relevant values in a JavaScript array.
Assume that I have the following table:
And having this array:
var arr = {"gender_value": "Gender", "age_range": "Age Range"}
So the final result of my updated table would be:
I tried something like:
var query = "
MERGE INTO mytable m
USING (SELECT * FROM "+arr+" )
";
But I don't think it is possible to SELECT from an object if it is not in a stage.

You are going to have to parse your array in SQL to get structure for the MERGE. Another alternative is to store array in temporary table. But I don't see any issues getting this to work, see my example:
create or replace table so_test
(
question_name varchar
,answer varchar
,question_label varchar
);
insert into so_test values ('gender_value','Female',null),('age_range','>60',null);
merge into so_test
using
(
select
g.key as join_key
,g.value as join_value
from
(
select parse_json(column1) as arr from values ('{"gender_value": "Gender", "age_range": "Age Range"}')
) x,
lateral flatten(input => x.arr) g
)src
on so_test.question_name = src.join_key
when matched then update
set question_label = src.join_value;
--proc example
CREATE OR REPLACE PROCEDURE "ARRAY_TEST_SP"(sp_input varchar)
RETURNS VARCHAR(16777216)
LANGUAGE JAVASCRIPT
EXECUTE AS OWNER
AS
$$
var v_array = SP_INPUT;
var v_merge = `merge into so_test
using
(
select
g.key as join_key
,g.value as join_value
from
(
select parse_json(column1) as arr from values (?)
) x,
lateral flatten(input => x.arr) g
)src
on so_test.question_name = src.join_key
when matched then update
set question_label = src.join_value;`;
var v_stmt = snowflake.createStatement(
{
sqlText: v_merge,
binds:[v_array]
}
);
try {
v_stmt.execute()
return "Succeeded."; // Return a success/error indicator.
}
catch (err) {
return "Failed: " + err; // Return a success/error indicator.
}
$$;
call ARRAY_TEST_SP('{"gender_value": "Gender", "age_range": "Age Range"}');

node-mssql Transaction insert - Returning the inserted id..?

I'm using node-mssql 3.2.0 and I need to INSERT INTO a table and return the id of the inserted record.
I can successfully use sql.Transaction() to insert data, but the only parameters given to callbacks (request.query() and transaction.commit()) are:
const request = new sql.Request();
request.query('...', (err, recordset, affected) => {});
const transaction = new sql.Transaction();
transaction.commit((err) => {});
So recordset is undefined for INSERT, UPDATE and DELETE statements, and affected is the number of rows affected, in my case 1.
Does anyone know a good way to obtain an inserted records id (just a primary key id) after a transaction.commit() using node-mssql..?

Instead of just doing an INSERT INTO... statement, you can add a SELECT... statement as well:
INSERT INTO table (...) VALUES (...); SELECT SCOPE_IDENTITY() AS id;
The SCOPE_IDENTITY() function returns the inserted identity column, which means recordset now contains the id:
const request = new sql.Request();
request.query('...', (err, recordset, affected) => {});
I don't think request.multiple = true; is required, because although this includes multiple statements, only one of them is a SELECT... and so returns.
So the answer was SQL related and is not specific to node-mssql.

I know this question has accepted answer.
I made the following way:
let pool = await sql.connect(config);
let insertItem = await pool.request()
.input('ItemId',sql.NVarChar, 'itemId1234')
.input('ItemDesc',sql.NVarChar, 'nice item')
.query("insert into itemTable (Id, ItemId,ItemDesc) OUTPUT INSERTED.ID
values (NEWID(), #ItemId, #ItemDesc);
var insertedItemId = insertItem.recordset[0].ID
This adds unique identifier to data that is saved to db (if table is created so)
create table itemTable(
Id UNIQUEIDENTIFIER primary key default NEWID(),
ItemId nvarchar(25),
ItemDesc nvarchar(25)
)

JS and ExpressionEngine - Remove KV pairs by duplicate values only?

We're building a site with ExpressionEngine. We are running a SQL query to gather up all member IDs for a specific member group. After that, we are using EE tags to get data from a custom member field for each member ID.
The ID and field data need to stay paired, as we will be populating a drop-down so that the ID is the value and the field data is the text, so we are currently putting them into a JS array as key/value pairs. The call is as follows:
var array= [
{exp:query sql="SELECT * FROM exp_members WHERE group_id = 5"}
{exp:member:custom_profile_data
member_id="{member_id}"}
{if company != ''}
{{member_id}:"{company}"},
{/if}
{/exp:member:custom_profile_data}
{/exp:query}
};
This gives us the output:
var array = [
{1:"name01"},
{2:"name02"},
{3:"name01"},
{4:"name03"}
];
Now, our problem. We need to remove objects based on duplicate field data (values) only, so the above array would look like this:
var array = [
{1:"name01"},
{2:"name02"},
{4:"name03"}
];
None of these IDs (keys) will ever be the same, but the field data (values) can be. So we want to keep the first KV pair that comes through with a unique value, but remove any subsequent dupes of that value - despite the fact that they will not be true "duplicate values" due to a different ID (key).
Keeping in mind that the KV pairs are all dynamic, is there any possible way to do this via JS so we can create a new array for the cleaned data to pass to the drop-down?

You could handle the duplications by modifying your MySQL query. (In my example, my custom field ID was 1.)
var myArray = [];
{exp:query sql="SELECT MIN(m.member_id) AS co_member_id, d.m_field_id_1 AS company FROM exp_members m INNER JOIN exp_member_data d ON m.member_id = d.member_id WHERE d.m_field_id_1 != '' AND m.group_id > 0 GROUP BY d.m_field_id_1;"}
myArray.push({{co_member_id}: "{company}"});
{/exp:query}
This query would use the first (in the ordinal sense) member_id found; you could also change the MIN to MAX and get the last.
This will give you a clean output in your source, without the need for any additional JS processing. I'd also recommend changing the names of the variables you're outputting as to not conflict in EE's parsing.

I would do it like...
function removeDups(arry){
var tmp = {}, retainIdx=[], newArry=[];
arry.forEach(function(obj, idx){
var val = obj[Object.keys(obj)[0]];
if(val && !tmp[val]){
retainIdx.push(idx);
tmp[val] = true;
}
});
retainIdx.forEach(function(i){
newArry.push(arry[i]);
});
return newArry;
};