Need better JSON Schema to validate sparse matrix data object - javascript

I need to validate complex object in JavaScript.
Object is based on dictionary:
var dict = {'1':true,'2':true,'3':true};
Object store matrix of pairs (usually not full):
var obj = {'1':{
'1': 'str1',
'2': 'str2',
'3': 'str3',
},'2':{
'1': 'str1',
'2': 'str2',
}
};
I make validation schema with AJV validator.
Schema requirements:
1st level object contains only properties from dictionary.
2nd level object contains only properties from dictionary.
data is a string
Generating schema:
var dict = {'1':true,'2':true,'3':true};
var subProperties = R.map(function(item){
return {
'type' : 'string',
"minLength": 1,
}
}, dict);
var root = {
"type" : "object",
"additionalProperties" : false
};
root.properties = R.map(function(item){
return {
"type" : "object",
'properties' : subProperties,
"additionalProperties" : false
};
}, dict);
console.log(root)
<script src="https://cdnjs.cloudflare.com/ajax/libs/ajv/4.9.0/ajv.min.js"></script>
<script src="//cdnjs.cloudflare.com/ajax/libs/ramda/0.22.1/ramda.min.js"></script>
This schema is working well but the problem is performance. When dictionary contains 200 elements it requires 10 seconds to compile this schema (validation is fast, compilation is slow). Moreover it throws out of memory exception time to time. Is it possible to make better validation schema?

Memory exceptions are not surprising here given that you are validating 40000 properties. Your validation function code size should be around 30Mb.
You can use propertyNames keyword that I added to v5/6 proposals and that is available in ajv-keywords package.
var dict = ['1', '2', '3'];
var schema = {
type: 'object',
propertyNames: { enum: dict },
additionalProperties: {
type: 'object',
propertyNames: { enum: dict },
additionalProperties: {
type: 'string',
minLength: 1
}
}
};
var ajv = require('ajv')();
require('ajv-keywords')(ajv, 'propertyNames');
var validate = ajv.compile(schema);
This schema is tiny but does the same.
You can achieve the same using patternProperties keyword that exists in the current standard version (with es6 property syntax):
var names = '^(1|2|3)$';
var schema = {
type: 'object',
additionalProperties: false,
patternProperties: {
[names]: {
type: 'object',
additionalProperties: false,
patternProperties: {
[names]: {
type: 'string',
minLength: 1
}
}
}
}
};
propertyNames looks simpler and should be faster I think.

By mapping over items the result is not 1 but N schema (i.e. if you have 200 items, 200 schema get created for the sole purpose of validating the key).
The alternative is to either use patternProperties with a huge 200 key long RegExp or more simply, just validate the object manually.
var dict = {'1':true,'2':true,'3':true};
var monsterRegex = '^' + Object.keys(dict).join('|') + '$'
var valSchema = {
type: 'string',
minLength: 1
}
var keySchema = {
type: 'object',
additionalProperties: false,
patternProperties: {}
}
keySchema.patternProperties[monsterRegex] = valSchema
var objSchema = {
type: 'object',
additionalProperties: false,
patternProperties: {}
}
objSchema.patternProperties[monsterRegex] = keySchema
console.log(objSchema)
<script src="https://cdnjs.cloudflare.com/ajax/libs/ajv/4.9.0/ajv.min.js"></script>

Related

How to use reduce with typescript to populate a typed array?

I have an array of objects which is populated with objects generated by a reduce loop. These objects already have a type, hence I need the object generated by reduce to have the same type, but I'm strugling with the initial value of the reduce, since it is an empty object.
How can I set a type to the initial object without getting an error saying that the object is missing properties?
Interface and base value:
const productFormLocal = [
{
field: 'id',
config: {
elementType: false,
value: '',
validation: {},
valid: true,
touched: false
}
},
{
field: 'nome',
config: {
elementType: 'input',
elementConfig: {
type: 'text',
placeholder: 'Nome do Produto',
name: 'nome',
},
label: 'Nome do Produto',
value: '',
validation: {
required: true,
},
valid: false,
touched: false
}
}
]
interface ProductsList {
id: number
nome: string
qtde: number
valor: number
valorTotal: number
}
const productsList: ProductsList[] = []
For instance, if I do that I get the reduce working fine, but I wouldn't be able to push the generated objects to the array:
const data: Record<string, any> = {}
const productValues = productFormLocal.reduce((obj, item) => (
obj[item.field] = item.config.value, obj
), data)
productsList.push(productValues)
And if I do that I would get an error saying that data is missing the ProductList properties:
const data: ProductsList = {}
const productValues = productFormLocal.reduce((obj, item) => (
obj[item.field] = item.config.value, obj
), data)
productsList.push(productValues)
How could I solve this? I looked a lot but I coudn't find a way to get this right.
I know I could set all properties of ProductsList as optional, but it doesn't seem to be the best approach.
Set the reduce generic type, which will type the default value as well as the reducer function return type.
const productValues = productFormLocal.reduce<Record<string, any>>((acc, item) => ({
...acc,
[item.field]: item.config.value
}), {})

Sails ORM doesn't understand numeric values?

Well this should work, everything (when going through it) step by step works. Except the actual writing to the database. In my sails application I have a route to the following function:
add: function (req, res) {
let params = req.allParams();
let idx = params.event_id;
let name = params.name;
let sport = params.sport;
let location = params.location;
let idNr = Number(idx);
let sportNr = Number(sport);
let locationNr = Number(location);
let dat = {
name:name,
id:idNr,
sport:sportNr,
location:locationNr
};
console.log(dat);
Event.create(dat)
.then(function (res){ return res.ok(); })
.catch(function (err){
console.log(err);
return res.send(err);
});
}
Really a simplistic function right? Yet on my "good" data it fails:
{
"error": "E_VALIDATION",
"status": 400,
"summary": "2 attributes are invalid",
"model": "Event",
"invalidAttributes": {
"location": [
{
"rule": "number",
"message": "Value should be a number (instead of \"3\", which is a string)"
}
],
"sport": [
{
"rule": "number",
"message": "Value should be a number (instead of \"2\", which is a string)"
}
]
}
}
How can this happen? I cleary am using dat which takes a numeric variation of the "sport" and "location". - Logging even shows that the dat is filled as expected with numeric values.
So it makes no sense at all that this fails; it shouldn't.
EDIT, the model is defined as:
module.exports = {
attributes: {
id: {
type: 'number',
required: true,
unique: true,
},
name: {
type: 'string',
required: true
},
location: {
type: 'number',
required: true,
},
sport: {
type: 'number',
required: true
},
},
};
This issue appears to be a limitation of sails support for
type:'number'
From the official sails documentation on model attributes. The following attribute types are supported:
string
text
integer
float
date
datetime
boolean
binary
array
json
mediumtext
longtext
objectid
As an alternative to using the javascript
Number()
function, you could consider using
parseInt()
for known integer values or
parsefloat()
for known floating point numbers.

Creating a mongoose model containing array types return undefined for those in my docker container

I have a mongoose model containing 2 properties Array of String and some others:
var Model = new Schema({
_id: {type: Number, required: true, unique: true},
name: {type: String, required: true},
oneList: [String],
anotherList: [String]
});
I create the model:
var model = new Model({
_id: 1,
name: 'model',
oneList: ['a', 'b'],
anotherList: ['c', 'd']
})
but when I inspect the model all the list are undefined:
model._doc === {
_id: 1,
name: 'model',
oneList: undefined,
anotherList: undefined
}
I tried some variations:
change the model definition from [String] to [ ]
create separately the data outside the model then pass it
create the model without list data then add it to the model
Even when I create an empty model:
var model = new Model({})
model._doc.oneList === undefined
model._doc.anotherList === undefined
Context:
The problem occurs on a docker container but not on my local machine
node: v4.4.7
mongoose: v4.6.0
GitHub
I had the same issue, apparently when you have a nested array within your model, mongoose has an open issue 1335 that saves an empty array when a property references a schema. I experimented with presaves to force the property to be an empty array if the property's length is 0 or undefined.
Also be careful when specifying unique=true in the property's schema, as empty or undefined properties will violate the indexing and throw an error.
Note:
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/test');
var barSchema = new mongoose.Schema({
baz: String
});
var fooSchema = new mongoose.Schema({
bars: [barSchema]
});
var Foo = mongoose.model('Foo', fooSchema);
var foo = new Foo();
console.log(foo); // { _id: 55256e20e3c38434687034fb, bars: [] }
foo.save(function(err, foo2) {
console.log(foo2); // { __v: 0, _id: 55256e20e3c38434687034fb, bars: [] }
foo2.bars = undefined;
foo2.save(function(err, foo3) {
console.log(foo3); // { __v: 0, _id: 55256e20e3c38434687034fb, bars: undefined }
Foo.findOne({ _id: foo3._id }, function(err, foo4) {
console.log(foo4); // { _id: 55256e20e3c38434687034fb, __v: 0, bars: [] }
mongoose.disconnect();
});
});
});
You can do like this:
var schema = new Schema({
_id: {
type: Number,
required: true,
unique: true
},
name: {
type: String,
required: true
},
oneList: [String],
anotherList: [String]
}),
Model = mongoose.model('Model', schema),
m = new Model();
m._id = 1;
m.name = 'model';
m.oneList.push('a', 'b');
m.anotherList.push('c', 'd');

How to add new attribute in data object?

For arrays vue.js good works with default array methods: push, pop, shift, reverse... But how to create new item in object and paste some value?
new Vue({
el: '#demo',
data: {
items: [
{
start: '12:15',
end: '13:15',
name: 'Whatch Vue.js Laracast',
description: 'Watched the Laracast series on Vue.js',
tags: ['learning', 'Vue.js', 'Laracast', 'PHP'],
note: "Vue.js is really awesome. Thanks Evan You!!!"
},
{
start: '13:15',
end: '13:30',
name: "Rubik's Cube",
description: "Play with my Rubik's Cube",
tags: ['Logic', 'Puzzle', "Rubik's Cube"],
note: "Learned a new algorithm."
}
],
},
methods: {
addItemAttribute: function(name, text) {
this.items[0][name] = text;
}
}
});
You can add properties to a Vue object similar to a normal javascript object. Reference: https://vuejs.org/api/#Options-Data
var obj = { a: 1};
obj.a // 1
obj.b // undefined
obj.b = 2;
// Now obj.b is defined
obj.b // 2
You can access data from Vue object like this
var vm = new Vue({
data: {
a: 1,
b: 2
}
});
Now we can access data from $data property
vm.$data.a === 1;
// You can add a new property
vm.$data.c = 3;
// it can be any object
vm.$data.d = {id: 1, name: 'ABC'}
You get and update props of a component using component.options.props
var component = Vue.component('props-demo-advanced', {
props: {
// just type check
size: Number,
// type check plus other validations
name: {
type: String,
required: true,
// warn if not two way bound
twoWay: true
}
}
});
Let's say, we want to make name twoWay binding to false
component.options.props.name.twoWay = faslse
Or even you can change the entire object.
component.options.props = {
size: Number,
type: {
type: Number,
required: false,
twoWay: false
}
}
I don't know the above configurations are correct or not, I am just trying to convey the idea.
You should learn Javascript first, how objects are made, how to change them etc.

Nested Javascript Object : recursion and "complex" transformation (with lodash)

I apologize in advance for the complex example here; I tried to trim it down as much as I could to illustrate what I try to achieve
I have a complex structure that I need to traverse and transform based on some conditions; Here's an (short) example of the structure that should cover most scenarios:
{ PROP1: {
metadata: Object, // somewhere deeper in metadata I have a `value` key
parent: { $ref: String },
employee: {
parent: { $ref: String },
id: String,
metadata: Object,
products: {
metadata: Object,
model: { $ref: String },
type: 'array',
value: ["String", "String" , "String"] ,
[...]
},
model: {
id: String,
email: {
metadata: Object,
value: 'a#b.com',
type: 'string',
validity: Object,
[...]
},
name: {
firstName: {
metadata: Object,
value: 'John',
type: String,
validity: Object,
[...]
},
lastName: {
metadata: Object,
value: 'Smith',
type: String,
validity: Object,
[...]
},
}
},
operations: {
id: String,
items: [
{ action: {value: "UPDATE", model: {$ref: String }, [...] },
{...}
],
metadata: Object,
[...]
}
}
},
PROP2: {
// similar as PROP1
},
[... and so on ...]
}
I basically need to clean that up before sending it to the backend;
Whenever a value contains $ref, I don't want the key/val pair (e.g.: PROP1.parent is of no use and can be omitted)
whenever a value contains value, I need to omit everything else and move the value of value as the value of key (e.g.: PROP1.employee.products should equal ['String', 'String', 'String'])
keys like id, metadata, validity (etc) can be completely omitted regardless of its content
So the end result should be along those lines:
{ PROP1: {
employee: {
products: ['item','item','item'],
model: {
email: 'a#b.com',
name: { firstName: 'John', lastName: 'Smith'},
},
operations: [
{action: 'UPDATE'}
]
}
},
PROP2: { ... }
}
I tried lots of different approaches using different lodash methods but couldn't wrap my head around this...
Any help will be greatly appreciated
Thanks
In pseudo code, try something like this. Implement the specifics and post more info when you run into trouble.
var ignoreKeyArray = ["id", ...] // keys to ignore and remove
var newJSON = "{}";
for (property in JSON) {
var newProp = parseJSON(key, property);
insert newProp in newJSON object
}
var parseJSON = function (key, jsonBlob) {
if (key in ignoreKeyArray || jsonBlob contains value "$ref")
return "";
var jsonOut = key + ": {}";
for (child in jsonBlob) {
add parseJSON(child) to jsonOut;
}
return jsonOut;
}
If you have any questions, comment so I can extend the answer and clarify.

Categories

Resources