CouchDB view composing JSON objects with embedded arrays from two separated documents - javascript

Lets say I have two types of documents stored in my CouchDB database. First is with property type set to contact and second to phone. Contact type document have another property called name. Phone type have properties number and contact_id so that it can reference to contact person. This is trivial one to many scenario where one contact can have N phone numbers (I know that they can be embedded in single contact document, but I need to demonstrate one to many relationship with different documents).
Raw example data with Scott having 2 phone numbers and Matt having 1 number:
{_id: "fc93f785e6bd8c44f14468828b001109", _rev: "1-fdc8d121351b0f5c6d7e288399c7a5b6", type: "phone", number: "123456", contact_id: "fc93f785e6bd8c44f14468828b00099f"}
{_id: "fc93f785e6bd8c44f14468828b000f6a", _rev: "1-b2dd90295693dc395019deec7cbf89c7", type: "phone", number: "465789", contact_id: "fc93f785e6bd8c44f14468828b00099f"}
{_id: "fc93f785e6bd8c44f14468828b00099f", _rev: "1-bd643a6b0e90c997a42d8c04c5c06af6", type: "contact", name: "Scott"}
{_id: "16309fcd03475b9a2924c61d690018e3", _rev: "1-723b7c999111b116c353a4fdab11ddc0", type: "contact", name: "Matt"}
{_id: "16309fcd03475b9a2924c61d69000aef", _rev: "3-67193f1bfa8ed21c68e3d35847e9060a", type: "phone", number: "789456", contact_id: "16309fcd03475b9a2924c61d690018e3"}
Map function:
function(doc) {
if (doc.type == "contact") {
emit([doc._id, 1], doc);
} else if (doc.type == "phone") {
emit([doc.contact_id, 0], doc);
}
}
Reduce function:
function(keys, values) {
var output = {};
for(var elem in values) {
if(values[elem].type == "contact") {
output = {
"ID": values[elem]._id,
"Name": values[elem].name,
"Type": values[elem].type,
"Phones": []
};
} else if (values[elem].type == "phone") {
output.Phones.push({
"Number": values[elem].number,
"Type": values[elem].type
});
}
}
return output;
}
group_level is set to 1 because of keys in Map function. Now I can get my contacts with included phones for example like this:
http://localhost:5984/testdb2/_design/testview/_view/tv1?group_level=1
Or search for some contact with startkey and endkey like this:
http://localhost:5984/testdb2/_design/testview/_view/tv1?group_level=1&startkey=[%22fc93f785e6bd8c44f14468828b00099f%22]&endkey=[%22fc93f785e6bd8c44f14468828b00099f%22,{}]
Results look exactly how I want - contacts will have embedded phones according to one to many relationship. And here goes the question: Is this the right way of how to use MapReduce functions in CouchDB? Are there any notable performance issues when using this approach?

Generally speaking you use less disk space if you do not emit(...,doc).
You may want to reconsider having a reduce function at all. It's really not necessary to get at the data you need. For example, something along the lines of the following may use less disk space and perform better if you have a huge number of records.
Also, I believe it is against the grain of CouchDB to build up more data in a reduce function than your documents contain. You're not doing that in this case but you are following a pattern that might lead you into trouble later. It's called reduce for a reason. :-)
So something like this is more the CouchDB way:
function(doc) {
if (doc.type == "contact") {
emit([doc._id, 0], {
"Name": doc.name,
"Type": doc.type
});
} else if (doc.type == "phone") {
emit([doc.contact_id, 1], {
"Number": doc.number,
"Type": doc.type
});
}
}
Query it for a particular contact like so:
http://localhost:5984/testdb2/_design/testview/_view/tv1?
startkey=[%22fc93f785e6bd8c44f14468828b00099f%22, 0]
&endkey=[%22fc93f785e6bd8c44f14468828b00099f%22,1]
Granted, you don't get results in the same JSON structure as before but I believe this performs better within CouchDB.

This answer is completely apocryphal and anecdotal, but that's pretty much exactly how I've worked with one-to-many relationships in CouchDB. If there are any scaling issues, I haven't seen them yet. (But I admit I haven't tried too hard to find them.)
Although, in your map function why do you have your Phone sorted to come out first (0) before the Contact (1)? Your reduce function requires the opposite order.

Related

How do I use the Bitrix API to filter contacts using multiple values for one key?

I'm trying to return a list of contacts that could match multiple "PHONE" values. Right now I can get a list that matches one phone value but not an array of phone values. Here's what I have:
let contactList = await bitrix.call('crm.contact.list', {
"filter": {
"PHONE": phoneArray, //example ["1112223344","5556651234"]
},
"select": ["*","EMAIL","PHONE"]
});
I'm basing this off their API documentation that shows how to match one phone value here
There's also another article I found that mentions using a "LOGIC":"OR" in a filter that could potentially work. It's written in PHP so I'm not exactly sure how it translates to javascript.
You can use crm.duplicate.findbycomm (https://training.bitrix24.com/rest_help/crm/auxiliary/duplicates/crm.duplicate.findbycomm.php):
BX24.callMethod(
"crm.duplicate.findbycomm",
{
entity_type: "CONTACT",
type: "PHONE",
values: [ "8976543", "11223355" ],
},
function(result)
{
if(result.error())
console.error(result.error());
else
{
console.dir(result.data());
}
}
);
but there are limitations:
An array containing up to 20 e-mails or phone numbers
Maybe it will do use batch (https://training.bitrix24.com/rest_help/js_library/rest/callBatch.php)
Unfortunately crm.contact.list can't match multiple "PHONE" values

mongoose find() sorting and organizing returned results from a product database in js

I have a problem with organizing my mongoDB data to send to my page in my res and cant figure out how to do the correct js. Here is a simplified version of my schema
var productSchema = new mongoose.Schema({
medium: String,
brand: String,
group: String
});
Here is what a typical entry looks like
medium :"Acrylic",
brand :"liquitex",
group :"heavy body"
there are many more entries in the schema, but these are the only ones I need to be able to sort and organize the returned results with. The problem is I have a route that returns all colors in my database and I want to be able to display them in sections on my page that are grouped under Brand, and then has the individual colors listed under the correct group.
The problem is there are paints from other brands that fall into the heavy body group and so when I use a filter function to sort my data by group, some brands get mixed together. I cant filter by brand, because some brands have acrylic and watercolor so then those get lumped together.
I need some way to filter the returned results of a
mongoose.find({})
that can use the group data as a filter, but then filter those results by the brands so they get separated into the correct brand categories.
I have this so far:
this is all a stripped down version of my app.js file:
//finds all colors in the DB
Color.find({}).lean().exec(function( err, colors)
var groups = [];
// find all groups in the databse
colors.forEach( function(color){
groups.push(color["group"]);
});
//returns only unique names to filter out duplicates
var groupTypes = Array.from(new Set(groups));
var tempVariableBrands = [];
// this sorts all returned paints into their respective group, but we get paints from multiple brands under the same group and that is not good
groupTypes.forEach( function(group){
var name = group;
var result = colors.filter(obj => { return obj.group === group });
tempVariable.push( {group : name, result } );
});
// the tempVariable gets sent to my page like so
res.render("landing", {colorEntry:tempVariable} );
and this works fine to allow me to display each paint by its grouping, but that fails when there is more than one paint from a different manufacturer that is considered the same group like a "heavy body". This is my ejs on my page that works fine:
<% colorEntry.forEach( function(entry){ %>
<div class="brandBlock">
<div class="brandTitle">
<span><%=entry.result[0].brand%> - <%=entry.result[0].group%></span>
I for the life of me cant seem to figure out the combination of filter() and maybe map() that would allow this kind of processing to be done.
My database has like 600 documents, colors from a number of different manufacturers and I don't know how to get this as a returned structure: lets say this is a few colors in the DB that get returned from a mongoose find:
[{ medium: "Oil",
brand: "Gamblin",
group: "Artists oil colors"},
{ medium: "Acrylic",
brand: "Liquitex",
group: "Heavy Body"},
{ medium: "Acrylic",
brand: "Golden",
group: "Heavy Body"}
]
i need to organize it like this or something similar. It can be anything that just sorts this data into a basic structure like this, I am not confined to any set standard or anything, this is just for personal use and a site I am trying to build to learn more.
returnedColors = [ { brand: "Gamblin", group: "Artists oil colors", { 50 paints colors returned} },
{ brand: "liquitex" , group: "heavy body", { 20 paint colors returned } },
{ brand: "golden" , group: "heavy body",{ 60 paint colors returned} }
];
I am not a web developer and only write some web code every 6 months or so and have been trying how to figure this out for the last 2 days. I can't wrap my head around some of the awesome filter and map combo's i have seen and cant get this to work.
Any help or advice would be great. I am sure there are many areas for improvement in this code, but everything was working up until I entered paints that were from different brands that had the same group type and i had to try to rewrite this sorting code to deal with it.
It boils down to needing to be able to iterate over the entire set of returned documents from the DB and then sort them based off 2 values.
UPDATE:
I was able to get something that works and returns the data in the format that I need to be able to send it to my ejs file and display it properly. The code is rather ugly and probably very redundant, but it technically works. It starts off by using the group value to run over paints since each set of paints will have a group name, but can sometimes share a group name with a paint from another brand like "heavy body".
groupTypes.forEach( function(group){
var name = group;
var result = colors.filter(obj => { return obj.group === group });
// this gets brand names per iteration of this loop so that we will know if more than one brand of paint
// has the same group identity.
var brands = [];
result.forEach( function(color){
brands.push(color["brand"]);
});
// This filters the brand names down to a unique list of brands
var brandNames = Array.from(new Set(brands));
// if there is more than one brand, we need to filter this into two separate groups
if( brandNames.length > 1){
//console.log("You have duplicates");
brandNames.forEach( x => {
var tmpResult = [...result];
var resultTmp = result.filter(obj => { return obj.brand === x });
result = resultTmp;
//console.log("FILTERED RESULT IS: ", result);
tempVariable.push( {brand: x ,group : name, result } );
result = [...tmpResult];
});
}else{
tempVariable.push( {brand: result[0].brand ,group : name, result } );
}
});
if anyone can reduce this to something more efficient, I would love to see the "better" way or "right" way of doing something like this.
UPDATE2
Thanks to the answer below, I was put on the right track and was able to rewrite a bunch of that long code with this:
Color.aggregate([
{
$sort: { name: 1}
},
{
$group: {
_id: { brand: '$brand', group: '$group' },
result: { $push: '$$ROOT' }
}
},
{ $sort: { '_id.brand': 1 } }
], function( err, colors){
if(err){
console.log(err);
}else{
res.render("landing", {colorEntry:colors, isSearch:1, codes: userCodes, currentUser: req.user, ads: vs.randomAds()} );
}
});
Much cleaner and appears to achieve the same result.
Since you're using MongoDB, "right" way is to utilize an Aggregation framework, precisely, $group stage.
Product.aggregate([{
$group: {
_id: { group: '$group', brand: '$brand' },
products: { $push: '$$ROOT' }
}
}])
This will output array of objects containing every combination of brand and group, and push all relevant products to corresponding subarray.
Combine it with $project and $sort stages to shape your data further.

Query CosmosDb - where array contains item(s) from array

I don't know if there is a word for this, guess there is, but right now I couldn't explain it better than "where array contains item(s) from array".
It might sound weird, but actually it not (I think), and I'm having a hard time figuring out how I can do this in Azure CosmosDB.
Here goes. I have a document like this (simplified):
{
"id": "2a62fcf4-988f-4ebe-aedc-fb0c664b85d8",
"Title": "Seks års fængsel for overgreb",
"ZipCodes": [
{
"Code": "6500",
"Name": "Vojens",
"FoundViaTerm": "Vojens"
},
{
"Code": "6400",
"Name": "Sønderborg",
"FoundViaTerm": "Sønderborg"
},
{
"Code": "6700",
"Name": "Esbjerg",
"FoundViaTerm": "Esbjerg"
}
],
"_rid": "k1sVAPf7SQAMAAAAAAAAAA==",
"_self": "dbs/k1sVAA==/colls/k1sVAPf7SQA=/docs/k1sVAPf7SQAMAAAAAAAAAA==/",
"_etag": "\"00001000-0000-0000-0000-5a14898e0000\"",
"_attachments": "attachments/",
"_ts": 1511295374
}
Ok, now I want to query documents like this and find all, where ZipCodes.Code is in a list of zipcodes, ex. ('6500', '2700').
I'm puzzle here...
I found the ARRAY_CONTAINS method and it works, if I only come in with one zipcode - my problem is I come with a list.
Hope somebody can help, thanks in advance.
Per my experience , expr in ARRAY_CONTAINS (arr_expr, expr [, bool_expr]) method is not supported list arguments.
According to your situation , I suggest you use UDF in Cosmos DB.
I created 3 sample documents as your description.
[
{
"id": "1",
"zip": [
{
"code": "1111"
},
{
"code": "2222"
}
]
},
{
"id": "2",
"zip": [
{
"code": "2222"
},
{
"code": "3333"
}
]
},
{
"id": "3",
"zip": [
{
"code": "4444"
},
{
"code": "1111"
},
{
"code": "2222"
}
]
}
]
Please refer to the snippet of UDF code as below :
function test(zipcode){
var arrayList = ["1111","2222"]
var ret = false ;
for(var i=0 ;i <zipcode.length;i++){
if(arrayList.indexOf(zipcode[i].code)){
ret= true;
}else{
ret = false;
break;
}
}
return ret;
}
You could select zip array (select c.zip from c) ,then loop the results and invoke the UDF above in your code with the zip[i] arguments.
Hope it helps you.
Just for summary:
Use the IN operator from Cosmos DB SQL APIs to query entry which is included in the list condition.
Like
SELECT * FROM c WHERE c.ZipCodes[0].Code IN ("6500", "6700")
Or
SELECT DISTINCT c FROM c JOIN zc IN c.ZipCodes WHERE zc.Code IN ("2720", "2610")
I would like to propose another solution to this problem.
Use EXISTS with ARRAY_CONTAINS in this way:
SELECT * FROM c
WHERE EXISTS
(SELECT VALUE z FROM z in c.ZipCodes WHERE ARRAY_CONTAINS(["6500","6700"], z))
You can do something like this:
For each item in ZipCodes, you get a zip and compare with the array of codes you are checking. This, IMHO, is much better than using UDF.
{
query: '
SELECT DISTINCT value r
FROM root r
JOIN zip IN r.zipCodes
WHERE ARRAY_CONTAINS(#zipIds, zip, true)
',
parameters: [{name: "#zipIds", value: zipIds}]
}
The last param of ARRAY_CONTAINS tells the function to accept partial matches.
Apart from the fact that using UDF looks as the easier option, i would not use UDFs in your query's filter, since it compromises the performance of your query. I faced the same problem in my work environment, where things are designed to use UDFs to help in the queries, but the reality is that most of the times we are doing queries by using single values, and using UDF will actually result on the query not using the index. So in that case if you need to validate multiple values in the array, depending on the volume of values you need to validate, you can always write something like ARRAY_CONTAINS(c, 1) or ARRAY_CONTAINS(c, 2) or ....
It doesn't look so elegant solution, but will ensure that it will use the index and will do the best performance in your query.

Look if loopback model property array contains a string

I have a Loopback moddel that looks like this:
{
"name": "string",
"elements": [
"string"
]
}
Now I want to filter if elements property conatins a certain string.
Something like this:
User.find({
filter: {
where: {elements: $scope.objects[i].id} //doesn't work, I want sth like "element contains $scope.objects[i].id
}}, function (user) {
console.log(user);
});
Warning: This solution was meant to answer the question "how do I filter a list of objects". It was accepted so I can't remove it. I don't know anything about LoopBack which has performance implications I'm not privy to. So please keep searching if you are looking for a "LoopBack" best practice.
This seems like a javascript question to me. The elements property contains an array so you can filter that array with filter().
yourModel = { // <-- Using a plain object for demo.
"name": "string",
"elements": [
"string"
]
}
matchingElements = yourModel.elements.filter(function(elm){ return elm === $scope.objects[i].id});
didMyModelHaveTheElement = matchingElments.length > 0;

How can I select the latest version of an object from a ForerunnerDb collection

I have a collection which contains a series of objects generated over time. Since I have disparate types stored in the same collection, I have a TypeId and a UID per object (where the UID identifies objects that refer to the same entity over time). I am trying to choose the most recent object from the collection, and running into serious difficulties grasping how to do so without manually iterating a query result - something I'd rather avoid since I think it could become expensive when the collection gets larger.
For example:
var db; // assigned elsewhere
var col = db.collection("res");
col.primaryKey("resId");
col.insert({
resId: 1,
TypeId: "Person",
UID: "Bob",
Data: {Age: 20, Name:Bob}
});
col.insert({
resId: 2,
TypeId: "Person",
UID: "Bob",
Data: {Age: 25, Name:Bob}
});
col.insert({
resId: 3,
TypeId: "Car",
UID: "TeslaModelX",
Data: {Manufacturer: "Tesla", Owner:"Bob"}
});
col.insert({
resId: 4,
TypeId: "Person",
UID: "Bill",
Data: {Age: 22, Name:Bill}
});
From col, I want the query to select all objects with TypeId="Person" ranked by resId descending, i.e. I'd expect to select objects 4 and 2, in that order.
The collection above is contrived, but in reality I'd expect there to be certainly 000s of entries and potentially 0000s, with maybe 00s of versions of each UID. In other words, I'd rather not return the full collection of objects, grouped or otherwise, and iterate it.
I have tried the following, but since the $distinct operator is applied before the $orderBy one, this doesn't help:
col.find(
{
TypeId : {$eq : "Person"}
$distinct: { UID: 1}
},
{
$orderBy: {
resId : -1
}
}
);
I have in mind that I should be able to use the $groupBy, $limit and $aggregate clauses to identify the per group desired IDs, and then use a subquery to find the precise (non-aggregated) elements, but as yet I haven't managed to get anything to do what I want. Any ideas?
My current solution is to include a Deleted property amongst my objects, and set it to true for all existing non-deleted objects in the DB before I insert new entries. This lets me do what I want but also stops me from, for instance, choosing the best available within a known timeframe or similar.
You can do this like:
var tmpObj = {};
col.sort({resId: -1}, coll.find({
"TypeId": "Person"
})).filter(function (doc) {
return col._match(doc, {
$distinct: {
UID: 1
}
}, {}, 'and', tmpObj);
});
It's a bit dirty since it's not neatly wrapped up in a single command, but it's as clean as you'll get it in ForerunnerDB v1.x.
Version 2 will have a new query pipeline system that would allow for exactly this sort of usage, something like this:
col.pipeline()
.find({"TypeId": "Person"})
.orderBy({"resId": 1})
.distinct({"UID": 1})
.then(function (err, data) {
console.log(data);
});
Source: I'm the developer of ForerunnerDB.

Categories

Resources