Develop Reference

Dynamic tables/models using Sequelize?

This question is more of a theoretical one, since I have not started implementation yet.
The situation is as follows:
I have an application in which users can upload structured data (like Excel, CSV etc.).
Due to requirements, I want to store them in the database, ideally creating a new table on the fly with the table name set to the file name and columns based on the file itself.
This initialisation is still doable with sequelize, I think.
However, as sequelize relies on models, then I am stuck as I am not sure what type and how many columns there will be, thus creating the need for something off 'Dynamic model' or 'Generic model'.
I am not sure how to do this, and I cannot find anything related when searching. I would appreciate your 2 cents on this approach, and if there are other ideas I am very eager to hear them.
Thanks in advance!

First, you need to add code to map your dynamic columns into Sequelize column definition like:
const columnDefinitions = [
{
field: 'field_name_from_csv',
type: DataTypes.INTEGER,
allowNull: true
},
{
field: 'field_name2_from_csv',
type: DataTypes.INTEGER,
allowNull: true
},
]
So you need to determine the data type of a certain column and create an appropriate column definition.
Second, you need to store all these mapping info in the special table(s) so you know what dynamic tables you have and how to register them in Sequelize.
Once you have a table name and column mappings you can register all tables as models in Sequelize:
// here you need to convert an array of mappings into the object where field names should be keys and column definitions should be values.
const columnMappingsAsObject = ...
const tableName = 'dynamic_table_name'
const dynamicModel = sequelize.define(tableName, columnMappingsAsObject, {
tableName
});
// now you can use it to get records and so on:
const records = await dynamicModel.findAll({})

couchdb views: return all fields in doc as map

I have a doc in couchDB:
{
"id":"avc",
"type":"Property",
"username":"user1",
"password":"password1",
"server":"localhost"
}
I want to write a view that returns a map of all these fields.
The map should look like this: [{"username","user1"},{"password","password1"},{"server","localhost"}]
Here's pseudocode of what I want -
HashMap<String,String> getProperties()
{
HashMap<String,String> propMap;
if (doc.type == 'Property')
{
//read all fields in doc one by one
//get value and add field/value to the map
}
return propMap;
}
I am not sure how to do the portion that I have commented above. Please help.
Note: right now, I want to add username, password and server fields and their values in the map. However, I might keep adding more later on. I want to make sure what I do is extensible.
I considered writing a separate view function for each field. Ex: emit("username",doc.username).
But this may not be the best way to do this. Also needs updates every time I add a new field.

First of all, you have to know:
In CouchDB, you'll index documents inside a view with a key-value pair. So if you index the property username and server, you'll have the following view:
[
{"key": "user1", "value": null},
{"key": "localhost", "value": null}
]
Whenever you edit a view, it invalidates the index so Couch has to rebuild the index. If you were to add new fields to that view, that's something you have to take into account.
If you want to query multiple fields in the same query, all those fields must be in the same view. If it's not a requirement, then you could easily build an index for every field you want.
If you want to index multiple fields in the same view, you could do something like this:
// We define a map function as a function which take a single parameter: The document to index.
(doc) => {
// We iterate over a list of fields to index
["username", "password", "server"].forEach((key, value) => {
// If the document has the field to index, we index it.
if (doc.hasOwnProperty(key)) {
// map(key,value) is the function you call to index your document.
// You don't need to pass a value as you'll be able to get the macthing document by using include_docs=true
map(doc[key], null);
}
});
};
Also, note that Apache Lucene allows to make full-text search and might fit better your needs.

DynamoDB: Query only every 10th value

I am querying data between two specific unixtime values. for example:
all data between 1516338730 (today, 6:12) and 1516358930 (today, 11:48)
my database receives a new record every minute. Now, when i want to query the data of last 24h, its way too dense. every 10th minute would be perfect.
my question now is: how can i read only every 10th database record, using DynamoDB?
As far as i know, theres no posibility to use modulo or something similar that pleases my needs.
This is my AWS Lambda Code so far:
var read = {
TableName: "user",
ProjectionExpression:"#time, #val",
KeyConditionExpression: "Id = :id and TIME between :time_1 and :time_2",
ExpressionAttributeNames:{
"#time": "TIME",
"#val": "user_data"
},
ExpressionAttributeValues: {
":id": event, // primary key
":time_1": 1516338730,
":time_2": 1516358930
},
ScanIndexForward: true
};
docClient.query(read, function(err, data) {
if(err) {
callback(err, null);
}
else {
callback(null, data.Items);
}
});
};

You say that you insert 1 record every minute?
The following might be an option:
At the time of insertion, set another field on the record, let's call it MinuteBucket, which is calculated as the timestamp's minute value mod 10.
If you do this via a stream function, you can handle new records, and then write something to touch old records to force a calculation.
Your query would change to this:
/*...snip...*/
KeyConditionExpression: "Id = :id and TIME between :time_1 and :time_2 and MinuteBucket = :bucket_id",
/*...snip...*/
ExpressionAttributeValues: {
":id": event, // primary key
":time_1": 1516338730,
":time_2": 1516358930,
":bucket_id": 0 //can be 0-9, if you want the first record to be closer to time_1, then set this to :time_1 minute value mod 10
},
/*...snip...*/
Just as a follow-up thought: if you want to speed up your queries, perhaps investigate using the MinuteBucket in an index, though that might come at a higher price.

I don't think that it is possible with dynamoDB API.
There are FilterExpression that contains conditions that DynamoDB applies after the Query operation, but before the data is returned to you.
But AFAIK it isn't possible to use a custom function. And build-in functions are poor.
As a workaround, you could mark each 10th item on the client side. And then query with checking attribute_exists (or attribute value) to filter them.
BTW, it would be nice to create the index for 'Id' attribute with sort key 'TIME' for improving query performance.

Select all the fields in a mongoose schema

I want to obtain all the fields of a schema in mongoose. Now I am using the following code:
let Client = LisaClient.model('Client', ClientSchema)
let query = Client.findOne({ 'userclient': userclient })
query.select('clientname clientdocument client_id password userclient')
let result = yield query.exec()
But I want all the fields no matter if they are empty. As always, in advance thank you

I'm not sure if you want all fields in a SQL-like way, or if you want them all in a proper MongoDB way.
If you want them in the proper MongoDB way, then just remove the query.select line. That line is saying to only return the fields listed in it.
If you meant in a SQL-like way, MongoDB doesn't work like that. Each document only has the fields you put in when it was inserted. If when you inserted the document, you only gave it certain fields, that document will only have those fields, even if other documents in other collections have different fields.
To determine all available fields in the collection, you'd have to find all the documents, loop through them all and build an object with all the different keys you find.
If you need each document returned to always have the fields that you specify in your select, you'll just have to transform your object once it's returned.
const fields = ['clientname', 'clientdocument', 'client_id', 'password', 'userclient'];
let Client = LisaClient.model('Client', ClientSchema)
let query = Client.findOne({ 'userclient': userclient })
query.select(fields.join(' '))
let result = yield query.exec()
fields.forEach(field => result[field] = result[field]);
That forEach loop will set all the fields you want to either the value in the result (if it was there) or to undefined if it wasn't.

MongoDB is schemaless and does not have tables, each collection can have different types of items.Usually the objects are somehow related or have a common base type.
Retrive invidual records using
db.collectionName.findOne() or db.collectionName.find().pretty()
To get all key names you need to MapReduce
mapReduceKeys = db.runCommand({
"mapreduce": "collection_name",
"map": function() {
for (var key in this) {
emit(key, null);
}
},
"reduce": function(key, stuff) {
return null;
},
"out": "collection_name" + "_keys"
})
Then run distinct on the resulting collection so as to find all the keys
db[mapReduceKeys.result].distinct("_id") //["foo", "bar", "baz", "_id", ...]

Filtering user Ids from an array of users

I wish to find all users, not the current User. A pair of users are stored within a "Room" array under this collection structure:
structure of each room (from another html page)
var newRoom = Rooms.insert({
owner : Meteor.userId(),
receiver : receiver,
people : [ owner , receiver ],
});
Collection.js (using dburles collection helper)
Rooms.helpers({
receiverName: function() {
return Meteor.users.findOne({ _id: this.receiver }).username;
}
});
html
<!-- **allRooms.html** Works fine, names appear -->
{{#each rooms}} {{receiverName}}{{/each }}
<!-- **roomDetail.html** names dont show, this.receiver undefined -->
{{receiverName}}
roomDetail js template helper
self.subscribe('room', Router.current().params._id);
self.subscribe('users');
});
How do I return and display the user's Id thats not the current user from the people field which is an array? I hope to show it in the child page (roomDetail).

Assuming:
Rooms is a collection, and you already have a room document to search on.
You only want to fetch a single user.
Give this a try:
// The list of userIds in room minus the current user's id.
var userIds = _.without(room.People, Meteor.userId());
// Assuming we want only one user...
var user = Meteor.users.findOne({ _id: userIds[0] });
Some thoughts about your original code:
You can't include references to Rooms in your Meteor.users selector unless Rooms is a field of users. Mongo has no notion of joins.
$ne isn't that you want. If you had 100 users published, and your array only contained 2 users (one of which you didn't want), using $ne would return 99 users.
Based on your comments, it looks like you need this in a collection helper. Maybe something like this:
Rooms.helpers({
findUser: function() {
var userIds = _.without(this.People, Meteor.userId());
return Meteor.users.findOne({ _id: userIds[0] });
},
});
And then elsewhere in your code, for a given room instance you could do:
room.findUser()