This question is more of a theoretical one, since I have not started implementation yet.
The situation is as follows:
I have an application in which users can upload structured data (like Excel, CSV etc.).
Due to requirements, I want to store them in the database, ideally creating a new table on the fly with the table name set to the file name and columns based on the file itself.
This initialisation is still doable with sequelize, I think.
However, as sequelize relies on models, then I am stuck as I am not sure what type and how many columns there will be, thus creating the need for something off 'Dynamic model' or 'Generic model'.
I am not sure how to do this, and I cannot find anything related when searching. I would appreciate your 2 cents on this approach, and if there are other ideas I am very eager to hear them.
Thanks in advance!
First, you need to add code to map your dynamic columns into Sequelize column definition like:
const columnDefinitions = [
{
field: 'field_name_from_csv',
type: DataTypes.INTEGER,
allowNull: true
},
{
field: 'field_name2_from_csv',
type: DataTypes.INTEGER,
allowNull: true
},
]
So you need to determine the data type of a certain column and create an appropriate column definition.
Second, you need to store all these mapping info in the special table(s) so you know what dynamic tables you have and how to register them in Sequelize.
Once you have a table name and column mappings you can register all tables as models in Sequelize:
// here you need to convert an array of mappings into the object where field names should be keys and column definitions should be values.
const columnMappingsAsObject = ...
const tableName = 'dynamic_table_name'
const dynamicModel = sequelize.define(tableName, columnMappingsAsObject, {
tableName
});
// now you can use it to get records and so on:
const records = await dynamicModel.findAll({})
Related
I have a JSON file which is 3000+ lines. What I'd like to do is create a NoSQL database with the same structure (it has embedded documents between 3-5 levels deep). But I want to add information to each level and create a schema for each item, so that I can go back at a later stage and update the information fields, and even have users log-in and change their own values.
I am using JavaScript to write a script that will iterate through the file and upload to MongoDB the schema that I want, based on the information at each level. But I'm struggling to write the code that does this efficiently. At this stage, I'm just wasting too much time trying this and that, and want to move on to the next step of my site.
Below is an example of the file. Basically, it's a bunch of embedded documents, and then at the final level (which will be at a different depth depending on which document it's in), there is an array where each of the fields is a string.
How can I use this data to create a MongoDB database while adding a schema to each item, but keeping the hierarchical nature of the documents? I want all of the documents to have one schema, and then each of the strings at the final depth to have their own, separate schema as well. I can't think of an efficient way to iterate through.
Example from the JSON file:
{
"Applied Sciences": {
"Agriculture": {
"Agricultural Economics": [
"Agricultural Environment And Natural Resources",
"Developmental Economics",
"Food And Consumer Economics",
"Production Economics And Farm Management"
],
"Agronomy": [
"Agroecology",
"Biotechnology",
"Plant Breeding",
"Soil Conservation",
"Soil Science",
"Theoretical Modeling"
],
Here's my schema for all but the strings at the end:
name: String,
completed: Boolean,
category: "Field",
items: {
type: Array
},
description: String,
resources: {
type: Array
}
};
And my rough code which at this stage just iterates through. I'm trying to use the same function call to create the Arrays in the schema, but I'm just not up to that stage yet because I can't even iterate properly through:
function createDatabase(data){
for (field in data){
items = {};
for (field in data){
if (typeof data[field] == "object");
items[field] = createDatabase(data[field]);
};
return items;
}
I am trying to uniquely store data for each server in a database, I want to be able to store multiple of the same data object. I am only able to store one because if I attempt to store another one it just replaces it, which is the issue here.
Schema
const schema = new mongoose.Schema({
reactionRole: {
type: Object,
required: false,
}
});
Screenshot of the Data
Trying to Accomplish
I want to be able to store the same object here but with different data obviously, would I have to make the schema take a Array and just insert it? I am not really sure how to work around this, thanks!
Example
This is how I want the data to be, the option to be able to add more onto the document instead of replacing it. Should I use an array or what's a solution?
reactionRole: {
<Role.name> config 1: {
Enabled: true,
Added_By: id,
MessageID: msg.id,
Emoji: <emoji>,
Role: id,
},
<Role.name> config 2: {
Enabled: true,
Added_By: id,
MessageID: msg.id,
Emoji: <emoji>,
Role: id,
}
}
I resolved this. I changed the schema to an array and sent the data as objects.
I have a User model that includes a json data type. I'm using MySQL as my database, but when looking through their documentation for how to query using JSON, there appears to be nothing on MySQL.
const User = db.define("User", {
UserId: {
primaryKey: true,
type: Sequelize.INTEGER
},
colors: {
type: Sequelize.JSON,
allowNull: false
}
}
I have Users that have an id, and an array with their favorite colors in it.
UserId | colors
1 |["black, "blue"]
2 |["blue"]
3 |["black"]
I want to query for all users that have the favorite color blue. I tried querying like this,
User.findAll({ where: {colors: ["blue"]} })
and expected to be returned users 1 and 2, but this doesn't work and I have not found any sources online that shows what to do. Some stackoverflows have solutions for JSON objects, like this Sequelize, MySQL - Filtering rows in table with JSON column values, but I can't find any sources for arrays. Any solutions?
You can use sequelize.fn and sequelize.col methods to specify an SQL function call and a table column, respectively. This way you can create your own where condition.
User.findAll({
where: sequelize.where(sequelize.fn('JSON_CONTAINS', sequelize.col('colors'), sequelize.literal('blue'), sequelize.literal('$')), 1)
});
It corresponds to below mysql query:
// SELECT * FROM Users WHERE JSON_CONTAINS(colors, 'blue', '$') = 1;
The answer should be:
User.findAll({
where: sequelize.where(sequelize.fn('JSON_CONTAINS', sequelize.literal('colors'), '"blue"', '$'), 1)
});
Note:
The 2nd parameter of JSON_CONTAINS should be string.
Because the data of question is an array of string, so we need to add a " for searching keyword (e.g: "blue").
I have a working model with Postgres and sequelize in NodeJS. Say the model is Person and has name and age fields. Now I want to dynamically inspect the model class and obtain information about it's attributes, like their name and most of all type.
Using Person.attributes
I get some information:
name:
{ type:
{ options: [Object],
_binary: undefined,
_length: 255 },
But as you can see the type object does not inform about name being a varchar or boolean.
Does anyone know, how to get this information with sequelize
You can iterate over rawAtributes of Model
for( let key in Model.rawAttributes ){
console.log('Field: ', key); // this is name of the field
console.log('Type: ', Model.rawAttributes[key].type.key); // Sequelize type of field
}
So the example for name being a Sequelize.STRING would be
Field: name
Type: STRING
Or you can do almost the same iteration but instead using Model.rawAttributes[key].type.key you can use Model.rawAttributes[key].type.toSql(), which would generate this result
Field: name
Type: VARCHAR(255)
EDIT
Accessing defaultValue of field:
Model.rawAttributes[field].defaultValue
Checking if field allows NULL values:
Model.rawAttributes[field].allowNull
You are looking for native type information, it seems.
I'm not familiar with Sequelize, except I know it uses node-postgres driver underneath, which automatically provides the type information with every query that you make.
Below is a simple example of dynamically getting type details for any_table, using pg-promise:
var pgp = require('pg-promise')(/*initialization options*/);
var db = pgp(/*connection details*/);
db.result('SELECT * FROM any_table LIMIT 0', [], a => a.fields)
.then(fields => {
// complete details for each column
})
.catch(error => {
// an error occurred
});
There are several fields available for each column there, including name and dataTypeID that you are looking for ;)
As an update, following the answer that does use Sequelize for it...
The difference is that here we get direct raw values as they are provided by PostgreSQL, so dataTypeID is raw type Id exactly as PostgreSQL supports it, while TYPE: STRING is an interpreted type as implemented by Sequelize. Also, we are getting the type details dynamically here, versus statically in that Sequelize example.
item.rawAttributes[key].type.key === this.DataTypes.JSONB.key;
#piotrbienias Is above comparison valid ?
(Sorry for writing here as I can't add comments)
I'm curious if there is a way to change the id used when specifying withRelated on a model?
Here's the situation:
I'm working on a platform that has a calendar component and that calendar needs to support recurring appointments. Rather than creating every potential instance of a recurring appointment, I'm generating these recurrences on the fly, which leads to much less maintenance going forward. My Appointment model also has relations that are associated with each instance. An Appointment can have many services, etc. When loading these relations using withRelated on an appointment that exists in the database (is not dynamically generated), the id of that appointment is passed into whatever relation withRelated attempt to load and works great.
However, with these dynamically generated appointments, I actually need to load the relations using the id of the parent appointment from which these instances were generated. Unfortunately, I can't seem to find a way to either specify to use that parent id or a place where I can intercept and change it before the relation is loaded.
My situation is an odd one, but it would be a simple fix if I could change what is passed in when withRelated is called.
Is this possible?
I'm also open to other suggestions.
Goal/Desired Results
Just to be clear, my goal is fairly simple. When Bookshelf loads a many-to-many relationship, it uses the idAttribute of the joining model in order to load related relations.
For example, Appointment is my primary model and has a many-to-many relationship with Pets. Therefore, when I load the pets relation, Bookshelf does a join on the appointment_pets table looking for all records where appointment_pet.appointment_id IN (?) where the parameters are ids from the appointment table.
My desired goal is to be able to substitute appointment.id with appointment.parent_id when looking up these relations as in my situation, these dynamic records don't actually exist in the database yet and therefore need to use the information attached to the parent record, which does exist.
Updates
I was finally able to get different ids injected into the above query by temporarily changing the idAttribute on the model from id to parent_id. The query is correct and parameters being passed are also correct, but the relationships are empty when the model is returned.
// appointment model
pets() {
return this.belongsToMany('Pet', 'appointment_pet', 'parent_id');
},
},
// Returns all appointments
fetchAll(businessId, options) {
options = options || {};
// If a start and end date are passed, return all appointments
// within that range, with recurrences expanded.
if (!_.isUndefined(options.startDate) &&
!_.isUndefined(options.endDate)) {
const startDate = Moment(options.startDate);
const endDate = Moment(options.endDate);
// temporarily change id attribute so `parent_id` is used when looking up the relation
Appointment.prototype.idAttribute = 'parent_id';
return Appointment
.query((qb) => {
qb.from(Bookshelf.knex.raw('get_appointments(?, ?, ?) AS appointment', [businessId, startDate.toISOString(), endDate.toISOString()]));
})
.fetchAll({
withRelated: [{
pets: (qb) => {
qb.select(['appointment_id as parent_id', 'pet.*']);
}
}]
});
}
// Otherwise, return all appointments
return Appointment.findAll({ businessId: businessId }, options);
},
I see the following debug output:
{ method: 'select',
options: {},
bindings:
[ null,
'35a2941c-d73f-4cf3-87de-bfbcbc92fbc5', <-- these are all `appointment.parent_id`
'5de28a57-ce4c-4fcc-865a-54cf97e08c6c',
'bf4b6784-b96a-4321-8335-e449aa8dcda1',
'695edc54-3a93-42a5-a3e0-b331f26912cf',
'76a99204-9659-4270-904c-2c42e5c40a15',
'28437e3c-abc1-4fd6-8c26-5543c09ab730',
'350da29d-0e82-4e6c-b9fe-be31c64655b4',
'bcaee041-c9ec-4a34-8c0d-8f21d2d79197',
'91380fcc-2f0d-4344-a24b-dc37abe3931f',
'31796797-5041-4fae-815b-0f2c368654b0',
'94800af3-fd23-4b59-af88-15b1f622e8e1',
'aa8c202d-249e-4ffc-ab55-e535f2c2121f',
'8a1fd093-45a6-4864-8c52-ea8851740f77',
'5f91ad6f-4231-4d18-b718-d99fcc17a18b',
'5f879e43-a44f-41dd-b9dd-11f06af4cc8d' ],
sql: 'select "pet".*, "appointment_pet"."parent_id" as "_pivot_parent_id", "appointment_pet"."pet_id" as "_pivot_pet_id" from "pet" inner join "appointment_pet" on "appointment_pet"."pet_id" = "pet"."id" where "appointment_pet"."parent_id" in (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)'}
Running this command in postgres, I see the correct output, but still no relations. I saw #1203 and tried adding the parent_id explicitly to the select, but still no luck.
Unfortunately I don't have enough data to replicate your problem. I tried it on a basic one-to-many example, and I think the easiest way is to define a new relation in the model itself. E.g.:
var Genre = bookshelf.Model.extend({
tableName: 'genres',
movie: function() {
return this.hasMany(Movie);
},
movie_parent: function() {
return this.hasMany(Movie, 'parent_id');
}
});
var Movie = bookshelf.Model.extend({
tableName: 'movies'
genre: function() {
return this.belongsTo(Genre);
},
genre_parent: function() {
return this.belongsTo(Genre, 'id');
}
});
So then, you can query it reasonably simple, in the withRelated property.
new Movie().fetchAll({
withRelated: ['genre_parent'] //can be replaced with withRelated: ['genre']
}).then(function(movies) {
res.json(movies);
});
If this is not the solution you're looking for, please provide additional data (database structure would be interesting to see)