Redis Search unable to query by text field - javascript

I have a redis database where i store has hset entries with
A MAC address called mid in the form XX:XX:XX:XX:XX:XX
A timestamp
A geographical position called position
A text payload called message
I would like to be able to index those objects by the mid, the timestamp and the position (I will never query from information about the message).
This is the code for the schema of the index,
await client.ft.create(
'idx:cits',
{
mid: {
type: SchemaFieldTypes.TEXT
},
timestamp: {
type: SchemaFieldTypes.NUMERIC,
sortable: true
},
position: {
type: SchemaFieldTypes.GEO
}
},
{
ON: 'HASH',
PREFIX: 'CITS'
}
)
And I insert new entries using
await client.hSet('CITS:19123123:0:0:00.00:5e:00:53:af', {
timestamp: 19123123,
position: '0,0',
mid: '00:00:5e:00:53:af',
message: 'payload'
})
I can perfectly query by timestamp and position both in the javascript code using
await client.ft.search('idx:cits', '#timestamp:[100 19123180] #position:[0 0 10 km]')
and in the redis-cli using
FT.SEARCH idx:cits "#timestamp:[100 19123180] #position:[0 0 10 km]"
But it does not work when querying by the mid field.
I have tried both
await client.ft.search('idx:cits', '#mid:"00:00:5e:00:53:af"')
and in redis-cli
FT.SEARCH idx:cits idx:cits '#mid:"00:00:5e:00:53:af"'
I have also tried exchanging the " and the ' as well as eliminating them to see if it worked without any result.
I have also tried storing the mac addresses as XX.XX.XX.XX.XX.XX instead of XX:XX:XX:XX:XX:XX and did not result either.
I have also seen that it will not work for 00:00:5e:00:53:af but instead it works for ff.ff.ff.ff.ff.ff.
I'm not sure why I am not able to query by the mid, I would really appreciate if someone could help me out.
Here is an example script
const { createClient, SchemaFieldTypes } = require('redis')
const client = createClient()
async function start(client) {
await client.connect()
try {
// We only want to sort by these 3 values
await client.ft.create(
'idx:cits',
{
mid: {
type: SchemaFieldTypes.TEXT
},
timestamp: {
type: SchemaFieldTypes.NUMERIC,
sortable: true
},
position: {
type: SchemaFieldTypes.GEO
}
},
{
ON: 'HASH',
PREFIX: 'CITS'
}
)
} catch (e) {
if (e.message === 'Index already exists') {
console.log('Skipping index creation as it already exists.')
} else {
console.error(e)
process.exit(1)
}
}
await client.hSet('CITS:19123123:0:0:00.00.5e.00.53.af', {
timestamp: 19123123,
position: '0,0',
mid: '00.00.5e.00.53.af',
message: 'payload'
})
await client.hSet('CITS:19123123:0.001:0.001:ff.ff.ff.ff.ff.ff', {
timestamp: 19123123,
position: '0.000001,0.000001',
mid: 'ff.ff.ff.ff.ff.ff',
message: 'payload'
})
const results = await client.ft.search('idx:cits', '#mid:00.00.5e.00.53.af')
console.log(results)
await client.quit()
}
start(client)
Thank you!

The TEXT type in RediSearch is intended for full-text search—i.e. text meant for humans to read and not meant for computers to parse. So, it removes and doesn't search for things like punctuation—i.e. periods, colons, commas, etc.—and common words like a, and, or the.
You want to use a TAG type instead. You can think of these like a tag cloud on a blog post. A TAG field should contain a comma-separated string of values—the tags. If there's just a single value, it's just a CSV of one.
To create an index that uses a TAG field, do this:
await client.ft.create('idx:cits', {
mid: { type: SchemaFieldTypes.TAG },
timestamp: { type: SchemaFieldTypes.NUMERIC, sortable: true },
position: { type: SchemaFieldTypes.GEO }
})
To search a TAG field, use curly braces. Like this:
await client.ft.search('idx:cits', '#mid:{00:00:5e:00:53:af}')
Hope this helps.

Related

How to update an array field on mongodb by pushing it if the element is not present and if it is present update it?

I am working on a IT ticketing system where every time a new comment or a note has been added to a ticket a notification needs to be send to the user who is following the ticket. Below code only inserts new ticket in list of tickets followed by the user if it is not already present, however if it is present it ignores it. What I need to do is if the ticket that has just been updated is already present, change clicked field to false. In my application's frontend, when user clicks the notification icon it will change clicked to TRUE but when a new comment is added clicked needs to be changed to FALSE so that the user gets notification that comment has been added to the same ticket. How do I go about achieving it?
const ReqNotificationSchema = new Schema({
user: { type: Schema.Types.ObjectId, ref: "User" },
notifications: [
{
request: { type: Schema.Types.ObjectId, ref: "Request" },
clicked: { type: Boolean, default: false },
},
],
});
if(updated){
await ReqNotificationModel.findOneAndUpdate(
{
user: follower.user,
"notifications.request": { $ne: updated._id },
},
{
$push: { notifications: { request: updated._id, clicked: false } },
},
{ new: true }
);
}
I wasn't able to do it in one step , so I tried in a 2 step approach.
const notification = await ReqNotificationModel.findOne({
user: follower.user,
});
let index = notification.notifications
.map((obj) => obj.request.toString())
.indexOf(updated._id.toString());
notification.notifications[index].clicked = false;
await notification.save();
});

Can't update(nor save) existing row using Sequelize

When trying to .update() or .save() a row I'm getting this error:
Unhandled rejection Error: You attempted to save an instance with no primary key,
this is not allowed since it would result in a global update
I tried all 4 ways the docs uses as examples(with and without defining the attributes I wanna save), nothing worked.
This is my actual code for updating:
Sydney.databases.guilds.findOrCreate({
attributes: ['guildLocale'],
where: {
guildID: _guild.id,
},
defaults: {
guildID: _guild.id,
guildLocale: 'en_US',
guildPrefix: '?',
},
}).spread((guild, created) => {
guild.update({guildLocale: args[1]})
.then(() => console.log(7))
.catch((e) => throw e);
});
And this is the guild model:
let model = sequelize.define('guild', {
guildID: {
field: 'guild_id',
type: DataTypes.STRING,
primaryKey: true,
},
guildLocale: {
field: 'guild_locale',
type: DataTypes.STRING,
},
guildPrefix: {
field: 'guild_prefix',
type: DataTypes.STRING,
},
}, {tableName: 'guilds'});
What am I missing here?
I had the same problem. It occurs when you specify the attributes you want to fetch from the database without including the primary key in the attributes. And when you attempt to save, it will throw the following error:
Unhandled rejection Error: You attempted to save an instance with no primary key, this is not allowed since it would result in a global update
So the simple solution is to include the primary key in the attributes like this:
Sydney.databases.guilds.findOrCreate({
attributes: ['guildLocale', 'guildID'], // include guideID here!!
where: {
guildID: _guild.id,
},
defaults: {
guildID: _guild.id,
guildLocale: 'en_US',
guildPrefix: '?',
},
}).spread((guild, created) => {
guild.update({guildLocale: args[1]})
.then(() => console.log(7))
.catch((e) => throw e);
});
Ok, so the problem seems that was the attributes: ['guildLocale'] in the .findOrCreate() method. I can't tell why, to be honest, gonna read that doc again to be sure, but I'll leave this answer if another newbie is getting trouble with this, ;P...

How to search all keys inside MongoDB collection using only one keyword

Is there a way for MongoDB to search an entire collection's keys' contents using only a single search keyword?
Suppose I have the following collection (let's call it foodCollection):
{
name: "Chocolate Mousse Cake",
type: "Cake"
},
{
name: "Mother's Cookies",
type: "Cookies"
},
{
name: "Dark Bar",
type: "Chocolate"
}
I want my search to look for matches that contain "Chocolate", meaning it should return "Chocolate Mousse Cake" and "Dark Bar".
I'm trying to do this using the ff: code:
Client-side controller
// Search Products
$scope.searchProduct = function () {
$http.get('/api/products/search/' + $scope.searchKeyword).success(function(data){
console.log(data);
})
.error(function(err) {
console.log("Search error: " + err);
});
}
Express.js
app.get('/api/products/search/:param', productController.search); // Search for product
Server-side controller (I used this reference from the MongoDB docs):
// Search
module.exports.search = function(req, res) {
console.log("node search: " + req.body);
Product.find({ $or: [{productName: req.body},
{productType: req.body}]
}, function(err, results) {
res.json(results);
});
}
When I executed this, I got nothing. Am I missing something?
Any help would be greatly appreciated. Thank you.
UPDATE (FINAL)
Finally solved this thanks to Joydip's and digit's tips. Here's my solution in case somebody else gets the same problem as I did:
Client-side controller
$scope.searchProduct = function () {
if ($scope.searchKeyword == '') {
loadFromMongoDB(); // reloads original list if keyword is blank
}
else {
$http.get('/api/products/search/' + $scope.searchKeyword).success(function(data){
if (data.length === 0) {
$scope.showNoRec = true; // my flag that triggers "No record found" message in UI
}
else {
$scope.showNoRec = false;
$scope.productList = data; // passes JSON search results to UI
}
});
}
}
Express.js
app.get('/api/products/search/:keyword', productController.search); // Search for product
Mongoose schema
var mongoose = require('mongoose');
var schema = new mongoose.Schema({
productName: String,
productType: String,
productMaker: String,
productPrice: Number,
createDate: Date,
updateDate: Date
});
schema.index({productName: "text", productType: "text", productMaker: "text"});
Server-side controller
module.exports.search = function(req, res) {
Product.find({$text: {$search : req.params.keyword}}, function(err, results){
res.json(results);
})
}
Thank you everyone for your help. :)
You can try by creating an Index:
db.yourollection.createIndex({"productName":1,"productType":1})
And then by searching for the value, Example:
Product.find({$text:{$search: 'Chocolate'}},{productName:1, productType:1});
If you want to search all key, then you can use
db.foodCollection.createIndex( { name: "text", description: "text" } )
then search by
db.foodCollection.find({ $text: { $search: "choco" } })

How to build up an elasticsearch query in a better way?

I am currently using the elasticsearch helper to run queries on my elasticsearch database. Currently I am using vue.js to help build this application.
When I am paginating my results I use this query:
this.client.search({
index: 'node',
type: 'vakantie',
from: 12 * index,
size: this.size,
}).then(function (resp) {
this.travels = resp.hits.hits;
this.currentPage = index;
}.bind(this), function (err) {
console.trace(err.message);
});
I also have an input box above my results that a user can type a search term into and it will instantly filter down results using this query:
index: 'node',
type: 'vakantie',
from: 0,
size: this.size,
body: {
query: {
match_phrase_prefix: {
title: {
query: this.query,
slop: 10,
max_expansions: 50
}
}
},
highlight: {
fields: {
title: {}
},
pre_tags: ["<span class='highlight'>"],
post_tags: ["</span>"]
}
}
I have filters and sort methods in place as well, and I know how to use a query with elasticsearch to combine multiple search queries.
Obviously I don't want to write a search query for every combination of filter + input + sort possible. Is there a better way to build up my search queries than using a javascript object like this?
Hopefully I am getting my point across, I have a bit too much custom code to simply paste it here.

Add multiple records to model's collection Sailsjs

I have the following models in my Sailsjs application with a many-to-many relationship:
event.js:
attributes: {
title : { type: 'string', required: true },
description : { type: 'string', required: true },
location : { type: 'string', required: true },
maxMembers : { type: 'integer', required: true },
currentMembers : { collection: 'user', via: 'eventsAttending', dominant: true },
creator : { model: 'user', required: true },
invitations : { collection: 'invitation', via: 'eventID' },
tags : { collection: 'tag', via: 'taggedEvents', dominant: true },
lat : { type: 'float' },
lon : { type: 'float' },
},
tags.js:
attributes: {
tagName : { type: 'string', unique: true, required: true },
taggedEvents : { collection: 'event', via: 'tags' },
},
Based on the documentation, this relationship looks correct. I have the following method in tag.js that accepts an array of tag strings, and an event id, and is supposed to add or remove the tags that were passed in:
modifyTags: function (tags, eventId) {
var tagRecords = [];
_.forEach(tags, function(tag) {
Tag.findOrCreate({tagName: tag}, {tagName: tag}, function (error, result) {
tagRecords.push({id: result.id})
})
})
Event.findOneById(eventId).populate('tags').exec(function(error, event){
console.log(event)
var currentTags = event.tags;
console.log(currentTags)
delete currentTags.add;
delete currentTags.remove;
if (currentTags.length > 0) {
currentTags = _.pluck(currentTags, 'id');
}
var modifiedTags = _.pluck(tagRecords, 'id');
var tagsToAdd = _.difference(modifiedTags, currentTags);
var tagsToRemove = _.difference(currentTags, modifiedTags);
console.log('current', currentTags)
console.log('remove', tagsToRemove)
console.log('add', tagsToAdd)
if (tagsToAdd.length > 0) {
_.forEach(tagsToAdd, function (tag) {
event.tags.add(tag);
})
event.save(console.log)
}
if (tagsToRemove.length > 0) {
_.forEach(tagsToRemove, function (tagId) {
event.tags.remove(tagId)
})
event.save()
}
})
}
This is how the method is called from the event model:
afterCreate: function(record, next) {
Tag.modifyTags(tags, record.id)
next();
}
When I post to event/create, I get this result: http://pastebin.com/PMiqBbfR.
It looks as if the method call itself is looped over, rather than just the tagsToAdd or tagsToRemove array. Whats more confusing is that at the end, in the last log of the event, it looks like the event has the correct tags. When I then post to event/1, however, the tags array is empty. I've also tried saving immediately after each .add(), but still get similar results.
Ideally, I'd like to loop over both the tagsToAdd and tagsToRemove arrays, modify their ids in the model's collection, and then call .save() once on the model.
I have spent a ton of time trying to debug this, so any help would be greatly appreciated!
There are a few problems with your implementation, but the main issue is that you're treating certain methods--namely .save() and .findOrCreate as synchronous methods, when they are (like all Waterline methods) asynchronous, requiring a callback. So you're effectively running a bunch of code in parallel and not waiting for it to finish before returning.
Also, since it seems like what you're trying to do is replace the current event tags with this new list, the method you came up with is a bit over-engineered--you don't need to use event.tags.add and event.tags.remove. You can just use plain old update.
So you could probably rewrite the modifyTags method as:
modifyTags: function (tags, eventId, mainCb) {
// Asynchronously transform the `tags` array into an array of Tag records
async.map(tags, function(tag, cb) {
// For each tag, find or create a new record.
// Since the async.map `cb` argument expects a function with
// the standard (error, result) node signature, this will add
// the new (or existing) Tag instance to the resulting array.
// If an error occurs, async.map will exit early and call the
// "done()" function below
Tag.findOrCreate({tagName: tag}, {tagName: tag}, cb);
}, function done (err, tagRecords) {
if (err) {return mainCb(err);}
// Update the event with the new tags
Event.update({id: eventId}, {tags: tagRecords}).exec(mainCb);
});
}
See the full docs for async.map here.
If you wanted to stick with your implementation using .add and .remove, you would still want to use async.map, and do the rest of your logic in the done method. You don't need two .save calls; just do run all the .add and .remove code first, then do a single .save(mainCb) to finish it off.
And I don't know what you're trying to accomplish by deleting the .add and .remove methods from currentTags (which is a direct reference to event.tags), but it won't work and will just cause confusion later!

Categories

Resources