Index data to ElasticSearch with Firebase functions, not all fields are indexed - javascript

I am using Firebase function to push data to ElasticSearch to be indexed. But sometimes the data set is not indexed completely. Which fields can be indexed seem to be a random behavior, sometimes all fields are indexed, sometimes only a few, sometimes only one.
The firebase function I used to push data to Elastic search as below:
exports.indexHashtagsToElastic = functions.database.ref('/valid_hashtags/{hashtag_id}')
.onWrite(event =>{
let hashtagData = event.data.val();
let hashtag_id = event.params.hashtag_id;
console.log('Indexing the hashtags: ', hashtagData);
let elasticSearchConfig = functions.config().elasticsearch;
let elasticSearchUrl = elasticSearchConfig.url + 'hashtags/hashtag/' + hashtag_id;
let elasticSearchMethod = hashtagData ? 'POST' : 'DELETE';
let elasticSearchRequest = {
method: elasticSearchMethod,
url: elasticSearchUrl,
auth:{
username: elasticSearchConfig.username,
password: elasticSearchConfig.password
},
body: hashtagData,
json: true
};
return request(elasticSearchRequest).then(response => {
console.log("ElasticSearch response", response);
});
});
according to this log console.log('Indexing the hashtags: ', hashtagData); sometimes the dataset is not coming through in full. However the dataset created on database is complete and correct.
Any idea what went wrong here? Thanks!

You need to specify the fields to index on your request.
let elasticSearchFields = ['field1','field2','field3',...'fieldn'];
Then in your request you should add the following
...
body: _.pick(hashtagData , elasticSearchFields ),
...
Hope that will work

Related

How do I filter through API information to find a perfect match of a specific search param in React?

I have an API call returning 1 movie name at random... The code has it set up so it returns 1 movie name only.
I then have a second API call that uses that movie name in a second axios request to find streaming services available for this specific movie title.
My issue is, the second API call returns an array of 20 or so choices. For example, if I searched for 'Jaws' it delivers results for Jaws, all the other Jaws movies and any other movie with Jaws in the name.
My question is, how do I filter through this return information, to find a perfect match with the right movie title and just show that info? I hope you can help and sure it is going to be some filtering logic. Thankyou! I am new to React so trying to learn.
const optionsParams = { method: 'GET', url: 'https://mdblist.p.rapidapi.com/', params: [], headers: { 'X-RapidAPI-Host': 'mdblist.p.rapidapi.com', 'X-RapidAPI-Key': '384177d302msh9d8e58dffccp1bfa57jsnf3cea9fef042', }, }; axios .request({ ...optionsParams, params: { s: ${movieData.title}} }) .then(function (response) { console.log(response.data); const traktid = response.data.search.traktid; const traktidOptions = { ...optionsParams, params: { t: ${traktid}` }
};
axios.request(traktidOptions).then(function (response) {
const streamingServices = response.data.streams;
setStreamingState(streamingServices);
//console.log(streamingState);
console.log(response.data.streams)
let temporaryArray = [];
for (let i = 0; i < response.data.streams.length; i++){
temporaryArray.push(response.data.streams[i].name)
console.log(temporaryArray);
}
setStreamingState(streamingState => [...streamingState, `${temporaryArray}`])
if (streamingState) {
console.log(streamingState)
} else return false;
})
})`
Array methods are the perfect tool for iterating and organising array data.
In this case the Array.prototype.find() will help you look through your array and return a single item based on your criteria.
For example:
const yourArray = ["jaws", "not this jaws", "definitely not jaws"];
const found = yourArray.find(arrayItem => arrayItem === "jaws");
This is a simple example without objects, so you would just need to set the criteria based on an object property.

Why does the "append" method in javaScript does not return an array?

Here is an API call, which needs some values to return a specific set of products,
issue
"category_slug" has to be an array but for some reason, the API says it's not. what is the problem here?
const url = new URL(
"https://someApi/products"
);
let params = {
"limit": "10",
"page": "1",
"category_slug": ["shoes"]
};
// "limit" is "how many products in on one page".
// "page" is the fetched page with the specific "limited" products.
//issue
// "category_slug" is a specific products category, this has to be an array but for
// some reason, the API says it's not. what is the problem here?
Object.keys(params)
.forEach(key => url.searchParams.append(key, params[key]));
//here I'm appending the specific value in {params} to the URL.
let headers = {
"Accept": "application/json",
"Content-Type": "application/json",
};
fetch(url, {
method: "GET",
headers: headers,
})
.then(response => response.json())
.then(json => console.log(json));
See, you expect too little and too much at the same time of such a beautiful thing as URLSearchParams.
Too little, because usually you can just pass the whole params object into its constructor without wasting time on keys, forEach, etc...
const url = new URL('https://example.com/');
const params = {
limit: 10,
page: 1
};
url.search = new URLSearchParams(params); // yes, that easy
console.log(url.toString());
// https://example.com/?limit=10&page=1
Too much, because URLSearchParams is not designed to work with arrays. When appended element is an array, it's just stringified:
const url = new URL('https://example.com/');
const params = {
slug: [1, 2, 3]
};
url.search = new URLSearchParams(params);
console.log(url); // https://example.com/?slug=1%2C2%2C3
In this case, slug param got 1,2,3 (result of [1, 2, 3].toString()) assigned to it (and all the commas were urlencoded - replaced by %2C sequence).
Your API might actually work with this, but there's a huge chance it expects array args to be passed in the following format:
https://example.com/?slug=1&slug=2&slug=3
... yet even this might not work, if API expects array args to be passed with [] appended to each key, like this:
https://example.com/?slug[]=1&slug[]=2&slug[]=3
So you'll have to check your API (it's hard to debug such things just by looking into crystal ball, you know...), take its flavor into account - and process your items separately. For example,
const url = new URL('https://example.com/');
const params = {
limit: 10,
page: 1
};
url.search = new URLSearchParams(params);
const slugs = [1, 2, 3];
url.search += ['', ...slugs.map(
slug => `category_slug[]=${encodeURIComponent(slug)}`)].join('&');
console.log(url.toString());

mongoosejs - find() using nested objects

question is possibly a duplicate but I haven't found anything that provides an appropriate answer to my issue.
I have an ExpressJS server which is used to provide API requests to retrieve data from a MongoDB database. I am using mongoosejs for the MongoDB connection to query/save data.
I am building a route that will allow me to find all data that matches some user input but I am having trouble when doing the query. I have spent a long while looking online for someone with a similar issue but coming up blank.
I will leave example of the code I have at the minute below.
code for route
// -- return matched data (GET)
router.get('/match', async (req, res) => {
const style_data = req.query.style; // grab url param for style scores ** this comes in as a string **
const character_data = req.query.character; // grab url param for character scores ** this comes in as a string **
// run matcher systems
const style_matches = style_match(style_data);
res.send({
response: 200,
data: style_matches
}); // return data
});
code for the query
// ---(Build the finder)
const fetch_matches_using = async function(body, richness, smoke, sweetness) {
return await WhiskyModel.find({
'attributes.body': body,
'attributes.richness': richness,
'attributes.smoke': smoke,
'attributes.sweetness': sweetness
});
}
// ---(Start match function)---
const style_match = async function (scores_as_string) {
// ---(extract data)---
const body = scores_as_string[0];
const richness = scores_as_string[1];
const smoke = scores_as_string[2];
const sweetness = scores_as_string[3];
const matched = [];
// ---(initialise variables)---
let match_count = matched.length;
let first_run; // -> exact matches
let second_run; // -> +- 1
let third_run; // -> +- 2
let fourth_run; // -> +- 3
// ---(begin db find loop)---
first_run = fetch_matches_using(body, richness, smoke, sweetness).then((result) => {return result});
matched.push(first_run);
// ---(return final data)---
return matched
}
example of db object
{
_id: mongoid,
meta-data: {
pagemd:{some data},
name: whiskyname
age: whiskyage,
price: price
},
attributes: {
body: "3",
richness: "3",
smoke: "0",
sweetness: "3",
some other data ...
}
}
When I hit the route in postman the JSON data looks like:
{
response: 200,
data: {}
}
and when I console.log() out matched from within the style match function after I have pushed the it prints [ Promise(pending) ] which I don't understand.
if I console.log() the result from within the .then() I get an empty array.
I have tried using the populate() method after running the find which does technically work, but instead of only returning data that matches it returns every entry in the collection so I think I am doing something wrong there, but I also don't see why I would need to use the .populate() function to access the nested object.
Am I doing something totally wrong here?
I should also mention that the route and the matching functions are in different files just to try and keep things simple.
Thanks for any answers.
just posting an answer as I seem to have fixed this.
Issue was with my .find() function, needed to pass in the items to search by and then also a call back within the function to return error/data. I'll leave the changed code below.
new function
const fetch_matches_using = async function(body, richness, smoke, sweetness) {
const data = await WhiskyModel.find({
'attributes.body': body,
'attributes.richness': richness,
'attributes.smoke': smoke,
'attributes.sweetness': sweetness
}, (error, data) => { // new ¬
if (error) {
return error;
}
if (data) {
console.log(data)
return data
}
});
return data; //new
}
There is still an issue with sending the found results back to the route but this is a different issue I believe. If its connected I'll edit this answer with the fix for that.

Sequelizejs - Then result issue

I'm work with NodeJS and Sequelize. I have the following issue:
Read the Settings Table:
Settings.findOne({where: {
user_id: data
}})
.then(settings => {
// Next request
});
I need to save the settings.device (Example) outside of the .then block.
But if I do that like
var device;
Settings.findOne({where: {
user_id: data
}})
.then(settings => {
device = settings.device;
});
It doesn't work.
Already the error output undefined
The output in .then result block with console.log(settings.device); works perfect.
Update
I need it like:
var array = [];
// Get Settings from table
Settings.findOne({where: {
user_id: data
}})
.then(settings => {
// Get token from other Table
Example.findOne({where: {
user_id: data
}})
.then(example => {
// push to array
array.push({
"user_id" : data,
"device":settings.device, // output: settings.device is undefined
"token": example.token
});
});
});
// Send array to client
This is really a question of how to handle multiple resolved values in a Promise chain. You can search for that and see lots of great examples on how to handle it. For example, you could return an array or object in each then handler, or re-assign values to higher-scoped variables (as you're doing with settings). I've used both methods heavily in the past, and the resulting code is obviously inelegant and not fun to write.
However, async/await is readily available in Node and simplifies your code quite a bit:
const array = [];
// Get settings.
const settings = await Settings.findOne({where: {
user_id: data
}});
// Get token.
const example = await Example.findOne({where: {
user_id: data
}});
array.push({
user_id : data,
device: settings.device,
token: example.token
});
Sequelize return the model object you can get value by dataValue
console.log(settings.dataValues.device);
or if you want lean data
Settings.findOne({where: {
user_id: data,
raw:true,
}})
.then(settings => {
device = settings.device;
console.log(device);
});

How to set only ONE element in an array in firebase

I have an array saved on my firebase, like this:
matches:[ {match:{id:1,data:...}}]
I want to save just one item on firebase at this array. For example, my match have id:32. I want to find it on the array saved in firebase, and change it.
Im trying to make this. But Im thinking that this is VERY UGLY to make a request to the firebase, copy the array, and save the entire array again.
const ref = `/users/${currentUser.uid}/matches`;
var list = [];
firebase.database().ref(ref).on('value', function (snap) { list = snap.val(); });
if (list.length > 0) {
const indexToUpdate = list.findIndex(k => k.name == match.name)
if (indexToUpdate >= 0) {
list[indexToUpdate] = match;
return dispatch => {
firebase
.database()
.ref(`/users/${currentUser.uid}/matches`)
.set(list)
.then(() => {
dispatch({ type: MATCH_UPDATE, payload: match });
});
};
}
}
Any light?
This line of code:
const indexToUpdate = list.findIndex(k => k.name == match.name)
Is a dead giveaway that your data structure is not ideal.
Consider storing the matches under their name, prefixing it with a string to prevent the array coercion that Kenneth mentioned. So:
matches
match1
date: "2018-06-14..."
...
match2
date: "2018-06-16..."
...
With this you can look up the node for a match without needing a query. It also prevents using an array, which is an anti-pattern in Firebase precisely because of the reason you mention.
For more on this, see the classic Firebase blog post on Best Practices: Arrays in Firebase.
Firebase stores arrays as object and converts it back to array when it comes back to the client if the keys are ordered numerically correctly
But basically it should be able to work the same where you make your path up to the object you want to update.
firebase
.database()
.ref(`/users/${currentUser.uid}/matches/${indexToUpdate}`)
.set(match)
.then(() => {
dispatch({ type: MATCH_UPDATE, payload: match });
});
You need to reference the ID of the item you are trying to set by ID in the URL
firebase
.database()
.ref(`/users/${currentUser.uid}/matches/32`)
.set(list)
.then(() => {
dispatch({ type: MATCH_UPDATE, payload: match });
});
Also, use snap.key to get the ID you need if index isn't working.

Categories

Resources