How to aggregate fields with null data and missing fields in elasticsearch? - javascript

I have elasticsearch database as following:
{
"market": "NEW YORK",
"hour_time_stamp": "2019-08-24 01:00:00",
"model_year": 2014,
"num_devices": 28
},
{
"market": "COLUMBIA",
"hour_time_stamp": "2019-08-12 01:00:00",
"model_year": 2019,
"num_devices": 2
},
{
"market": null,
"hour_time_stamp": "2019-08-11 01:00:00",
"model_year": 2016,
"num_devices": 1
},
{
"hour_time_stamp": "2019-08-02 01:00:00",
"model_year": 2014,
"num_devices": 750
},
{
"market": "COLUMBIA",
"hour_time_stamp": "2019-07-12 01:00:00",
"model_year": 2018,
"num_devices": 21
}
As you can see, some values of market are null and in some cases the market field is missing. I have written the following elastic.js query to fetch the market and perform aggregation on it:
let queryBody = ejs.Request()
.size(0)
.query(
ejs.BoolQuery()
.must(
ejs.RangeQuery('hour_time_stamp').gte(this.lastDeviceDate).lte(this.lastDeviceDate)
)
)
.agg(ejs.TermsAggregation('market_agg').field('market').order('sum', 'desc').size(50000)
.agg(ejs.SumAggregation('sum').field('num_devices'))
);
The data that I need is market name and sum of num_devices for that market(since market can be repeated). The query I have written returns only the market where market value is defined. I need to include the missing market as well as "market": null in my bucket. Any ideas how I can do that?

Related

Convert csv output toobject with nested arrays and objects using javascript

I have object after CSV conversion in this shape:
const data = {
"assets/0/description": "propertyImage",
"assets/0/filename": "a31989e07277954d93bca65e2d02dcb9.jpeg?p=cc_vo_small",
"assets/0/mimetype": "jpeg?p=cc_vo_small",
"assets/0/ordinal": "0",
"assets/0/url": "/usados/2022/09/04/0/a31989e07277954d93bca65e2d02dcb9.jpeg?p=cc_vo_small",
"assets/1/description": "propertyImage",
"assets/1/filename": "3bc3a9a4982508941ea178cca2ad31f1.jpeg?p=cc_vo_small",
"assets/1/mimetype": "jpeg?p=cc_vo_small",
"assets/1/ordinal": "1",
"assets/1/url": "",
combustion: "diesel",
createdAt: "2022-09-14T13:42:55.807Z",
engine: "235 CV",
firstDateRegistration: "2021",
gearbox: "automática secuencial",
"location/city": "Vitoria-Gasteiz",
"location/country": "España",
"location/distributor": "",
"location/houseNumber": "",
"location/postalCode": "",
"location/province": "Alava",
"location/street": "C/ Portal de Betoño, 13-15",
make: "volvo",
"metadata/0/key": "Año",
"metadata/0/value": "2021",
mileage: "29000",
model: "v90 cross country b5 pro awd aut.",
"optionals/0": "",
originalColorString: "gris",
"price/currency": "EUR",
"price/financialPrice": "53700",
"price/price": "56999",
saleDate: "Wed Sep 14 2022",
seller_type: "professional",
traction: "",
version: "V90 Cross Country B5 Pro Awd Aut."
};
Im trying to convert nested items that csv made flat into the correct format:
assets/0/description -> {
assets: [
{
description: "propertyImage",
filename: "...",
...
}
]
}
probably number between is an array and just / is an object. I tried to build reduce function and create it recursively but it looks more complicated than this. Any chance that there is a library for it? Or has someone already met this problem in the past?

JSON NESTED STRUCTURE - PENTAHO

I have this following json nested structure that i need to construct in pentaho:
{
"num_matricula": "4242424",
"limit_date": "2018-01-01",
"admission_date": "2018-01-01",
"cost_center": "anywhere",
"pos_number": "pos-test",
"role": "b63e065f-d7e0-49e1-91b7-88f74516e3fe",
"department": "bf559996-a8b9-4f5e-af57-86111b0dbde3",
"pagamento": {
"vinculo": "clt",
"valor": "4200",
"recorrencia": "mensalista",
"contaBancaria": {
"banco": "001",
"carta": "c9160763-db6c-4e8c-a1ad-ad8709c99be2"
}
},
"deficiencia": false,
"jornada": "De segunda a sexta das 15 as 19",
"profile": {
"name": "John Doe",
"email": "john.doe#acessodigital.com.br",
"mobile": "11911111111"
},
"exame": {
"clinica": "6dc84ce4-7d9f-48ec-b9b1-a8a895a21fd4",
"data": "2018-01-01",
"hora": "14:00",
"obs": "Comparecer de manhã",
"guia": "e37dab24-c7a4-4b92-b9d1-32ed538b8300",
},
"docs": ["c9e26093-5e0c-4bd2-bea3-ac5182a6179f"],
"send_sms": true,
"send_email": true
}
All the values come from the step "Select Values" which one gather the columns and the values to construct the json structure how it is showed in the following screenshot:
I've seen some solutions about using JavaScript to construct nested json structure in Pentaho, but i was not so sure about to create this using values from other steps. Can someone help me to construct this in pentaho? Pleasee.

getTime() method behaves differently in js

I have an array of object like :
[
{
"order_id": 1,
"customer": "Karita Klimochkin",
"country": "Sweden",
"address": "8978 Westridge Park",
"product_title": "Yellow-bellied marmot",
"product_description": "Bread - Flat Bread",
"date": "21/08/2020",
"status": "Delivered"
},
{
"order_id": 2,
"customer": "Ferne Roman",
"country": "China",
"address": "1370 Ridge Oak Pass",
"product_title": "Two-toed sloth",
"product_description": "Asparagus - White, Fresh",
"date": "24/07/2020",
"status": "Completed"
}
]
I want to sort objects by date. so when I use getTime() method it gives me different result.
orders.map(order => new Date(order.date).getTime())
results are :
1628100000000
NaN
What is the problem here?
You need to convert the date to mm/dd/yyyy format from dd/mm/yyyy so that JS can understand it properly
orders.map(order => {
const parts= order.date.split("/")
return new Date(`${parts[1]}/${parts[0]}/${parts[2]}`).getTime()
})
You cannot count on the default date parser to parse your DD/MM/YYYY format correctly. Parsing date strings through the constructor in this way is highly discouraged because it is implementation-dependent. Different browsers/runtimes will parse dates differently.
Instead, manually parse the date yourself and then construct the date object:
orders.map(order => {
const [d, m, y] = order.date.split('/');
return +new Date(+y, m-1, +d);
})
Friendly reminder: do you have a sorting functionality yet? .map is just an iteration through your array.
More about map: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/map
Add a sort function based on your date(properly parsed) property and return a new array would help.
The dates are in the wrong format:
// from: "21/08/2020"
let format = obj.date.split('/').reverse().join('-')
// to: "2020-08-21"
In order to be sortable, dates must be in ms since Jan 1, 1970. Assign the new value to a new key:
obj.pDate = Date.parse(format);
Sort by the new key/value:
let results = orders.sort((a, b) => a.pDate = b.pDate)
Then remove all of the new key/values:
results.map(order => delete order.pDate)
const data = [{
"order_id": 1,
"customer": "Karita Klimochkin",
"country": "Sweden",
"address": "8978 Westridge Park",
"product_title": "Yellow-bellied marmot",
"product_description": "Bread - Flat Bread",
"date": "21/08/2020",
"status": "Delivered"
},
{
"order_id": 2,
"customer": "Ferne Roman",
"country": "China",
"address": "1370 Ridge Oak Pass",
"product_title": "Two-toed sloth",
"product_description": "Asparagus - White, Fresh",
"date": "24/07/2020",
"status": "Completed"
}, {
"order_id": 3,
"customer": "zer00ne",
"country": "US",
"address": "123 Main St",
"product_title": "Jackalope",
"product_description": "Chili Cheese Fries",
"date": "12/05/2020",
"status": "Delivered"
},
];
const sortOrders = orders => {
let result = orders.sort((a, b) => {
a.pDate = Date.parse(a.date.split('/').reverse().join('-'));
b.pDate = Date.parse(b.date.split('/').reverse().join('-'));
return a.pDate - b.pDate;
})
result.map(order => delete order.pDate);
return result;
};
console.log(sortOrders(data));

Javascript: stuck with comparison function

I am stuck and need help finishing this fetch request.
I need a function to check if the movie (a single object) in the request has previously been rated by the user.
The ratings are in the ratedMovies array. If the movie was rated I need the userRating property with it's value to be added to the response. If it has not been rated, I need the userRating value to be null
const ratedMovies = useStore((state) => state.ratedMovies)
const getMovieDetails = () => {
const key = `https://www.omdbapi.com/?i=${movie.imdbID}&apikey=b46dc190`
axios
.get(key)
.then((response) => {
// Here a comparison is required, to check if the movie in the response (a single object)
// has ever been rated before and its ID (imdbID) and userRating (property to be added) is
// present in the ratedMovies array
// if it is present I need the property userRating nad it's value to be added to the response
// or when it is empty, to have the userRating be null
setModalDetails(response.data)
})
.then((response) => {
console.log(modalDetails)
})
.catch((error) => console.log(error))
}
Sample axios response:
{
"Title": "Star Wars: Episode V - The Empire Strikes Back",
"Year": "1980",
"Rated": "PG",
"Released": "20 Jun 1980",
"Runtime": "124 min",
"Genre": "Action, Adventure, Fantasy",
"Director": "Irvin Kershner",
"Writer": "Leigh Brackett, Lawrence Kasdan, George Lucas",
"Actors": "Mark Hamill, Harrison Ford, Carrie Fisher",
"Plot": "After the Rebels are brutally overpowered by the Empire on the ice planet Hoth, Luke Skywalker begins Jedi training with Yoda, while his friends are pursued across the galaxy by Darth Vader and bounty hunter Boba Fett.",
"Language": "English",
"Country": "United States, United Kingdom",
"Awards": "Won 2 Oscars. 25 wins & 20 nominations total",
"Poster": "https://m.media-amazon.com/images/M/MV5BYmU1NDRjNDgtMzhiMi00NjZmLTg5NGItZDNiZjU5NTU4OTE0XkEyXkFqcGdeQXVyNzkwMjQ5NzM#._V1_SX300.jpg",
"Ratings": [
{
"Source": "Internet Movie Database",
"Value": "8.7/10"
},
{
"Source": "Rotten Tomatoes",
"Value": "94%"
},
{
"Source": "Metacritic",
"Value": "82/100"
}
],
"Metascore": "82",
"imdbRating": "8.7",
"imdbVotes": "1,209,128",
"imdbID": "tt0080684",
"Type": "movie",
"DVD": "21 Sep 2004",
"BoxOffice": "$292,753,960",
"Production": "N/A",
"Website": "N/A",
"Response": "True"
}
Sample rating:
ratedMovies = [{imdbID: 'tt0080684', userRating: 8}]
Ok if I understand it correctly it goes like this:
let data = response.data;
let newMovieId = data.imdbID;
ratedMovies.forEach((movie) => {
if(movie.imdbID === newMovieId) {
data.userRating = movie.userRating;
}
});
setModalDetails(data)
code above goes inside the axios success callback
You can use .filter function like this:
ratedMovies = [{imdbID: 'tt0080684', userRating: 8}]
notratedMovies = [{imdbID: 'tt0080111', userRating: 8}]
let result = ratedMovies.filter(obj => {
return obj.imdbID === imdb.imdbID
});
console.log(result);
let notresult = notratedMovies.filter(obj => {
return obj.imdbID === imdb.imdbID
});
console.log(notresult);

Javascript convert object key names based on mapping

I have a javascript object that is coming from Source1 and I am unable to change its native structure or naming convention. I am using this data to feed into a 3rd party plugin to generate some chart data. This plugin however is using the key names as the identifiers on the chart and they are not descriptive or clear enough.
I am trying to run the object through a conversion function where it will change all of the key names to their defined equivalent.
Here is an example of what I am trying to do:
var obj = [{
SubmissionID: "28935",
MetaCreatedDate: "12 Mar 2018",
Program: "Brand Risk Management",
ViewedByInvestigator: "No",
},
{
SubmissionID: "28936",
MetaCreatedDate: "12 Mar 2018",
Program: "Brand Risk Management",
ViewedByInvestigator: "Yes",
}]
function convertNames(obj){
// Converted names
var map = [{
SubmissionID: 'SubmissionIdentifier',
MetaCreatedDate: 'CreationDate',
Program: 'ProgramName',
ViewedByInvestigator: 'Viewed'
}];
// Loop through the object and convert all key names to their equivalent
for(var prop in obj){
// Convert Here
}
return obj;
}
Desired Output:
[{
SubmissionIdentifier: "28935",
CreationDate: "12 Mar 2018",
ProgramName: "Brand Risk Management",
Viewed: "No",
},
{
SubmissionIdentifier: "28936",
CreationDate: "12 Mar 2018",
ProgramName: "Brand Risk Management",
Viewed: "Yes",
}]
https://jsfiddle.net/hbg4sfqh/7/
I'd combine the .map array method and a function to convert your key names to get the result you want. To convert the key names, you'll want to use bracket notation, so something like: newObj[keyMap[oldKey]] = oldObj[oldKey] should work.
Here's a simple implementation for your example:
const obj = [{
SubmissionID: "28935",
MetaCreatedDate: "12 Mar 2018",
Program: "Brand Risk Management",
ViewedByInvestigator: "No",
}, {
SubmissionID: "28936",
MetaCreatedDate: "12 Mar 2018",
Program: "Brand Risk Management",
ViewedByInvestigator: "Yes",
}];
const nameMap = {
SubmissionID: 'SubmissionIdentifier',
MetaCreatedDate: 'CreationDate',
Program: 'ProgramName',
ViewedByInvestigator: 'Viewed'
}
function renameKeys(obj, map) {
const newObj = {};
for (let key in obj) {
newObj[map[key]] = obj[key];
}
return newObj;
}
console.log(obj.map(item => renameKeys(item, nameMap)));
I'd also note that if you happen to be using the lodash library, you can also use it's _.mapKeys method to do this.
I'm gonna use .map() function to change the key names. The input data will remain unchanged. Hope this helps.
var obj = [{
SubmissionID: "28935",
MetaCreatedDate: "12 Mar 2018",
Program: "Brand Risk Management",
ViewedByInvestigator: "No",
},
{
SubmissionID: "28936",
MetaCreatedDate: "12 Mar 2018",
Program: "Brand Risk Management",
ViewedByInvestigator: "Yes",
}]
var output = obj.map(element => ({
SubmissionIdentifier: element.SubmissionID,
CreationDate: element.MetaCreatedDate,
ProgramName: element.Program,
Viewed: element.ViewedByInvestigator
}));

Categories

Resources