Develop Reference

Does the Elasticsearch bulk API JavaScript sample code work correctly?

I want to use Elasticsearch bulk index with JavaScript, and follow the official document. Basically I just used it as is, but it throws an exception saying that the bulk request must be terminated by a newline code, which seems to be JSON lines.
My question is, if so, how can I convert the JSON to JSON lines and then pass it to bulk API? or am I missing some options when calling client.bulk?
ResponseError: illegal_argument_exception: [illegal_argument_exception] Reason: The bulk request must be terminated by a newline [\n]
at SniffingTransport.request
I use Elasticsearch v7.15 and Node.js v16.15 on macOS.

I second this. I'm using 8.2 and it doesn't work. I followed the example exactly and it still throws the newline error.
You used to be able to pass in the operation followed by the document in an array and it would work just fine but it seems to have been "updated" for the worse.
I'll keep trying and see if I come up with something.
EDIT: OK. I found something that works for me and I hope it works for you as well.
I made a child client to reuse the connection. It's pretty straight forward but here's the link: https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/child.html
I modified the accept and content-type headers for that client and the code for setting up the client looks like so:
const es = require('#elastic/elasticsearch');
const es_hosts = [
'https://elastic_host1',
'https://elastic_host2'
];
const es_client = new es.Client({ node: es_hosts });
const es_bulk = es_client.child({
headers: {
'accept': 'application/json',
'content-type': 'application/json'
}
})
I'm assuming you're passing in an array which has an operation and a document. We'll convert that array into what equates to be a large string, remembering to tack on the newline.
let bulk_data = [
{ index: { _index: 'my-index' } },
{ id: 1, name: 'dave', job: 'janitor' } }
]
let bulk_body = {
body: bulk_data.map(JSON.stringify).join('\n') + '\n'
}
const results = await es_bulk.bulk(bulk_body);
Let me know if this helps.

Extract variables from json string

Context :
I'm making a Discord bot using the discord.js library and MongoDB (with mongoose). I created a script that can construct messages from data stored in my MongoDB database. For exemple, if an user on my Discord server type "!hello", it will query my database to construct a reply like "Hello to you!".
I store my reply message data in an document like this :
{
"command": "!hello",
"content": "Hello to you!"
}
(I voluntary omitted some properties here that are useless in the context of my question)
And retrieve it in my code with something like :
let retrievedReply;
await mongo().then(async mongoose => {
try {
let query = {command: message.content};
retrievedReply = await replySchema.findOne(query);
} finally {
mongoose.connection.close();
}
});
message.reply(retrievedReply.content);
My issue :
Now, imagine I want it to respond "Hello username!".
If I didn't construct my reply using data from mongo I could simply do something like :
message.reply(`Hello ${message.member.nickname}!`);
or
message.reply("Hello " + message.member.nickname + "!");
So, in other words, using backticks and ${} syntax or (but I like it less), splitting my string and concatenate my property value like the second example.
The thing is, we can't store a string inside backticks in json format (used by MongoDB). So even if I build my document like so :
{
"command": "hello",
"content": "Hello ${message.member.nickname}!"
}
${message.member.nickname} would not be interpreted like a property value but just like a simple string.
The 1000$ question :
How can I use this property inside my code? I thought about extracting it using regex but it does not change the fact that it will still be a string. Is there a way to "convert" from a string to an usable variable?
EDIT
Since Brettski asked me (rightfully), here is an example of what I want :
message.member.send(`Hello ${message.member}!`);
This message.member property here represent a member object of a Discord guild. It contains a lot of things like its permissions on the guild, its roles, etc.
When we use a member object into a message, Discord will make a mention of the member like bellow.
It works the same with other properties like channels for example. Users can then click on those to see informations about the member or to go directly to the channel, etc. I want to be able to use this.

That is correct you will not be able to store a string literal in Mongo or other way.
One though I had is to put a token in your string and then do a replace on it in message.reply() or before it.
example:
const helloCommand = {
"command": "hello",
"content": "Hello ~~membernickname~~!"
};
In this example the token to replace is ~~membernickname~~
You can use something like this to do the replace:
const command = JSON.parse(helloCommand)
message.member.send(command.content.replace('~~membernickname~~', message.member));
The resulting string being sent to the send() command is the same. So if the nickname is #brettski, the string 'Hello #brettski' will be sent to the send() command.
In your example:
message.reply(`Hello ${message.member.nickname}!`);
the part:
`Hello ${message.member.nickname}!`
is a string literal (template literals). What the function message.reply() ends up getting is 'Hello Nickname' a string. So the replace should work. As a test you can hard code the string used in message.reply() to a known nickname and it should provide the results you expect.

Ok so, #Brettski put me on the right track with the fact that I can use users' or channels' IDs directly as strings within the message and between specific tags.
To reuse the example in my question, I can use this syntax to mention an user :
"<#idOfTheUser>"
What I did is if I want to mention the author of the command in my reply, I put the text in my mongoDB document as so :
const helloCommand = {
"command": "hello",
"content": "Hello $member$!"
};
Then, in my code, I can simply use :
message.reply((retrieveReply.content).replace(/\$member\$/g, `<#${message.member.id}>`))
Of course, I wanted to use more properties so I did a function like :
const messageContentReplace = function(content) {
const replaceableStrings = {
"$member$": `<#${message.member.id}>`,
"$memberName$": message.member.displayName,
"$guild$": message.guild.name
}
return content.replace(/(\$member\$)|(\$memberName\$)|(\$guild\$)/g, match => {
return replaceableStrings[match];
});
(in reality, I did something more complexe because I use an embed message as my response with multiple fields etc... See here about embeds)

Reading a JSON response in Javascript

I'm making an ajax post request to a super simple python function that takes a student's name and spits out a url that corresponds to it. Currently, the Python function passes this back in json and looks like so when console.log(JSON.stringify(response)) is called:
{"readyState":4,"responseText”:”\ {\”studentURL\”: \”https://prepacademy.jackjohnson.com\”} ”,”responseJSON”: {“studentURL”:”https://prepacademy.jackjohnson.com”},”status":200,"statusText":"OK"}
I was wondering how do I take this larger chunk of information and filter it so that I would only get the https://prepacademy.jackjohnson.com part?

response is a JavaScript Object of which you can access the properties using either Dot-notation or bracket-notation, like so:
let response = {
"readyState": 4,
"responseText": "\ {\"studentURL\": \"https://prepacademy.jackjohnson.com\"} ",
"responseJSON": {
"studentURL": "https://prepacademy.jackjohnson.com"
},
"status": 200,
"statusText": "OK"
};
// dot-notation
console.log(response.responseJSON.studentURL)
// bracket-notation (allows for computed paths)
console.log(response["responseJSON"]["studentURL"])

response.responseJSON.studentURL

How to get valid objects in a string in JavaScript?

So currently, I am getting data via TCP connection and we can tell if a string is a valid JSON object using something like:
let body = '';
client.on('data', (chunk) => {
body += chunk.toString();
try {
let data = JSON.parse(body);
// data is valid json
}
catch (e) {}
});
While this works, can we get all valid Objects in a chunk? Let's say we have
const chunk = // an incomplete object like { "list": [ { "id": 1 }, { "id":
So in the sample, how can we get a valid object in a string? I'm trying to get { "id": 1 } because it's the only valid object in the string.

What you need is a streaming JSON parser, such as this one: https://github.com/chrisdickinson/json-parse-stream
As you probably already know, unfinished objects are invalid and somewhat arbitrary. So, you'll have to choose how to handle these situations. That is, only if you're in control of the data and have decided how to handle these situations can you know what to do. The streaming JSON parser will emit objects and such as they're passed in, which should help you get around any incomplete objects.

uncaught TypeError: undefined is not a function when using JSON object with name

I'm using D3js with Mongodb and AnguarlJS to display my data. All is good that it works until when I give my JSON array a name. Then angular starts complaining about stuff and I'm not sure why.
this is the original json array that works with this original code
[
{
"defaultCategory":"Happy",
"timesUsed":2704659
},
{
"defaultCategory":"Sad",
"timesUsed":4499890
},
{
"defaultCategory":"Laughter",
"timesUsed":2159981
},
{
"defaultCategory":"Smirk",
"timesUsed":3853788
},
{
"defaultCategory":"Flirty",
"timesUsed":14106543
}
]
d3.json("data.json", function(error, data) {
data.forEach(function(d) {
d.timesUsed =+ d.timesUsed;
});
but when i change the json to this format, it breaks
{
"mymood":[
{
"defaultCategory":"Happy",
"timesUsed":2704659
},
{
"defaultCategory":"Sad",
"timesUsed":4499890
},
{
"defaultCategory":"Laughter",
"timesUsed":2159981
},
{
"defaultCategory":"Smirk",
"timesUsed":3853788
},
{
"defaultCategory":"Flirty",
"timesUsed":14106543
}
]
}
d3.json("data.json", function(error, data) {
data.mymood.forEach(function(d) {
d.timesUsed =+ d.timesUsed;
});
In the chrome console the error is happening at the line data.mymood.foreach line,
but i dont understand why because its exactly returning the same json as if there was no name like this
[object, object, object,object,object]
and the parameter d in the function also returns the same object within the array
edit
Error:
Uncaught TypeError: undefined is not a function
console.log(data) -> Object {mymood: Array[5]}
console.log(data.mymood) -> [Object, Object, Object, Object, Object]
gist for those who are interested in the full code
https://gist.github.com/gwong89/e3a29b64d94ad20256bb

Like Oscar had pointed out, the forEach loop works fine. However, after look closely at your gist, if you try to use the mymood key, make sure you also change line 55 where you were using
var g = svg.selectAll('g')
.data(pie(data))
to
var g = svg.selectAll('g')
.data(pie(data.mymood))
Also on line 76 where you had
var total = d3.sum(data.map(function(d) {
to
var total = d3.sum(data.mymood.map(function(d) {
I grabbed your project repo and tried these out on my local environment, and I can render the pie chart and popup without seeing any errors.

Your code seems to be working fine (from what I've tested). I will suggest to try debugging your code or if you feel more comfortable try doing many console.log(...) as possible and keep checking your browser console from the beginning (I have nothing else to say). Try to also use Developer Tools from Chrome which is a better option.
I've tried to replicate your context by using d3.js library and uploaded a json file with your data to my Dropbox just to be able to perform an ajax request and everything is fine again (also new JSON structure is valid). Here's what I did, probably could help you to learn something new (see below).
Possible hints/suggestions to fix your issue:
According to the question title, it seems that data.mymood is undefined so you are not able to do a forEach of something that doesn't exist. Try validating things and avoid null pointers (see example).
JSON structures are valid so that's not the problem.
Syntax seems to be valid from what I've tested.
Check if there's a conflict between libraries, try to do some research about using all the libraries needed together (not sure if this is your situation but could happen).
Check your browser console and do some debugging as the first paragraph says.
Check if your request is not timing out or something else is happening, check the Network tab or log your requests in browser console using Developer Tools from Chrome (just saying).
Live Demo: http://jsfiddle.net/29f6n8L7/
d3.json('https://dl.dropboxusercontent.com/u/15208254/stackoverflow/data.json', function(data) {
// Some ways to avoid null pointers
var obj = (data || {}), // If 'data' is null, initialize as an Object
moods = (obj.mymood || []), // If 'obj.mymood' is null, initialize as an Array
time = 0,
mood;
// Same way to get 'mymood' from the object
console.log('Your obj.mymood contains: %o', obj.mymood);
console.log('Your obj["mymood"] contains: %o', obj['mymood']);
/**
* I will suggest to use a traditional loop instead of 'forEach'.
* Please read: http://stackoverflow.com/a/9329476/1178686
* for a better explanation
*/
for (var i = 0; i < moods.length; ++i) {
mood = moods[i];
if (mood) {
time += mood.timesUsed;
}
}
// Output
console.log('Sum of timesUsed is: %s', time);
});

Arrays have a built in for each in es5, objects do not.