Develop Reference

Does the Elasticsearch bulk API JavaScript sample code work correctly?

I want to use Elasticsearch bulk index with JavaScript, and follow the official document. Basically I just used it as is, but it throws an exception saying that the bulk request must be terminated by a newline code, which seems to be JSON lines.
My question is, if so, how can I convert the JSON to JSON lines and then pass it to bulk API? or am I missing some options when calling client.bulk?
ResponseError: illegal_argument_exception: [illegal_argument_exception] Reason: The bulk request must be terminated by a newline [\n]
at SniffingTransport.request
I use Elasticsearch v7.15 and Node.js v16.15 on macOS.

I second this. I'm using 8.2 and it doesn't work. I followed the example exactly and it still throws the newline error.
You used to be able to pass in the operation followed by the document in an array and it would work just fine but it seems to have been "updated" for the worse.
I'll keep trying and see if I come up with something.
EDIT: OK. I found something that works for me and I hope it works for you as well.
I made a child client to reuse the connection. It's pretty straight forward but here's the link: https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/child.html
I modified the accept and content-type headers for that client and the code for setting up the client looks like so:
const es = require('#elastic/elasticsearch');
const es_hosts = [
'https://elastic_host1',
'https://elastic_host2'
];
const es_client = new es.Client({ node: es_hosts });
const es_bulk = es_client.child({
headers: {
'accept': 'application/json',
'content-type': 'application/json'
}
})
I'm assuming you're passing in an array which has an operation and a document. We'll convert that array into what equates to be a large string, remembering to tack on the newline.
let bulk_data = [
{ index: { _index: 'my-index' } },
{ id: 1, name: 'dave', job: 'janitor' } }
]
let bulk_body = {
body: bulk_data.map(JSON.stringify).join('\n') + '\n'
}
const results = await es_bulk.bulk(bulk_body);
Let me know if this helps.

Signing JWT - do I do it wrong?

I'm trying to make a JWT generator in JavaScript for educational purposes. There is a jwt.io tool to create and/or validate JWT.
I'm struggling to get my results match the results from the validator. The problem is the signature.
Here's my code:
function base64url(input) {
return btoa(typeof input === 'string' ? input : JSON.stringify(input))
.replace(/=+$/, '')
.replace(/\+/g, '-')
.replace(/\//g, '_');
}
const JWT = {
encode(header, payload, secret) {
const unsigned = [base64url(header), base64url(payload)].join('.');
return [unsigned, base64url(sha256.hmac(secret, unsigned))].join('.');
}
};
To encrypt HMAC SHA256 I'm using js-sha256 library with sha256.hmac(key, value) prototype. I compared it with online tools and it works fine.
Now, I test it with the following code:
const jwt = JWT.encode(
{
alg: 'HS256',
typ: 'JWT'
},
123,
'xxx'
);
The result I get is:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.MTIz.NzhlNTFmYzUxOGQ2YjNlZDFiOTM0ZGRhOTUwNDFmMzEwMzdlNmZkZWRhNGFlMjdlNDU3ZTZhNWRhYjQ1YzFiMQ
On the other hand, the result from jwt.io is:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.MTIz.eOUfxRjWs-0bk03alQQfMQN-b97aSuJ-RX5qXatFwbE
As you can see, the two out of three chunks of JWT are identical in my result and jwt.io result. The signature is different and if you ask me, the signature generated by it is surprisingly short. That tool also marks my own JWT as invalid.
I checked with online HMAC SHA256 generators and it looks like my code creates a valid signature, so:
base64url(sha256.hmac('xxx', 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.MTIz')) ===
'NzhlNTFmYzUxOGQ2YjNlZDFiOTM0ZGRhOTUwNDFmMzEwMzdlNmZkZWRhNGFlMjdlNDU3ZTZhNWRhYjQ1YzFiMQ'
Is jwt.io just broken or does it do it some other way?

I wouldn't say you're doing it wrong, but you missed a small but important detail.
The result from jwt.io is correct and the hash you calculate is also correct. But the signature you create with your hash is not correct.
The hash you calculate with sha256.hmac(secret, unsigned) is a large number but the return value of the function is a hexadecimal string representation of that large number. For the signature you need to base64url encode the original number instead of it's string representation.
I modified your code, so that it encodes the hash value directly to base64url (node.js version):
const JWT = {
encode(header, payload, secret) {
const unsigned = [base64url(header), base64url(payload)].join('.');
const hash = sha256.hmac(secret, unsigned);
console.log(hash);
var signature = new Buffer.from(hash, 'hex').toString('base64').replace(/\+/g,'-').replace(/\=+$/m,'');
return [unsigned, signature].join('.');
}
};
or, if you don't use node.js, you can use this instead (as suggested by Robo Robok):
const JWT = {
encode(header, payload, secret) {
const unsigned = [base64url(header), base64url(payload)].join('.');
return [unsigned, base64url(sha256.hmac(secret, unsigned).replace(/\w{2}/g, byte => String.fromCharCode(parseInt(byte, 16))))].join('.');
}
};
The result is a token, which is identical to the one created with jwt.io:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.MTIz.eOUfxRjWs-0bk03alQQfMQN-b97aSuJ-RX5qXatFwbE
See also my answer here, in which I explained the steps to compare the results from different tools.

How to pass array in the url?

I'd like to filter and want to pass an array to the url.
handleFilter = (search, page = 1) => {
const requestOption = {
method: "GET"
};
fetch("http://127.0.0.1:8000/api/home?search=" + JSON.stringify(search) + "&page=" + page, requestOption)
.then(res => res.json())
.then(data => (
this.setState({
data
})
))
};
Just want to pass the array of data to the api to call the query

I don't know if this will be what exactly you need, but I would...
a.) Join it on a strange character, and pass it as a string.
b.) On the receiving side (client or server), split on that same character.

If you want send it exactly as GET method (not Post), you can form your URL like this:
"http://127.0.0.1:8000/api/home?search[]=value1&search[]=value2&search[]=value3"
That is if you don't want send JSON string.
And what the problem with JSON version or POST method?

Really strange error post with google script

I'm trying to make a post with google Apps.
function makePost(url,page,status,token) {
var form = {
"page" : 1,
"statuses[]": status.toString().toLowerCase(),
}
var options = {
'method': 'post',
'headers' : {"Authorization" : "BASIC "+token},
'contentType': 'application/x-www-form-urlencoded',
'payload' : form,
};
var response = UrlFetchApp.fetch(url, options);
var jsonResponse = JSON.parse(response.getContentText());
return jsonResponse;}
The problem seems to be that page is being past as 1.0 what makes the destination url return an error when tries to cast.
Already tried the answer in How to convert a string to number using Google Apps Script .
I cant pass it as a String , because in that case the API returns the result with an strange behaviour.
Is it possible to pass this as Integer without being change it to 1.0 ?
Any suggestion or help will be helpfull,
thanks
UPDATE: So , i couldnt find a way to accomplish this. UrlFetchApp changes it every time. Lucky for me we could solve it by order the response in the API , and the behaviour passing a string stoped being erratic.
Thanks for the help and suggestions!

Same here connecting with Stripe REST which enforces a form-based body. It must be a bug of UrlFetchApp.
When I use
payload: {
amount: 1000,
currency: 'USD',
}
UrlFetchApp always encoded it into, note the decimal .0
"payload":"amount=1000.0&currency=USD"
But I resolved it by converting it to a String.
payload: {
amount: String(1000),
currency: 'USD',
}
Oddly, the payload will now be encoded correctly into
"payload":"amount=1000&currency=USD"
since the form-urlencoded doesn't specify the type of value, in this case, either a numeric String or an Integer. It's the server's implementation for interpreting the value.

Gmail API - Parse message content (Base64 decoding?) with Javascript

I'm trying to use the Gmail API to get a user's email, grab the message subject and body, and then display it on a webpage. I'll be doing other stuff with it, but this is the part that I am having difficulty with. I am using Angular.js.
Here is my API call:
function makeApiCall() {
gapi.client.load('gmail', 'v1', function() {
var request = gapi.client.gmail.users.messages.list({
labelIds: ['INBOX']
});
request.execute(function(resp) {
var content = document.getElementById("message-list");
angular.forEach(resp, function(message) {
var email = gapi.client.gmail.users.messages.get({'id': message.id});
// var raw = email.payload.parts;
// console.log(raw);
content.innerHTML += JSON.stringify(email) + "<br>";
})
});
});
}
So gapi.client.gmail.users.messages.list returns an array of my messages, with their ID numbers. That is working.
The call to gapi.client.gmail.users.messages.get({<specific message ID>}) outputs this - {"B":{"method":"gmail.users.messages.get","rpcParams":{},"transport":{"name":"googleapis"}}}.
Not sure what that is, but trying to get the message payload (email.payload.parts), results in undefined. So, how can I get the message content?
Also, I would assume that if I can get the message contents, I would then have to Base64 decode the contents to get some English out of it. Any suggestions for that would be of great help also. I've found this: https://github.com/kvz/phpjs, but since I'm not sure how to go about getting the message contents so that I can try and decode them, so not sure if that php.js is of an help in that regard.

Regarding the Base64 decoding, you can use
atob(dataToDecode)
For Gmail, you'll also want to replace some characters:
atob( dataToDecode.replace(/-/g, '+').replace(/_/g, '/') );
The above function is available to you in JavaScript (see ref). I use it myself to decode the Gmail messages. No need to install extra stuff. As an interesting tangent, if you want to encode your message to Base64, use btoa.
Now, for accessing your message payload, you can write a function:
var extractField = function(json, fieldName) {
return json.payload.headers.filter(function(header) {
return header.name === fieldName;
})[0].value;
};
var date = extractField(response, "Date");
var subject = extractField(response, "Subject");
referenced from my previous SO Question and
var part = message.parts.filter(function(part) {
return part.mimeType == 'text/html';
});
var html = atob(part.body.data.replace(/-/g, '+').replace(/_/g, '/'));

Depending on what your emails look like (single text/plain part? multipart with text/html? attachments, etc?) you may or may not have any "parts" in your email.payload and instead you'll have what you're looking for in "email.payload.body.data" (for single-part messages). This is all assuming you're doing a message.get with the default format ("full"). If you instead want to get the entire email in the message.raw field and deal with it in email libraries for your language you can call message.get(format=raw).
For more info check out the "body" and "parts[]" field documentation for "Message" at https://developers.google.com/gmail/api/v1/reference/users/messages

Ah! I figured it out. parts is an array, so I should have been calling it like: gapi.client.gmail.users.messages.get({'id': <message ID>}).payload.parts[0].body.data
Now my problem is decoding the emails, which is proving successful in plain text emails, but failing in emails from non-personal locations (businesses, social media update emails, etc.). But I'll make a new question to get answers for that.

You need to search where the body for a given mime type is, I have written a recursive function for that:
function searchBodyRec(payload, mimeType){
if (payload.body && payload.body.size && payload.mimeType === mimeType) {
return payload.body.data;
} else if (payload.parts && payload.parts.length) {
return payload.parts.flatMap(function(part){
return searchBodyRec(part, mimeType);
}).filter(function(body){
return body;
});
}
}
So now you can call
var encodedBody = searchBodyRec(this.message.payload, 'text/plain');
See the flatMap method up there? Classic FP method missing in js, here is how to add it (or you can use lodash.js, or underscore.js if you don't want to mess with the native objects)
Array.prototype.flatMap = function(lambda) {
return Array.prototype.concat.apply([], this.map(lambda));
};