Develop Reference

How to handle DirectLine connection errors

we run a bot using bot framework and connect to it from our website using direct line in JS. We get a token from a custom API endpoint and we store the token in sessionStorage. Then we connect to the bot using
directLine = await window.WebChat.createDirectLine({
token,
conversationId,
watermark: "0"
});
Everything is working fine, but when I leave the page open for too long the token in the sessionStorage expires. A page refresh or navigating to a different page causes a 403 error inside the createDirectLine method. Resulting in a chat bot that can't connect for as long as the sessionStorage holds that token. This behavior is not a surprise to me, but I don't know how to handle this.
What I want is to simply clear the sessionStorge, request a new token and start a new conversation when this happens. But I don't know how to do that. How do I get the 403 error from the createDirectLine method? Or is there a way to validate the token upfront?
I already tried putting a try/catch block around the createDirectLine method, but the 403 error did not show up in the catch.
Thanks in advance!

This solution is only to address the 403 error from occurring since the token expires (in 30 minutes I think). A better solution is to store the conversationId with the token and get a new token with that. Check official bot service documentation for that.
// to shorten code, we store in sessionStorage as separate items.
const expirationDuration = 1000 * 60 * 30; // 30 minutes
const currentTime = new Date().getTime();
const timeTokenStored = sessionStorage.getItem("timeTokenStored") || currentTime;
// if token is stored over 30 minutes ago, ignore it and get a new one. Otherwise, use it.
if ((currentTime - timeTokenStored) > expirationDuration) {
const res = await fetch('https://<yourTokenEndpoint>', { method: 'POST' });
const { token } = await res.json();}
const currentTime = new Date().getTime();
sessionStorage.setItem("timeTokenStored", currentTime);
sessionStorage.setItem('token', token);
else {
const token = sessionStorage.getItem("token")
}
While your're at it, you might as well store it in localStorage. This way, your bot will follow the user.

I have found the solution. We can check that a token is valid by refreshing it. If refreshing causes an error, the token is not valid anymore. If refreshing succeeds, the toke will be valid for another hour.
So we added (resusable) a function to our back end to refresh the token using https://directline.botframework.com/v3/directline/tokens/refresh.
And we changed the front end code to call our new refresh function.
Front end code:
// Gets a new token from the cloud.
async function requestToken() {
if (!sessionStorage['webchatToken']) {
const res = await fetch('https://' + serviceName + '.azurewebsites.net/api/token');
// If the request was succesfull, store the token and userId.
if (res.status == 200) {
const jsonResult = await res.json();
sessionStorage['webchatToken'] = jsonResult.token;
sessionStorage['webchatUserId'] = jsonResult.userId;
console.log(`Got token from cloud`);
// refresh the token every 15 minutes.
setTimeout(() => {
refreshToken();
}, 60000 * 15); // 15 minutes
}
// If the request was not succesfull, retry.
else {
console.log(`Tried to get token, but goterror ` + res.status + `. Retrying.`);
await requestToken();
}
}
// If there is already a token in storage, refresh the existing one instead of requesting a new one.
else {
console.log(`Got token from sessionStorage`);
await refreshToken();
}
}
// Refreshes an existing token so it doesn't expire.
async function refreshToken() {
// Refresh the token if it exists in storage.
if (sessionStorage['webchatToken']) {
const res = await fetch('https://' + serviceName + '.azurewebsites.net/api/token/refresh?token=' + sessionStorage['webchatToken'],
{
method: 'POST'
});
// If refresh was succesfull we are done.
if (res.status == 200) {
console.log(`Refreshed token`);
}
// If refresh was not succesfull, clear the token from storage and request a new one. The token is probably expired.
else {
console.log(`Tried to refresh token, but got error ` + res.status + `. Requesting new token.`);
sessionStorage.clear();
await requestToken();
}
}
// If there is no token in storage, request a new token.
else {
console.log(`Tried to refresh token, but token is not defined. Requesting new token.`);
sessionStorage.clear();
await requestToken();
}
}
Back end code:
[HttpGet]
[Route("api/token")]
public async Task<ObjectResult> GetToken()
{
HttpClient client = new HttpClient();
HttpRequestMessage request = new HttpRequestMessage(
HttpMethod.Post,
$"https://directline.botframework.com/v3/directline/tokens/generate");
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", _configuration.DirectLineKey);
var userId = $"dl_{Guid.NewGuid()}";
request.Content = new StringContent(
JsonConvert.SerializeObject(new { User = new { Id = userId } }),
Encoding.UTF8,
"application/json");
var response = await client.SendAsync(request);
string token = String.Empty;
int expiresIn = 0;
if (response.IsSuccessStatusCode)
{
var body = await response.Content.ReadAsStringAsync();
token = JsonConvert.DeserializeObject<DirectLineToken>(body).token;
expiresIn = JsonConvert.DeserializeObject<DirectLineToken>(body).expires_in;
}
return Ok(new { token, userId, expiresIn });
}
[HttpPost]
[Route("api/token/refresh/")]
public async Task<ObjectResult> RefreshToken(string token)
{
HttpClient client = new HttpClient();
HttpRequestMessage request = new HttpRequestMessage(
HttpMethod.Post,
$"https://directline.botframework.com/v3/directline/tokens/refresh");
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
var response = await client.SendAsync(request);
token = String.Empty;
int expiresIn = 0;
if (response.IsSuccessStatusCode)
{
var body = await response.Content.ReadAsStringAsync();
token = JsonConvert.DeserializeObject<DirectLineToken>(body).token;
expiresIn = JsonConvert.DeserializeObject<DirectLineToken>(body).expires_in;
}
if (string.IsNullOrEmpty(token))
return Problem("Token incorrect");
return Ok(new { token, expiresIn });
}
I hope posting this may be useful to somebody.

Empty Response headers received from WebAPI

I'm using the ASP.NET WebApi with ReactJs application in the front, I'm creating a Get method to download file from the server, and I trying to set both Content-Type and Content-Length in the response headers :
var result = new HttpResponseMessage(HttpStatusCode.OK);
result.Content = new StreamContent(new MemoryStream(bytes));
result.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
result.Content.Headers.ContentLength = bytes.Length;
When I'm calling this method from the ReactJs Application using fetch method like below:
await fetch(`someservice/${clientid}/download/${fileName}`, { responseType: "arraybuffer" })
.then((response) => {
const reader = response.body.getReader();
//get total length
const contentLength = response.headers.get('Content-Length');
console.log(response.headers);
//read the data
let receivedLength = 0; // received that many bytes at the moment
let chunks = []; // array of received binary chunks (comprises the body)
while (true) {
const { done, value } = reader.read();
if (done) {
break;
}
console.log(value);
chunks.push(value);
receivedLength += value.length;
console.log(`Received ${receivedLength} of ${contentLength}`)
}
//concatenate chunks into single Uint8Array
let chunksAll = new Uint8Array(receivedLength);
let position = 0;
for (let chunk of chunks) {
chunksAll.set(chunk, position);
position += chunk.length;
}
});
I got a response without Content-Type and Content-Length headers:
But isn't Content-Type and Content-Length a valid headers ?

response headers in fetch api JavaScript has been implemented as an iterator rather than an object , you could try below to access the headers :-
for (var header of respone.headers.entries()) {
console.log(header);
}
each header item in the loop will be an array of string where the first element is the key and 2nd element is value.

Design fetch with cache mechanism using SessionStorage: why does it converting the cached result to blob when making new response

https://www.sitepoint.com/cache-fetched-ajax-requests/
I am reading this article on making a fetch with SessionStorage as its cache. When the url is identical, we read from SessionStorage to fulfill the request.
const cachedFetch = (url, options) => {
// Use the URL as the cache key to sessionStorage
let cacheKey = url
// START new cache HIT code
let cached = sessionStorage.getItem(cacheKey)
if (cached !== null) {
// it was in sessionStorage! Yay!
let response = new Response(new Blob([cached]))
return Promise.resolve(response)
}
// END new cache HIT code
return fetch(url, options).then(response => {
// let's only store in cache if the content-type is
// JSON or something non-binary
if (response.status === 200) {
let ct = response.headers.get('Content-Type')
if (ct && (ct.match(/application\/json/i) || ct.match(/text\//i))) {
// There is a .json() instead of .text() but
// we're going to store it in sessionStorage as
// string anyway
// If we don't clone the response, it will be
// consumed by the time it's returned. This
// way we're being un-intrusive.
response.clone().text().then(content => {
sessionStorage.setItem(cacheKey, content)
})
}
}
return response
})
}
What I don't understand is why do we need to do this in order to make a new response?
let response = new Response(new Blob([cached]))
I understand that sessionStorage only stores text, and I remember blob is more related to binary data like images/ videos/ audios. So why do we need to convert the text into blob first in order to make a new Response object.
I also asked a related question before What should the intermediate result that a fetch request returns be called? A blob or just response?
Essential it's about whether a response object is also a blob and the answer seems to be no

How do i read a large SOLR response object by object while the response is still returning

I'm querying SOLR7.5 for some large objects and would like to render them to a Browser UI as they are returned.
What are my options for reading the response bit by bit using when using the select request handler

I don't think there is anything native to Solr to do what you are asking.
One approach to handle this would be to return only the ID of the documents that match the criteria in your query (and not include the heavy part of the document) and then fetch the large part of the document asynchronously from the client.

i was looking in the wrong place. I just needed to read up on my webAPI fetch().
the response.json() reads the response to completion.
response.body.getReader() allows you to grab the stream in chunk and decode it from there.
let test = 'https://my-solr7/people/select?q=something'
fetchStream(test);
function fetchStream(uri, params = {}){
const options = {
method: 'GET',
};
var decoder = new TextDecoder();
fetch(uri, options)
.then ()
.then( (response) => {
let read;
const reader = response.body.getReader();
reader.read()
.then(read = (result) => {
if (result.done) return;
console.log(result.value);
let chunk = decoder.decode(result.value || new Uint8Array, {stream: !result.done});
console.log(chunk)
reader.read().then(read);
});
});
}

Fetch vs Request

I'm consuming a JSON stream and am trying to use fetch to consume it. The stream emits some data every few seconds. Using fetch to consume the stream gives me access to the data only when the stream closes server side. For example:
var target; // the url.
var options = {
method: "POST",
body: bodyString,
}
var drain = function(response) {
// hit only when the stream is killed server side.
// response.body is always undefined. Can't use the reader it provides.
return response.text(); // or response.json();
};
var listenStream = fetch(target, options).then(drain).then(console.log).catch(console.log);
/*
returns a data to the console log with a 200 code only when the server stream has been killed.
*/
However, there have been several chunks of data already sent to the client.
Using a node inspired method in the browser like this works every single time an event is sent:
var request = require('request');
var JSONStream = require('JSONStream');
var es = require('event-stream');
request(options)
.pipe(JSONStream.parse('*'))
.pipe(es.map(function(message) { // Pipe catches each fully formed message.
console.log(message)
}));
What am I missing? My instinct tells me that fetch should be able to mimic the pipe or stream functionality.

response.body gives you access to the response as a stream. To read a stream:
fetch(url).then(response => {
const reader = response.body.getReader();
reader.read().then(function process(result) {
if (result.done) return;
console.log(`Received a ${result.value.length} byte chunk of data`);
return reader.read().then(process);
}).then(() => {
console.log('All done!');
});
});
Here's a working example of the above.
Fetch streams are more memory-efficient than XHR, as the full response doesn't buffer in memory, and result.value is a Uint8Array making it way more useful for binary data. If you want text, you can use TextDecoder:
fetch(url).then(response => {
const reader = response.body.getReader();
const decoder = new TextDecoder();
reader.read().then(function process(result) {
if (result.done) return;
const text = decoder.decode(result.value, {stream: true});
console.log(text);
return reader.read().then(process);
}).then(() => {
console.log('All done!');
});
});
Here's a working example of the above.
Soon TextDecoder will become a transform stream, allowing you to do response.body.pipeThrough(new TextDecoder()), which is much simpler and allows the browser to optimise.
As for your JSON case, streaming JSON parsers can be a little big and complicated. If you're in control of the data source, consider a format that's chunks of JSON separated by newlines. This is really easy to parse, and leans on the browser's JSON parser for most of the work. Here's a working demo, the benefits can be seen at slower connection speeds.
I've also written an intro to web streams, which includes their use from within a service worker. You may also be interested in a fun hack that uses JavaScript template literals to create streaming templates.

Turns out I could get XHR to work - which doesn't really answer the request vs. fetch question. It took a few tries and the right ordering of operations to get it right. Here's the abstracted code. #jaromanda was right.
var _tryXhr = function(target, data) {
console.log(target, data);
var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function () {
console.log("state change.. state: "+ this.readyState);
console.log(this.responseText);
if (this.readyState === 4) {
// gets hit on completion.
}
if (this.readyState === 3) {
// gets hit on new event
}
};
xhr.open("POST", target);
xhr.setRequestHeader("cache-control", "no-cache");
xhr.setRequestHeader("Content-Type", "application/json");
xhr.send(data);
};