In my Meteor 1.0 app, I'm running a batch of server-side HTTP requests in order to retrieve fixture data in a synchronous fashion. Once a request completes and computations are run on that data, startNumber is incremented (by 5000) and the request is re-run with that new value. This loop system will continue until the API returns a blank response array, signifying all the data has been captured. This HTTP request is part of a larger, complex function that helps set the context of the request.
functionName = function(param1,param2,param3) {
// ...
// ...
var startNumber = 1;
do {
var request = Meteor.http.call("GET", "https://url-to-api-endpoint",
{ params:
{
"since": startNumber
},
timeout: 60000
}
);
if(request.statusCode === 200) {
var response = request.data;
// perform calculations on the response
startNumber+=5000;
}
} (while response.length>0);
// ...
// ...
};
The do-while loop system is working fine, except that every few iterations the request is returning with Error: getaddrinfo ENOTFOUND. The URL is perfectly valid, and it appears these errors are resulting from a finicky/unreliable API as sometimes the same exact request will go through or error out. I want to replay failed requests in order to make sure my app is retrieving data chronologically before proceeding.
How can I replay a failed HTTP request as though it were being run for the first time? In other words, without losing the current context of all the variables, etc., in functionName?
FYI, incase someone else ends up in this predicament, I solved this problem by wrapping the HTTP request in a try-catch block. In the case of an error such as getaddrinfo ENOTFOUND or ETIMEDOUT, the error gets caught. Within the catch block, I call the functionName and pass in parameters for the current state (i.e. the current startNumber) - this allows me to essentially "replay" the request all over again.
// ...
// ...
try {
var request = Meteor.http.call("GET", "https://url-to-api-endpoint",
{ params:
{
"since": startNumber
},
timeout: 60000
}
);
} catch(err) {
console.log(err + '\nFailed...retrying');
functionName(param1,param2,param3);
}
// ...
// ...
Related
Im currently building a webapp that has two clear use cases.
Traditional client request data from server.
Client request a stream from the server after wich the server starts pushing data to the client.
Currently im implementing both 1 and 2 using json message passing over a websocket. However this has proven hard since I need to handcode lots of error handling since the client is not waiting for the response. It just sends the message hoping it will get a reply sometime.
Im using Js and react on the frontend and Clojure on the backend.
I have two questions regarding this.
Given the current design, what alternatives are there for error handling over a websocket?
Would it be smarter to split the two UC using rest for UC1 and websockets for UC2 then i could use something like fetch on the frontend for rest calls.
Update.
The current problem is not knowing how to build an async send function over websockets can match send messages and response messages.
Here's a scheme for doing request/response over socket.io. You could do this over plain webSocket, but you'd have to build a little more of the infrastructure yourself. This same library can be used in client and server:
function initRequestResponseSocket(socket, requestHandler) {
var cntr = 0;
var openResponses = {};
// send a request
socket.sendRequestResponse = function(data, fn) {
// put this data in a wrapper object that contains the request id
// save the callback function for this id
var id = cntr++;
openResponses[id] = fn;
socket.emit('requestMsg', {id: id, data: data});
}
// process a response message that comes back from a request
socket.on('responseMsg', function(wrapper) {
var id = wrapper.id, fn;
if (typeof id === "number" && typeof openResponses[id] === "function") {
fn = openResponses[id];
delete openResponses[id];
fn(wrapper.data);
}
});
// process a requestMsg
socket.on('requestMsg', function(wrapper) {
if (requestHandler && wrapper.id) {
requestHandler(wrapper.data, function(responseToSend) {
socket.emit('responseMsg', {id: wrapper.id, data; responseToSend});
});
}
});
}
This works by wrapping every message sent in a wrapper object that contains a unique id value. Then, when the other end sends it's response, it includes that same id value. That id value can then be matched up with a particular callback response handler for that specific message. It works both ways from client to server or server to client.
You use this by calling initRequestResponseSocket(socket, requestHandler) once on a socket.io socket connection on each end. If you wish to receive requests, then you pass a requestHandler function which gets called each time there is a request. If you are only sending requests and receiving responses, then you don't have to pass in a requestHandler on that end of the connection.
To send a message and wait for a response, you do this:
socket.sendRequestResponse(data, function(err, response) {
if (!err) {
// response is here
}
});
If you're receiving requests and sending back responses, then you do this:
initRequestResponseSocket(socket, function(data, respondCallback) {
// process the data here
// send response
respondCallback(null, yourResponseData);
});
As for error handling, you can monitor for a loss of connection and you could build a timeout into this code so that if a response doesn't arrive in a certain amount of time, then you'd get an error back.
Here's an expanded version of the above code that implements a timeout for a response that does not come within some time period:
function initRequestResponseSocket(socket, requestHandler, timeout) {
var cntr = 0;
var openResponses = {};
// send a request
socket.sendRequestResponse = function(data, fn) {
// put this data in a wrapper object that contains the request id
// save the callback function for this id
var id = cntr++;
openResponses[id] = {fn: fn};
socket.emit('requestMsg', {id: id, data: data});
if (timeout) {
openResponses[id].timer = setTimeout(function() {
delete openResponses[id];
if (fn) {
fn("timeout");
}
}, timeout);
}
}
// process a response message that comes back from a request
socket.on('responseMsg', function(wrapper) {
var id = wrapper.id, requestInfo;
if (typeof id === "number" && typeof openResponse[id] === "object") {
requestInfo = openResponses[id];
delete openResponses[id];
if (requestInfo) {
if (requestInfo.timer) {
clearTimeout(requestInfo.timer);
}
if (requestInfo.fn) {
requestInfo.fn(null, wrapper.data);
}
}
}
});
// process a requestMsg
socket.on('requestMsg', function(wrapper) {
if (requestHandler && wrapper.id) {
requestHandler(wrapper.data, function(responseToSend) {
socket.emit('responseMsg', {id: wrapper.id, data; responseToSend});
});
}
});
}
There are a couple of interesting things in your question and your design, I prefer to ignore the implementation details and look at the high level architecture.
You state that you are looking to a client that requests data and a server that responds with some stream of data. Two things to note here:
HTTP 1.1 has options to send streaming responses (Chunked transfer encoding). If your use-case is only the sending of streaming responses, this might be a better fit for you. This does not hold when you e.g. want to push messages to the client that are not responding to some sort of request (sometimes referred to as Server side events).
Websockets, contrary to HTTP, do not natively implement some sort of request-response cycle. You can use the protocol as such by implementing your own mechanism, something that e.g. the subprotocol WAMP is doing.
As you have found out, implementing your own mechanism comes with it's pitfalls, that is where HTTP has the clear advantage. Given the requirements stated in your question I would opt for the HTTP streaming method instead of implementing your own request/response mechanism.
I've been working on integrating Google Recaptcha into a Meteor and AngularJS web application. Everything was smooth sailing until I had to validate the recaptcha response -- for some bizarre reason, I can't get an async response from the backend to the frontend.
I've tried a lot of different variations and have read many, many posts on SO and the internet in general, but with no luck -- so I opted to post my own question.
Here's what I'm doing:
Client:
Meteor.call('recaptcha.methods.validateRecaptcha', { 'response' : this.recaptcha.getResponse(this.id) }, function(error, result) {
// error and result are both undefined
console.log('Do something with the ' + error + ' or ' + result + '.');
}
So, I'm calling a Meteor method and passing in a callback that is run when the method is done. However, the error and result parameters are both undefined.
Server:
run: function(data) {
if (this.isSimulation) {
/*
* Client-side simulations won't have access to any of the
* Meteor.settings.private variables, so we should just stop here.
*/
return;
}
return Meteor.wrapAsync(HTTP.post)(_someUrl, _someOptions);
}
That last line is a shortened version of the sync/async structure that I've found in several Meteor guides (I also tried this version), namely:
var syncFunc = Meteor.wrapAsync(HTTP.post);
var result = syncFunc(Meteor.settings.private.grecaptcha.verifyUrl, _options);
return result;
I've also tried a version using Futures:
var Future = Npm.require( 'fibers/future' );
var future = new Future();
var callback = future.resolver();
HTTP.post(Meteor.settings.private.grecaptcha.verifyUrl, _options, callback);
return future.wait();
Now, the intention here is that I use Meteor.call() to call this method from the client, the client-side stub runs (to prevent simulation errors since we use private Meteor.settings variables in the real non-SO server-side code) and returns immediately (which happens), and the server hits Google's Recaptcha API (which happens and the server receives a response) before returning the result to the client (which doesn't happen -- the callback occurs but with no error/success data).
My thought is that one of two things are happening:
I'm just doing something wrong and I'm not properly sending the data back to the client.
The synchronous client stub (which returns immediately) is telling the client that the server response isn't important, so it never waits for the proper asynchronous response.
Could any of the Meteor gurus weigh in here and let me know what's going on and how to get async requests to play nicely in a Meteor application?
Thanks!
From the documentation for HTTP.call, which is the generic version of HTTP.post, it says
Optional callback. If passed, the method runs asynchronously, instead of synchronously, and calls asyncCallback. On the client, this callback is required.
So, on server, you can run it asynchronously like this
run: function(data) {
if (this.isSimulation) {
/*
* Client-side simulations won't have access to any of the
* Meteor.settings.private variables, so we should just stop here.
*/
return;
}
// No need to pass callback on server.
// Since this part is not executed on client, you can do this
// Or you can use Meteor.isClient to run it asynchronously when the call is from client.
return HTTP.post(Meteor.settings.private.grecaptcha.verifyUrl, _options);
}
Is there a way to cancel requests/queries to Elasticsearch using elasticjs? The web app I am working on performs a request/query every 5 seconds, but I would like to cancel the request if for some reason the response doesn't show up in 5 seconds (so the browser doesn't pile up a bunch of requests that are unnecessary since the queries are happening repeatedly). I understand this would not prevent Elasticsearch from completing the query, but I would like to at least cancel the request in the browser.
Example:
var request = ejs.Request().doSearch();
var dataFromElasticsearch;
request.then(function (data) {
dataFromElasticsearch = data;
});
setTimeout(function () {
if (!dataFromElasticsearch) {
//do something here to cancel request
}
}, 5000)
Per documentation for elasticsearch.js (3.1, at the time of writing):
...calling the API will return an object (either a promise or just a plain object) which has an abort() method. Calling that abort method ends the HTTP request, but it will not end the work Elasticsearch is doing.
Specifically for your example:
setTimeout(function () {
if (!dataFromElasticsearch) {
request.abort();
}
}, 5000)
I'm writing a mobile app with Appcelerator Titanium that makes a lot of different xhr requests. This is not really an Appcelerator Titanium specific question. But if you do write some code, I hope it's javascript.
The app needs to authenticate itself, the user must be logged for some interactions, etc.
I've come to a point where any request might get any kind of response such as:
not authenticated
not logged
bad params
successful
...
The requests are wrapped in different model methods or helpers.
The thing is, I'm not familiar with this kind of app. I was wondering what are the best practices.
Some real questions for example would be:
If the app is not authenticated (token expired, first launch), should the app try to authenticate itself and then send again the request that was denied ? (transparent to user)
Should I send an authentication request each time the app launches and then "forget" about it?
The problem I'm facing is that the code becomes quickly big if I try to handle this for each request. Full of nested callbacks, retry conditions, various events listeners to manage, etc. It just does not feel very "nice". And it's not DRY at all, when what I really need is for any request, check what was wrong, try to fix it (authenticate if not, automatic login if possible or show the login UI, etc..) then if that works retry the original request a couple of times, abort if needed.
I've been looking at the promise pattern but only know theory and don't know if it could be what I need.
So I welcome any advice regarding this particular problem. I wonder how apps like "Facebook" handle this.
Thank you for your help
This question is not easily answered, but let me try to give you some Ideas:
The most important thing, before coding anything in your app, is the API itself. It has to be reliable and adhere to standards. I will not go into too much detail here, but a well written RESTful API can reduce the complexity of your httpClient significantly. It has to respond with standard http status codes and to methods like POST, GET, PUT, DELETE...
A pretty good read is The REST API Design Handbook by George Reese.
My approach to httpClients with Titanium is a single module, which is loaded via require() wherever needed. I stick to one single client at a time, as I had massive problems with multiple parallel calls. Whenever a call is made, the client checks if there is already a call in progress and sends it to a queue if necessary.
Let me show you an example. I have left out lots of stuff for sake of brevity:
// lib/customClient.js
var xhrRequest; // This will be our HTTPClient
var callQueue = []; // This will be our queue
// Register the request
// params are:
// method (e.g. 'GET')
// url (e.g. 'http://test.com/api/v1/user/1')
// done (callback function)
function registerRequest(params) {
if(!xhrRequest) {
sendRequest(params);
} else {
queueRequest(params);
}
}
// This simply sends the request
// to the callQueue
function queueRequest(params) {
callQueue.push(params);
}
// Send the request with the params from register
// Please note that I do not hardcode error messages,
// I just do it here so it is easier to read
function sendRequest(params) {
// Set callback if available and valid
var callback = params.done && typeof(params.done) === "function" ? params.callback : null;
// Set method
var method = params.method || 'GET';
// Create the HTTP Client
xhrRequest = Ti.Network.createHTTPClient({
// Success
onload: function() {
// You can check for status codes in detail here
// For brevity, I will just check if it is valid
if (this.status >= 200 && this.status < 300) {
if(this.responseText) {
// You might want to check if it can be parsed as JSON here
try {
var jsonData = JSON.parse(this.responseText);
if(callback) callback({ success: true, response: jsonData });
} catch(e) {
if(callback) callback({ success: false, errormessage: 'Could not parse JSON data' });
}
processQueue();
} else {
if(callback) callback({ success: false, errormessage: 'No valid response received' });
processQueue();
}
} else {
if(callback) callback({ success: false, errormessage: 'Call response is success but status is ' + this.status });
processQueue();
}
},
// Error
onerror: function(e) {
if(this.responseText) {
try {
var jsonData = JSON.parse(this.responseText);
if(callback) callback({ success: false, reponse: jsonData });
} catch(e) {};
}
processQueue();
},
});
// Prepare and send request
// A lot more can (and should) be configured here, check documentation!
xhrRequest.setTimeout(10000);
xhrRequest.open(method, params.url);
xhrRequest.send();
}
// Checks if there is anything else in the queue
// and sends it
function processQueue() {
xhrRequest = null;
var nextInQueue = callQueue.shift();
if(nextInQueue) sendRequest(nextInQueue);
}
// Our public API
var publicAPI = {
sendRequest: function(params) {
registerRequest(params);
}
};
module.exports = publicAPI;
I can then send a call from any other controller/view
var customClient = require('lib/customClient'); // omit 'lib' if you use alloy
// Send the request
customClient.sendRequest({
method : 'GET',
url : 'http://test.com/api/v1/user/1',
done : function(response) {
Ti.API.debug(JSON.stringify(response));
}
});
Note that this is not complete and does not check for connectivity, has no real error handling etc., but it might help you to get an idea.
I think there is loads of stuff to talk about here, but I will stop here for now...
I have a use case where my http requests are caching the intermediate result on server.
If the cache is not present the request builds it by requesting another server.
These requests are fired in succession (loop) using AJAX to Node Server and the number of requests can be in range of 50 to 500.
The Problem:
Since the requests are made in a loop and the cache is already not present first few of them all try to build the cache and sometimes consequent requests find the semi-built cache, which returns wrong result.
I can circumvent this problem with polling:
(function next(){
if(!wait){
fs.readFile(cacheFile, function(err){
if(err) {
wait = true;
createCache(); // sets wait = false;
} else {
useCache();
}
});
} else {
setTimeout(next,waitTime);
}
})();
My Query:
Can the requests be halted without polling, and continue only after the first request has completed the cache building process?
Yes, it is possible in combination with Futures/Promise. You can take this one.
Outside of the scope define var cachePromise and you can use something like this below:
if (!cachePromise) {
cachePromise = require('future').create()
buildCache(function() {
cachePromise.fulfill();
});
}
cachePromise.when(next); // this one triggers next route in middleware stack
Put the code in route stack before the route which gives result and you are good to go.
thanks.