What I'm trying to do
I'm building a node.js logging tool logminer to learn and expirement about logging/monitoring services. I found an interesting problem that I couldn't solve yet:
The problem
Let's say we want to log the URL and the IP and Location of the Client when an HTTP Request is emitted to see what users are entering into the omnibar (or for whatever reason).
Let's say the URL is logged right away, but for the IP we do an Async Request (maybe we want to get the location based on the IP)
// hypothetical http request handler
function httpRequestHandler(req, res){
// first log (after 0 seconds)
var log = new Log('Request')
if(req.url = 'home'){
// second log (after 0.1 seconds)
log.message('welcome home')
// ... do something
} else {
// second log (after 0.1 seconds);
log.message('the url is', req.url)
asyncFunction(req, function(value){
// third log (after 5 seconds)
log.message('the ip is ', value.ip)
log.message('the location is ', value.location)
// ... do something
})
}
}
The output can very quickly become from this:
(at +0 seconds) User Requests: http://example.com/other
Request
in Request: welcome home
(at +5 seconds) User Requests: http://example.com/
Request
in Request: the url is http://example.com/other
in Request: the ip is 127.0.0.1
in Request: the location is 'San Francisco'
Into this:
4 requests after another with 1 second between page request
(+0 second) User Requests: http://example.com/
(+1 second) User Requests: http://example.com/other <-- this needs 5 -seconds to finish
(+2 second) User Requests: http://example.com/
(+3 second) User Requests: http://example.com/
.
// at +0 second
Request
in Request: welcome home
// at +1 second
Request
in Request: the url is http://example.com/other
// at +2 second
Request
in Request: welcome home
// at +3 second
Request
in Request: welcome home
// at +5 second
in Request: the ip is 127.0.0.1
in Request: the location is 'San Francisco'
As you can see the one with +5 seconds creates confusion especially because it actually looks like this:
Request
in Request: welcome home
in Request: the ip is 127.0.0.1
in Request: the location is 'San Francisco'
And this doesn't make any sense! The 2nd and the 3rd rows are within a different Scope from the 1st one and this makes it seem like they are the same. With many concurent users this is completely unreadable. That's why I think there has to be a way to identify to the origins's of each line, that's why I have an ID in logminer. Including a Scope ID in every row works but it's still very hard to connect the dots and it takes a lots of screen-space as well.
The Question
Now my question is it possible to group things in an asynchronous environment based on their Scope without an ID or using Paging/Filters without a GUI?
Is this a design problem that cannot be solved because of the linear nature of terminals so a GUI is required at this point or am I missing something?
What are the best ways to handle this?
The usual solution to this is to not output your message in pieces that are separated in time because of async results.
Instead, accumulate all the data for your entire message into a string or various variables and then output the whole message at once. This will avoid some part of the message being output and then sometime later (after an async operation) so other part of the message being output.
If you want to make sure you know "when" the event occurred, then record a timestamp from the beginning of the event and include that in the message too.
For example:
// hypothetical http request handler
function httpRequestHandler(req, res){
// first log (after 0 seconds)
var log = new Log('Request');
if(req.url = 'home'){
// second log (after 0.1 seconds)
log.message('welcome home');
// ... do something
} else {
asyncFunction(req, function(value){
log.message('the url is', req.url);
log.message('the ip is ', value.ip);
log.message('the location is ', value.location);
// ... do something
})
}
}
The usual way is to gather all the information and print in stages. The problem becomes when you're impatiently waiting for things to finish and your async program has lots to do and you want some status update.
In that case you can use you can use the nodejs builtin readline to hack the terminal lines and overwrite the old information effectively dividing it into sections. However there is a few libraries out there that already do that, take logger master that does exactly that (find code # github). Depends on how slow your program is.
Related
I am trying to change the URL request on an already connected socket but I can't figure out how or if it is even possible.
I am working with the WebSocket API and CoinCap.
What I am doing right now is closing the connection and creating a new one with the new parameters.
// Create a new WS connection
const webSocketURL = `wss://ws.coincap.io/prices?assets=${loadedKeys}`
// loadedKeys could be a string of one coin (e.g. bitcoin) or an array
// or an array (e.g. bitcoin,ethereum,monero,litecoin), has to be dynamic.
pricesWs = new WebSocket(webSocketURL);
pricesWs.onopen = function () {
console.log(`conected: ${pricesWs.readyState}`)
}
pricesWs.onmessage = function (msg) {
handleUpdateCB(msg.data);
}
// then when I need to receive different coin prices
// I close the connection and reopen a new one.
anotherFunction() {
pricesWs.close();
pricesWs = new WebSocket(aNewWebSocketURL);
}
I tried sending parameters as messages with send() function without success, I keep receiving the same data, let's say I first connect asking for bitcoin and the I want to receive bitcoin and ethereum I tried this
pricesWs = new WebSocket(`wss://ws.coincap.io/prices?assets=bitcoin);
//then tried
pricesWs.send(bitcoin,ethereum)
this doesn't work, I also tried sending as JSON but I kept getting the same data just for the first query(bitcoin)
UPDATE:
This is the the Git for the app, if you are interested seeing the whole thing together.
Git
UPDATE 2:
I created this pen to make it easier to understand, note that the pen is made on VueJS, but that isn't important. The important part is on line 60 JS panel
Is there any reason why you want to switch the URL?
According to the coin cap documentation, you can request information about multiple crypto currency at once. Is it not an option for you?
Generally you should avoid opening and closing connections to a socket as there is slight latency albeit very insignificant. Leaving the connection open is better since you will be notified if price is changed for any of the currencies you are interested it.
The answer to your original question "Is it possible to change URL for a web socket connection?" is no! You can't change URL however you can create as many connections as you need. In your case you are closing the connection and opening it immediately but in the comments I noticed that you mentioned that it is based on user interaction. You can open connection just for the currencies you care about when user requests it and keep the connection opened until user switches the currency again because at that point you'll probably switch to another currency.
I also agree with #Taylor Spark, you can also just hide the dom for the currencies user don't care and render the ones they are interested in.
I was thinking about how to make an instant messaging application, and wanted to not have to send an AJAX request so often (one every .2s), and I came across the following idea:
Send an AJAX request from the user side, to the server.
Only respond once there is a change in the MySQL database
And then send the next AJAX request once the response has been recorded and parsed
I'm aware of how to do the first and third steps, but the second one is going over my head.
I'm assuming that for step 2, I'll need to store the request somewhere, while the PHP script is continuously looping and looking for some changes, and once there is a change, the saved request would be responded to.
EDIT
Didn't know about WebSockets, should've used those.
You could use recursion and query the database every 2 seconds, until you find new data to be served to the user. So basically you could do something like
public function isDataUpdated($lastId) {
$query = "SELECT * FROM `messages` WHERE `messages`.`message_id` > $lastId";
return (bool)(count($this->executeSQL($query)) > 0);
}
public function fetchNewMessages () {
if ($this->isDataUpdated($_GET['last_id'])) {
/* We have new data! Send it to the user */
} else {
sleep(2); // wait for 2 seconds
$this->fetchNewMessages(); // we use recursion to query the database every 2 seconds to find new data
}
}
Although, it is not the best of solutions, it would hopefully work. I would recommend taking a look at Sockets in PHP to better achieve what you want
I'm trying to figure out how to create calls from my Twilio number after I've dialed into it and entered a number. After reading the docs I see that this is done with the gather feature, which can then be redirected to another Twiml document to handle the response. However, I can't quite get it to work. I'm extremely confused on how to execute Twiml correctly and how to access the request parameters in another Twiml doc. I've also looked into Twimlets but I wasn't able to construct what I needed there correctly either.
I've gone back and tried to just make a simple voice message play when only my number calls. If it's not me calling then it needs to be redirected to a Twiml url which will try to connect to my phone. If that fails it will prompt the caller to leave a message.
//Handle incoming call requests
app.post('/call', function(req, res) {
var twiml = new twilio.TwimlResponse();
res.type('text/xml');
if ( req.body.From === "+1555555555") {
twiml.say('Hello', {voice: alice});
res.send(twiml.toString());
} else {
// Do something here.
}
});
I've found the correct solution for my problem. I wasn't initiating the twilio.TwimlResponse() correctly.
In order to solve this issue, I needed to use == instead of === so that my req.body.from value wouldn't get coerced.
So I have js that I want to only activate if it is snowing in the users area.
Here is a JSFiddle Here
<h1 onclick="snowStorm.toggleSnow();">
text
</h1>
<script src="https://web.archive.org/web/20161109225359/http://live.superedu.ml/snow.js"></script>
<script>
snowStorm.followMouse = false;
snowStorm.freezeOnBlur = true;
snowStorm.autoStart = false;
</script>
if you click test it activates snow.
how would I make this to activate the snow when it is snowing where the user is. thank you
First, you need to know if it's snowing or not at the user's location. So the very first thing you need to know is where they are. For this you use an IP geolocation service. Google it. One of them is https://freegeoip.net/
Next you need to ask a weather service about the weather.
Look into https://darksky.net/dev/docs/forecast
In order to ask something to a service like DarkSky API you will need to tell them the location you are interested in for the forecast, which is where you use the location coordinates you received from the geolocation service above.
You will then receive a response "object" from the DarkSky API, which will contain a bunch of information, amongst which the precipitation info as described below:
precipType optional
The type of precipitation occurring at the given time. If defined, this property will have one of the following values: "rain", "snow", or "sleet" (which refers to each of freezing rain, ice pellets, and “wintery mix”). (If precipIntensity is zero, then this property will not be defined.)
https://darksky.net/dev/docs/response
After which you can code something along the lines of
if (data.precipType === 'snow') { // do something }
Overall, it goes like this:
Send request to GeoIP (where is ip 8.8.8.8 ?)
Receive response from GeoIP (it's at lat:1.2345 lon:1.432)
Send request to DarkSky (what's the weather today at lat:1.2345 lon:1.432 ?)
Receive response from DarkSky (snow!)
… do stuff …
How
In order to succeed at this exercise, you will ned to familiarise a bit with basic asynchronous programming in JS, and how to send an AJAX request, and consume the response you receive.
In short you need to learn how to write code that is able to start function B only after function A is done (after an unknown amount of time, like when requesting over the net)
So things will look more like this:
1. Send request to GeoIP (where is ip 8.8.8.8 ?)
2. Receive response from GeoIP (it's at lat:1.2345 lon:1.432)
3. Send request to DarkSky (what's the weather today at lat:1.2345 lon:1.432 ?)
4. Receive response from DarkSky (snow!)
5. … do stuff …
Good keywords to search for this are jQuery AJAX, callbacks, and a tad more advanced, Promises. But start with the first two.
Good luck!
I'm trying to write back end functionality that is handling requests to particular API, but this API has some restrictive quotas, especially for requests/sec. I want to create API abstraction layer that is able of delaying function execution if there are too many requests/s, so it works like this:
New request arrives (to put it simple - library method is invoked)
Check if this request could be executed right now, according to given limit (requests/s)
If it can't be executed, delay its execution till next available moment
If at this time a new request arrives, delay its execution further or put it on some execution queue
I don't have any constraints in terms of waiting queue length. Requests are function calls with node.js callbacks as the last param for responding with data.
I thought of adding delay to each request, which would be equal to the smallest possible slot between requests (expressed as minimal miliseconds/request), but it can be a bit inefficient (always delaying functions before sending response).
Do you know any library or simple solution that could provide me with such functionality?
Save the last request's timestamp.
Whenever you have a new incoming request, check if a minimum interval elapsed since then, if not, put the function in a queue then schedule a job (unless one was already scheduled):
setTimeout(
processItemFromQueue,
(lastTime + minInterval - new Date()).getTime()
)
processItemFromQueue takes a job from the front of the queue (shift) then reschedules itself unless the queue is empty.
The definite answer for this problem (and the best one) came from the API documentation itself. We use it for a couple of months and it perfectly solved my problem.
In such cases, instead of writing some complicated queue code, the best way is to leverage JS possibility of handling asynchronous code and either write simple backoff by yourself or use one of many great libraries to use so.
So, if you stumble upon any API limits (e.g. quota, 5xx etc.), you should use backoff to recursively run the query again, but with increasing delay (more about backoff could be found here: https://en.wikipedia.org/wiki/Exponential_backoff). And, if finally, after given amount of times you fail again - gracefully return error about unavailability of the API.
Example use below (taken from https://www.npmjs.com/package/backoff):
var call = backoff.call(get, 'https://someaddress', function(err, res) {
console.log('Num retries: ' + call.getNumRetries());
if (err) {
// Put your error handling code here.
// Called ONLY IF backoff fails to help
console.log('Error: ' + err.message);
} else {
// Put your success code here
console.log('Status: ' + res.statusCode);
}
});
/*
* When to retry. Here - 503 error code returned from the API
*/
call.retryIf(function(err) { return err.status == 503; });
/*
* This lib offers two strategies - Exponential and Fibonacci.
* I'd suggest using the first one in most of the cases
*/
call.setStrategy(new backoff.ExponentialStrategy());
/*
* Info how many times backoff should try to post request
* before failing permanently
*/
call.failAfter(10);
// Triggers backoff to execute given function
call.start();
There are many backoff libraries for NodeJS, leveraging either Promise-style, callback-style or even event-style backoff handling (example above being second of the mentioned ones). They're really easy to use if you understand backoff algorithm itself. And as the backoff parameters could be stored in config, if backoff is failing too often, they could be adjusted to achieve better results.