I'm facing troubles trying to accomplish pretty banal task. I need to create nodejs Readable stream from input txt file. I need to perform some transforming on this stream (create JSON object for each line).
Problem is that I want this stream to be infinitive: after last line is read, stream should just start from beginning. My solution works bit I'm getting warning message:
(node) warning: possible EventEmitter memory leak detected. 11 drain listeners added. Use emitter.setMaxListeners() to increase limit.
I hoped to find simple solutions without reading and buffering file directly.
//Transform stream object
var TransformStream = function () {
Transform.call(this, {objectMode: true});
};
util.inherits(TransformStream, Transform);
TransformStream.prototype._transform = onTransform;
TransformStream.prototype._flush = onEnd;
var ts = new TransformStream();
var infinStream = function () {
var r = fs.createReadStream(filePath);
r.pipe(split(), {end: false})
.pipe(ts, {end: false});
r.once('end', function () {
//console.log('\n\n\n\nRead file stream finished. Lines counted:\n\n\n\n' + detectionCounter);
r.removeAllListeners();
r.destroy();
infinStream();
});
return r;
};
infinStream();
return ts;
From the comments:
I need server that will be live for 24h/day, and will simulate device
output all the time.
To do that a recursive function is a good idea. The approach you make is ok. When you don't need different transforming tasks on your data, a stream is not really needed. Simple Events can do exactly what you want and they are easier to understand.
The error in your code is the point, where you put your listener. The listenster r.once is inside your recursive function. You are defining r inside your function, so with every function call a new r is created. Because of that r.once does not work like you are expecting it.
What you can do:
Make a recursive function which emitts an event
use the data from your event outside
This is just a simple concept by using simple events, which are fireing the whole time the data from your file:
// Your recursive function
var simulateDeviceEvents = function(){
fs.readFile('device.txt', function (err, data) {
if (err) throw err;
// Just emit the event here
Emitter.emit('deviceEvent', data);
});
//If this happens to fast you could also call it with
//a timeout.
simulateDeviceEvents();
};
// Start the function
simulateDeviceEvents();
//IMPORTANT: The listener must be defined outside your function!
Emitter.on('deviceEvent', function(data){
// Do something with your data here
});
Related
I'm beginner to NodeJS, so I'm not entirely sure what the best method to achieve this would be. Basically I want to create a global variable with a string, for instance 'USD', that would get updated whenever my 'set currency' event is fired. I want it to remain that way until the event is called again.
I am using EventEmitter to fire off some events, in one of my files I have the following.
var event = require('./events');
if (msg.content.includes('!currency set currency')) {
split = msg.content.split(' ');
event.emit('setCurrency', split[3])
}
And then inside the events file I'm doing something like the following.
var exchangePref;
var event = new events.EventEmitter();
event.on('setExchange', (exchange) => {
exchangePref = exchange;
return exchangePref;
});
modules.exports = event;
I understand that re-writing the variable inside a callback isn't going to do what I need it to do, but I'm quite lost with how to achieve what I need it to do due to the modules.exports = event part at the bottom, the calling function simply never gets the data. I've played around with creating a constructor, but even still I couldn't get it to work.
Any suggestions/ideas would be greatly appreciated.
I wouldn't use event emitter for this. Instead create a module along the lines of:
var exchangePrefs = { currency: "JPY" };
module.exports = {
setCurrency : function(newVal){ exchangePrefs.currency = newVal; },
getCurrency : function(){ return exchangePrefs.currency; }
};
Then in your various other modules you just:
require('./mymodule').setCurrency('USD');
and somewhere else
var currency = require('./mymodule').getCurrency();
I'm sure it can be made prettier, but I think you get the point. For almost all intents and purposes modules work like singletons. There are some gotchas, but nothing you'll run into too often. (Singleton pattern in nodejs - is it needed?)
Personally I'd use some sort of data persistence in the exchangePref-module just for peace of mind. Like redis, or saving to a json-file.
I wanted to use rxjs for the first time but am a bit stucked 'cause it doesn't behave exactly like I want it to: In my scenario I want to create an observable from a promise. But I want the promise only being called once (not on every subscription) and I want it not being called on creation time (defer the call to the first subscription).
First I tried this:
var source = Rx.Observable.fromPromise(_this.getMyPromise())
which causes a call to the getMyPromise function right on creation time. This is not satisfying because at that time I don't know if the source really will be used.
Then I tried:
var source = Rx.Observable.defer(function() { return _this.getMyPromise() })
which causes a call to the getMyPromise function each time a new subscription is being made to source. This makes way too many unnecessary calls to the web server. The Rx.Observable.create function seems to have the same issue.
So what is left or what am I missing?
.shareReplay() does this, e.g.:
var source = Rx.Observable.defer(function() { return _this.getMyPromise() }).shareReplay();
If you're using rxjs5, you'll want to read: Pattern for shareReplay(1) in RxJS5
In answer to your comment below, I can think of a fairly straightforward extension to the above logic that will do what you want, but it has a caveat. Let's say the events you want to use to trigger a "refresh" are represented in a stream, s$, then you could do something like:
var source = Rx.Observable.of({}).concat(s$)
.flatMapLatest(function() {
return Rx.Observable.defer(function() {
return _this.getMyPromise()
})
})
.shareReplay(1)
What we have here is a stream starting with a dummy object to get things rolling, followed by a stream consisting of your refresh events. Each of these is projected into a new observable created from a fresh invocation of your getMyPromise method, and the whole thing is flattened into a single stream. Finally, we keep the shareReplay logic so we only actually make calls when we should.
The caveat is that this will only work properly if there's always at least one subscriber to the source (the first subscription after all others are disposed will run the promise again, and will receive both the previously-cached value and the result of the promise it caused to run).
Here is an answer that does not require at least one subscriber at the source at all times using a simple helper:
var _p = null;
var once = function() { return _p || (_p = _this.getMyPromise());
var source = Rx.Observable.defer(once);
Or if you're using lodash, you can _.memoize your getMyPromise and get this automatically.
I am developing a web application in node.js to collect data from devices on a network using snmp. This is my first real encounter with node.js and javascript. In the app each device will be manipulated through a module I named SnmpMonitor.js. This module will maintain basic device data as well as the snmp and database connection.
One of the features of the app is the ability to constantly monitor data from smart metering devices. To do this I created the following code to start and stop the monitoring of the device. It uses setInterval to constantly send a snmp get request to the device. Then the event listener picks it up and will add the collected data to a database. Right now the listener just prints to show it was successful.
var dataOIDs = ["1.3.6.1.2.1.1.1.0","1.3.6.1.2.1.1.2.0"];
var intervalDuration = 500;
var monitorIntervalID;
var dataCollectionEvent = "dataCollectionComplete";
var emitter = events.EventEmitter(); // Uses native Event Module
//...
function startMonitor(){
if(monitorIntervalID !== undefined){
console.log("Device monitor has already started");
} else {
monitorIntervalID = setInterval(getSnmp,intervalDuration,dataOIDs,dataCollectionEvent);
emitter.on(dataCollectionEvent,dataCallback);
}
}
function dataCallback(recievedData){
// receivedData is returned from getSnmp completion event
// TODO put data in database
console.log("Event happened");
}
function stopMonitor(){
if(monitorIntervalID !== undefined){
clearInterval(monitorIntervalID);
emitter.removeListener(dataCollectionEvent,dataCallback);
} else {
console.log("Must start collecting data before it can be stopped");
}
}
//...
I also have a test file, test.js, that requires the module, starts monitoring, waits 10 seconds, then stops it.
var test = require("./SnmpMonitor");
test.startMonitor();
setTimeout(test.stopMonitor,10000);
My problem is that the setInterval function in startMonitor() is not being run. I have tried placing console.log("test"); before, inside, and after it to test it. The inside test output never executes. The monitorIntervalID variable is also returned as undefined. I have tested setInterval(function(){ console.log("test"); },500); in my test.js file and it runs fine with no issues. I feel like this is a noobie mistake but I just can't seem to figure out why it won't execute.
Here is a link to the entire module: SnmpMonitor.js
I not sure exactly what was wrong but I got it to work by overhauling the whole class/module. I thought the way I had it was going to allow me to create new monitors objects but I was wrong. Instead I created two functions inside the monitor file that do the same thing. I changed the start function to the following.
SnmpMonitor.prototype.start = function() {
var snmpSession = new SNMP(this.deviceInfo.ipaddress,this.emitter);
var oids = this.deviceInfo.oids;
var emit = this.emitter;
var duration = this.intervalDuration;
this.intervalID = setInterval(function(){
snmpSession.get(dataCollectionEvent,emit,oids);
},duration);
};
The setInterval function seems to work best when the callback function is set inside an anonymous function, even though technically you can pass it directly. Using the this. notation I created some class/module/function variables (whatever its called in js) that are in scope of the whole class. For some reason the variables accessed through this. do not work so well when directly in a function or expression so I created temp variables for them. In my other version all the variables were global and js doesn't seem to like that.
There must be something simple I am missing, but alas, I do not know what I do not know. Below is the code I have thus far for trying to get current streamflow conditions from the USGS.
// create site object
function Site(siteCode) {
this.timeSeriesList = [];
this.siteCode = siteCode;
this.downloadData = downloadData;
this.getCfs = getCfs;
// create reference to the local object for use inside the jquery ajax function below
var self = this;
// create timeSeries object
function TimeSeries(siteCode, variableCode) {
this.variableCode = variableCode;
this.observations = [];
}
// create observation object
function TimeSeriesObservation(stage, timeDate) {
this.stage = stage;
this.timeDate = timeDate;
}
// include the capability to download data automatically
function downloadData() {
// construct the url to get data
// TODO: include the capability to change the date range, currently one week (P1W)
var url = "http://waterservices.usgs.gov/nwis/iv/?format=json&sites=" + this.siteCode + "&period=P1W¶meterCd=00060,00065"
// use jquery getJSON to download the data
$.getJSON(url, function (data) {
// timeSeries is a two item list, one for cfs and the other for feet
// iterate these and create an object for each
$(data.value.timeSeries).each(function () {
// create a timeSeries object
var thisTimeSeries = new TimeSeries(
self.siteCode,
// get the variable code, 65 for ft and 60 for cfs
this.variable.variableCode[0].value
);
// for every observation of the type at this site
$(this.values[0].value).each(function () {
// add the observation to the list
thisTimeSeries.observations.push(new TimeSeriesObservation(
// observation stage or level
this.value,
// observation time
this.dateTime
));
});
// add the timeSeries instance to the object list
self.timeSeriesList.push(thisTimeSeries);
});
});
}
// return serialized array of cfs stage values
function getCfs() {
// iterate timeseries objects
$(self.timeSeriesList).each(function () {
// if the variable code is 00060 - cfs
if (this.variableCode === '00060') {
// return serialized array of stages
return JSON.stringify(this.observations);
}
});
}
}
When I simply access the object directly using the command line, I can access individual observations using:
> var watauga = new Site('03479000')
> watauga.downloadData()
> watauga.timeSeriesList[0].observations[0]
I can even access all the reported values with the timestamps using:
> JSON.stringify(watauga.timeSeriesList[0].observations)
Now I am trying to wrap this logic into the getCfs function, with little success. What am I missing?
I don't see anything in the code above that enforces the data being downloaded. Maybe in whatever execution path you're using to call getCfs() you have a wait or a loop that checks for the download to complete prior to calling getCfs(), but if you're simply calling
site.downloadData();
site.getCfs()
you're almost certainly not finished loading when you call site.getCfs().
You'd need to do invoke a callback from within your success handler to notify the caller that the data is downloaded. For example, change the signature of Site.downloadData to
function downloadData(downloadCallback) {
// ...
Add a call to the downloadCallback after you're finished processing the data:
// After the `each` that populates 'thisTimeSeries', but before you exit
// the 'success' handler
if (typeof downloadCallback === 'function') {
downloadCallback();
}
And then your invocation would be something like:
var watauga = new Site('03479000');
var downloadCallback = function() {
watauga.timeSeriesList[0].observations[0];
};
watauga.downloadData(downloadCallback);
That way, you're guaranteed that the data is finished processing before you attempt to access it.
If you're getting an undefined in some other part of your code, of course, then there may be something else wrong. Throw a debugger on it and step through the execution. Just bear in mind that interactive debugging has many of the same problems as interactively calling the script; the script has time to complete its download in the background before you start inspecting the variables, which makes it look like everything's hunky dory, when in fact a non-interactive execution would have different timing.
The real issue, I discovered through just starting over from scratch on this function, is something wrong with my implementation of jQuery.().each(). My second stab at the issue, I successfully used a standard for in loop. Here is the working code.
function getCfs() {
for (var index in this.timeSeriesList) {
if (this.timeSeriesList[index].variableCode === '00060'){
return JSON.stringify(this.timeSeriesList[index].observations);
}
}
}
Also, some of the stuff you are talking about #Palpatim, I definitely will have to look into. Thank you for pointing out these considerations. This looks like a good time to further investigate these promises things.
I've written happily a node.js server, which uses socket.io to communicate with the client.
this all works well.
the socket.on('connection'...) handler got a bit big, which made me think of an alternative way to organize my code and add the handlers in a generator function like this:
sessionSockets.on('connection', function (err, socket, session) {
control.generator.apply(socket, [session]);
}
the generator takes an object that contains the socket events and their respective handler function:
var config = {
//handler for event 'a'
a: function(data){
console.log('a');
},
//handler for event 'b'
b: function(data){
console.log('b');
}
};
function generator(session){
//set up socket.io handlers as per config
for(var method in config){
console.log('CONTROL: adding handler for '+method);
//'this' is the socket, generator is called in this way
this.on(method, function(data){
console.log('CONTROL: received '+method);
config[method].apply(this, data);
});
}
};
I was hoping that this would add the socket event handlers to the socket, which it kind of does, but when any event comes in, it always calls the latest one added, in this case always the b-function.
Anyone any clues what i am doing wrong here?
The problem appears because by that time this.on callback triggers (let's say in a few seconds after you bind it), the for loop is finished and method variable becomes the last value.
To fix that you may use some JavaScript magic:
//set up socket.io handlers as per config
var socket = this;
for(var method in config){
console.log('CONTROL: adding handler for '+method);
(function(realMethod) {
socket.on(realMethod, function(data){
console.log('CONTROL: received '+realMethod);
config[realMethod].apply(this, data);
});
})(method); //declare function and call it immediately (passing the current method)
}
This "magic" is hard to understand when you first see it, but when you get it, the things become clear :)