Develop Reference

Node.js garbage collection and synchronized objects

This is a little bit tricky to explain, but I'll give it a try:
In a node.js server application I would like to deal with data objects that can be used in more than one place at once. The main problem is, that these objects are only referred to by an object id and are loaded from the database.
However, as soon as an object is already loaded into one scope, it should not be loaded a second time when requested, but instead the same object should be returned.
This leads me to the question of garbage collection: As soon as an object is no longer needed in any scope, it should be released completely to prevent having the whole database in the server's memory all the time. But here starts the problem:
There are two ways I can think of to create such a scenario: Either use a global object reference (which prevents any object from being collected) or, really duplicate these objects but synchronize them in a way that each time a property in one scope gets changed, inform the other instances about that change.
Again, therefore each instance would have to register an event handler, which in turn is pointing back to that instance thus preventing it from being collected again.
Did anyone come up with a solution for such a scenario I just didn't realize? Or is there any misconception in my understanding of the garbage collector?
What I want to avoid is manual reference counting for every object in the memory. Everytime when an object is being removed from any collection, I would have to adapt the reference count manually (there is even no destructor or "reference decreased" event in js)

Using the weak module, I implemented a WeakMapObj that works like we originally wanted WeakMap to work. It allows you to use a primitive for the key and an object for the data and the data is retained with a weak reference. And, it automatically removes items from the map when their data is GCed. It turned out to be fairly simple.
const weak = require('weak');
class WeakMapObj {
constructor(iterable) {
this._map = new Map();
if (iterable) {
for (let array of iterable) {
this.set(array[0], array[1]);
}
}
}
set(key, obj) {
if (typeof obj === "object") {
let ref = weak(obj, this.delete.bind(this, key));
this._map.set(key, ref);
} else {
// not an object, can just use regular method
this._map.set(key, obj);
}
}
// get the actual object reference, not just the proxy
get(key) {
let obj = this._map.get(key);
if (obj) {
return weak.get(obj);
} else {
return obj;
}
}
has(key) {
return this._map.has(key);
}
clear() {
return this._map.clear();
}
delete(key) {
return this._map.delete(key);
}
}
I was able to test it in a test app and confirm that it works as expected when the garbage collector runs. FYI, just making one or two objects eligible for garbage collection did not cause the garbage collector to run in my test app. I had to forcefully call the garbage collector to see the effect. I assume that would not be an issue in a real app. The GC will run when it needs to (which may only run when there's a reasonable amount of work to do).
You can use this more generic implementation as the core of your object cache where an item will stay in the WeakMapObj only until it is no longer referenced elsewhere.
Here's an implementation that keeps the map entirely private so it cannot be accessed from outside of the WeakMapObj methods.
const weak = require('weak');
function WeakMapObj(iterable) {
// private instance data
const map = new Map();
this.set = function(key, obj) {
if (typeof obj === "object") {
// replace obj with a weak reference
obj = weak(obj, this.delete.bind(this, key));
}
map.set(key, obj);
}
// add methods that have access to "private" map
this.get = function(key) {
let obj = map.get(key);
if (obj) {
obj = weak.get(obj);
}
return obj;
}
this.has = function(key) {
return map.has(key);
}
this.clear = function() {
return map.clear();
}
this.delete = function(key) {
return map.delete(key);
}
// constructor implementation
if (iterable) {
for (let array of iterable) {
this.set(array[0], array[1]);
}
}
}

Sounds like a job for a Map object used as a cache storing the object as the value (along with a count) and the ID as the key. When you want an object, you first look up its ID in the Map. If it's found there, you use the returned object (which will be shared by all). If it's not found there, you fetch it from the database and insert it into the Map (for others to find).
Then, to make it so that the Map doesn't grow forever, the code that fetches something from the Map would also need to release an object from the Map. When the useCnt goes to zero upon a release, you would remove an object from the Map.
This can be made entirely transparent to the caller by creating some sort of cache object that contains the Map and has methods for getting an object or releasing an object and it would be entirely responsible for maintaining the refCnt on each object in the Map.
Note: you will likely have to write the code that fetches it from the DB and inserts it into the Map carefully in order to not create a race condition because the fetching form the database is likely asynchronous and you could get multiple callers all not finding it in the Map and all in the process of getting it from the database. How to avoid that race condition depends upon the exact database you have and how you're using it. One possibility is for the first caller to insert a place holder in the Map so subsequent callers will know to wait for some promise to resolve before the object is inserted in the Map and available to them to use.
Here's a general idea for how such an ObjCache could work. You call cache.get(id) when you want to retrieve an item. This always returns a promise that resolves to the object (or rejects if there's an error getting it from the DB). If the object is in the cache already, the promise it returns will be already resolved. If the object is not in the cache yet, the promise will resolve when it has been fetched from the DB. This works even when multiple parts of your code request an object that is "in the process" of being fetched from the DB. They all get the same promise that is resolved with the same object when the object has been retrieved from the DB. Every call to cache.get(id) increases the refCnt for that object in the cache.
You then call cache.release(id) when a given piece of code is done with an object. That will decrement the internal refCnt and remove the object from the cache if the refCnt hits zero.
class ObjCache() {
constructor() {
this.cache = new Map();
}
get(id) {
let cacheItem = this.cache.get(id);
if (cacheItem) {
++cacheItem.refCnt;
if (cacheItem.obj) {
// already have the object
return Promise.resolve(cacheItem.obj);
}
else {
// object is pending, return the promise
return cacheItem.promise;
}
} else {
// not in the cache yet
let cacheItem = {refCnt: 1, promise: null, obj: null};
let p = myDB.get(id).then(function(obj) {
// replace placeholder promise with actual object
cacheItem.obj = obj;
cacheItem.promise = null;
return obj;
});
// set placeholder as promise for others to find
cacheItem.promise = p;
this.cache.set(id, cacheItem);
return p;
}
}
release(id) {
let cacheItem = this.cache.get(id);
if (cacheItem) {
if (--cacheItem.refCnt === 0) {
this.cache.delete(id);
}
}
}
}

Ok, for anyone who faces similar problems, I found a solution. jfriend00 pushed me towards this solution by mentioning WeakMaps which were not exactly the solution themselves, but pointed my focus on weak references.
There is an npm module simply called weak that will do the trick. It holds a weak reference to an object and safely returns an empty object once the object was garbage collected (thus, there is a way to identify a collected object).
So I created a class called WeakCache using a DataObject:
class DataObject{
constructor( objectID ){
this.objectID = objectID;
this.dataLoaded = new Promise(function(resolve, reject){
loadTheDataFromTheDatabase(function(data, error){ // some pseudo db call
if (error)
{
reject(error);
return;
}
resolve(data);
});
});
}
loadData(){
return this.dataLoaded;
}
}
class WeakCache{
constructor(){
this.cache = {};
}
getDataObjectAsync( objectID, onObjectReceived ){
if (this.cache[objectID] === undefined || this.cache[objectID].loadData === undefined){ // object was not cached yet or dereferenced, recreate it
this.cache[objectID] = weak(new DataObject( objectID )function(){
// Remove the reference from the cache when it got collected anyway
delete this.cache[this.objectID];
}.bind({cache:this, objectID:objectID});
}
this.cache[objectID].loadData().then(onObjectReceived);
}
}
This class is still in progress but at least this is a way how it could work. The only downside to this (but this is true for all database-based data, pun alert!, therefore not such a big deal), is that all data access has to be asynchronous.
What will happen here, is that the cache at some point may hold an empty reference to every possible object id.

Meteor - Meteor.Call inside helper not return value to helper [duplicate]

How can I define a Meteor method which is also callable in a template helper?
I have these two files:
file: lib/test.js
Meteor.methods({
viewTest : function (str) {
return str;
}
});
file: client/myView.js
Template.helloWorld.helpers({
txt : function () {
var str = Meteor.call('viewTest', 'Hello World.');
return str;
}
});
When I give "str" a normal string everything works fine. But in this case my template does not get any value. I defined - for the test - in the same file where the method is a normal function and tried to call the function. The error I got was that the function does not exist. So I think that Meteor tries to render the template before it knows anything about the methods I defined for it. But I think that this is a bit unusual - isn't it?

There is now a new way to do this (Meteor 0.9.3.1) which doesn't pollute the Session namespace
Template.helloWorld.helpers({
txt: function () {
return Template.instance().myAsyncValue.get();
}
});
Template.helloWorld.created = function (){
var self = this;
self.myAsyncValue = new ReactiveVar("Waiting for response from server...");
Meteor.call('getAsyncValue', function (err, asyncValue) {
if (err)
console.log(err);
else
self.myAsyncValue.set(asyncValue);
});
}
In the 'created' callback, you create a new instance of a ReactiveVariable (see docs) and attach it to the template instance.
You then call your method and when the callback fires, you attach the returned value to the reactive variable.
You can then set up your helper to return the value of the reactive variable (which is attached to the template instance now), and it will rerun when the method returns.
But note you'll have to add the reactive-var package for it to work
$ meteor add reactive-var

Sashko added a neat little package called meteor-reactive-method to solve this problem.
$ meteor add simple:reactive-method
Template.helloWorld.helpers({
txt: function() {
return ReactiveMethod.call('viewTest', 'Hello World.');
}
});
As I point out in common mistakes, helpers should be side-effect free, so I'd use this technique with caution. However, it's a really handy shortcut for cases where:
The helper should fire only once (it doesn't depend on reactive state).
The invoked method doesn't mutate the database.

You need to interface your return value with a Session variable as the request is asynchronous:
Template.helloWorld.helpers({
txt : function () {
return Session.get("txt") || "Loading";
}
});
Template.helloWorld.created = function() {
Meteor.call('viewTest', 'Hello World.', function(err, result) {
Session.set("txt", result);
});
}
So .rendered should be called once when your template loads (at least it should with the newer version of Meteor.)
The value would be called and displayed. Otherwise it would say "Loading".

Methods on the client side are asynchronous, and their return value is always undefined. To get the actual value returned by the method, you need to provide a callback:
Meteor.call('method', 'argument', function(error, result) {
....
});
Now, there's no easy way to use the result in your helper. However, you can store it in your template as a data object and then return it in the helper:
Template.template.created = function() {
var self = this;
self.data.elephantDep = new Deps.Dependency();
self.data.elephant = '';
Meteor.call('getElephant', 'greenOne', function(error, result) {
self.data.elephant = result;
self.data.elephantDep.changed();
});
};
Template.template.showElephant = function() {
this.elephantDep.depend();
return this.elephant;
};

This is expected behavior. You are not using methods as they are intended.
Your code defines a server method viewTest and a corresponding method stub on the client with the same name.
Meteor.call('viewTest', 'Hello World.'); remotely calls viewTest on the server and in parallel runs the stub on the client.
Regarding the return value of the stub please see the documentation here, in particular:
On the client, the return value of a stub is ignored. Stubs are run
for their side-effects: they are intended to simulate the result of
what the server's method will do, but without waiting for the round
trip delay.
Regarding the return value of the server method please see the documentation here, in particular:
On the client, if you do not pass a callback and you are not inside a
stub, call will return undefined, and you will have no way to get the
return value of the method. That is because the client doesn't have
fibers, so there is not actually any way it can block on the remote
execution of a method.

There is a fine little package for this by #msavin:
https://atmospherejs.com/msavin/fetcher

Factory returning same value

I am trying to get songs from soundcloud, I am using some input to set value and send it to my factory to get all the related list of songs and display it.
The issue is the the first time all works correctly, but when I am trying to input new values I am getting same results as first time.
My code looks like:
.controller('DashCtrl', function ($scope, SongsService) {
$scope.formData = {};
$scope.searchSong = function () {
SongsService.setData($scope.formData.songName);
};
UPDATE
the factory :
.factory('SongsService', function ($rootScope) {
var List = {};
List.setData = function (tracks) {
var page_size = 6;
SC.get('/tracks', {limit: page_size, linked_partitioning: 1}, function (tracks) {
// page through results, 100 at a time
List = tracks;
$rootScope.$broadcast('event:ItemsReceived');
});
};
List.getItems = function () {
return List;
};
return List;
}).value('version', '0.1');
Thanks for help!

It's hard to tell without a plunkr reproducing the issue and showing all your relevant code, but I think your problem is that you're overwriting the List variable in the async answer, and this List (I assume) is the object you originally returned from your factory.
I see two noteworthy concepts here:
the fact that angular factories are effectively singletons
and that javascript objects are passed by reference-by-value (see call-by-sharing, or one of many stackoverflow discussions).
An angular factory is a singleton, meaning the factory function will only be called once, before the first injection, and every controller it's injected into will work with the same object reference it returned. If you overwrite this object reference, well, the previous value (which the controller has) is still a reference to the original object.
Edit: In fact, by overwriting List you're creating a new object which doesn't even have a setData method anymore!
You probably want to make List private to SongsService, and return a more complex object from the factory that captures List in a closure, and offers some public getter/setter methods for it. (If you insist on replacing the contents of the returned List variable, empty the object and extend it with the new properties - including this method again. But this is much more work, and not a nice solution.)

In Angular Service constructors and Factory methods are singleton objects. You need to return a method that you can call. Your code examples are incomplete so it is hard to tell what is going on. What is returned by your factory method, the List object?
If so, when the first call is completed, it overwrites the List object so that the setData method can't be called a second time. What is the SC object, I can not see in your example how you are injecting it. You probably want to fix that too.
Consider this possible solution.
Service
Songs.$inject = ['$http'];
function Songs($http) {
this.$http = $http;
}
Songs.prototype.getSongs = function(searchTerm) {
return this.$http.get('http://someendpoint/songs', {searchTerm: searchTerm});
}
service('songs', Songs);
Controller
DashController.$inect = ['songs'];
functionDashController(songs) {
this.songs = songs;
this.results = [];
}
DashController.prototype.searchSongs = function(searchTerm) {
var self = this;
this.songs.getSongs(searchTerm).then(function(results) {
this.results = results;
});
}
controller('DashController', DashController);
This is example uses the best practice controllerAs syntax explained here: http://toddmotto.com/digging-into-angulars-controller-as-syntax/

I found the issue,
I got same results all the time because I didnt use cooreclty the api of soundcloud, I didnt send the title on the api... also you are correct, I should not set the list as empty..I should set some value to the list...

Why does it work correct with an array but not with an object?

I am developing an AngularJS application and found the following behavior.
I have two functions in my service. The first function returns all the categories stored in the database and the second returns one category by its id.
Here is my service:
angular.module('categoriesRepository', [])
.service('categoriesRepository', ['$cordovaSQLite', 'sqliteHelper',
function ($cordovaSQLite, sqliteHelper) {
//this works - returns an array with all categories
this.getAll = function () {
var categories = [];
$cordovaSQLite.execute(sqliteHelper.getDb(),
"SELECT * FROM categories;")
.then(function (res) {
for (var i = 0; i < res.rows.length; i++) {
categories.push(res.rows[i]);
}
});
return categories;
}
//this works not - returns undefined
this.getById = function (id) {
var category;
$cordovaSQLite.execute(sqliteHelper.getDb(),
"SELECT * FROM categories WHERE id = ?;", [id])
.then(function (res) {
category = res.rows[0];
});
return category;
}
}]);
I know that I can use Angulars $q to run functions asynchronously, and use their values when they are done processing.
Why does the getById function return the category directly and the getAll wait until the array is filled?
EDIT
I had the getAll function posted wrong. There is no return statement before $cordovaSQLite.execute

UPDATE:-
After your question is updated.
In the first example your are creating an array first by doing var categories = [];and then returning this array before finishing your async call. When your async call completes it just pushes certain elements into the array thus not destroying the reference to the array (categories ) variable. When it is returned back if you will debug it you will find the function returning an empty array and later when the async call succeeds only then the array will be filled.
In the second example you are creating just a variable and then returning it before the async call finishes. But then the async call is finished you assign the variable to a new value. thus destroying the earlier reference.
Solution:-
Though not a preffered approach to make it work. you will have to maintain the category variable reference. for this you can use angular.copy OR angular extend
So the second part of your code should be like
this.getById = function (id) {
var category;
$cordovaSQLite.execute(sqliteHelper.getDb(),
"SELECT * FROM categories WHERE id = ?;", [id])
.then(function (res) {
angular.copy(res.rows[0], category);
//now the reference to the category variable
//will not be lost
});
return category;
}
Better Practice:-
The way you have been developing this application is wrong. Async calls should not be handled this way. I earlier asked a question just to clarify the way to handle the async calls and state inside the angular app, factories and controllers please have a look here. It provides two ways to handle the state and async calls. There might be many more practices out there but these two suit me best.

It is unfortunate that this approach appears to 'work' because it is caused by the modification of the returned array object "at some unspecified time" after it is returned.
In the usage the array is accessed/observed after1 it has been modified by the asynchronous call. This makes it appear to function correctly only because of the (accidental) asynchronous-later-than observation.
If the observation was prior to the actual completion of the SQLite operation - such as immediately after the getAll function call - it would reveal an empty array.
Both functions are incorrectly written and the first accidently releases Zalgo (or perhaps his sibling).
See How do I return the response from an asynchronous call? for more details.
1 Chrome's console.log can be confusing as it works like console.dir and thus may be showing the current value and not the value when it was invoked.

As stated already, this is a bad approach. You can't use result of your function immediately after it returns.
However I didn't see the answer to your exact question: why do they behave differently?
It happens because with an array you return a reference to an object (type Array). Later on you use same reference to modify contents of the object, i.e. push new items into the array.
However in second function you modify the reference itself. You make you local variable categories point to a new object. Thus old object (reference to which was returned to outer scope) remains untouched. To make it work the same way you should have written
category.row = res.rows[0];

You return the result of the execute in the first case, whereas you return the variable in the second case, which has most likely not been populated yet.

module pattern setters that has methods

EDIT:
Everything is working as I expected. It was just an error calling the template method. I mistyped a () so I was trying template.method instead of template().method;
Anyway, if somebody would like to explain me if this is a valid design pattern or if I should go in a different way I will be definitively very grateful.
I read about the module pattern and I'm trying to implement it in some of my projects. The problem is that, in my opinion, I'm twisting it too much.
I'm inspired by the google apps script style where many objects returns other objects with methods and so on and they pass arguments.
something like
object.method(var).otherMethod();
What I want to achieve is a method that receives a parameter, sets an internal variable to that parameter and then returns an object with methods that uses that variable. Here is a minified version of the code that does not work:
var H_UI =(function (window) {
var selectedTemplate,
compileTemplate = function(){},
parseTemplateFields = function(){};
//template subModule. Collect: collects the template fields and returns a JSON representation.
var template = function(templateString){
if(templateString) selectedTemplate = templateString;
return {
getHtml:function(){ return compileTemplate( parseTemplateFields( selectedTemplate ) ) } ,
collect:function(){
.. operating over selectedTemplate ...
return JSON.stringify(result)}
} };
return {
template:template
};
})(window);
If I remove the line :
if(templateString) selectedTemplate = templateString;
and replace selectedTemplate with the parameter templateString in the methods of the returned object it works as expected. I know that I cant create a set() method in the returned object and use it like this
H_UI.template().set(var)
But I find it ugly. Anyway I think that I'm messing things up.
What is the best way to construct this?

If you want H_UI.template() creates a new object every time you call template() on it, your solution does not work. Because the variable selectedTemplate is created only once when the immediate function is called.
However if your intent is this your solution works fine. (variable selectedTemplate is shared for all calls to template()).
But if you want to every call to template creates a new object. Please tell me to write my idea

Is this a valid design pattern or if I should go in a different way
Yes, enabling chaining is definitely a valid design pattern.
However, if your template() method returns a new object, that object and its methods should only depend on itself (including the local variables and parameters of the template call), but not on anything else like the parent object that template was called on.
So either remove that "global" selectedTemplate thing:
var H_UI = (function () {
function compileTemplate(){}
function parseTemplateFields(){}
// make a template
function template(templateString) {
return {
getHtml: function(){
return compileTemplate(parseTemplateFields(templateString));
},
collect: function(){
// .. operating over templateString ...
return JSON.stringify(result)
}
}
}
return {template:template};
})();
or make only one module with with a global selectedTemplate, a setter for it, and global methods:
var H_UI = (function () {
var selectedTemplate;
function compileTemplate(){}
function parseTemplateFields(){}
return {
template: function(templateString){
if (templateString)
selectedTemplate = templateString;
return this; // for chaining
},
getHtml: function(){
return compileTemplate(parseTemplateFields(selectedTemplate));
},
collect: function(){
// .. operating over selectedTemplate ...
return JSON.stringify(result)}
}
};
})();
The difference is striking when we make two templates with that method:
var templ1 = H_UI.template("a"),
templ2 = H_UI.template("b");
What would you expect them to do? In a functional design, templ1 must not use "b". With the first snippet we have this, and templ1 != templ2. However, if .template() is a mere setter, and every call affects the whole instance (like in the second snippet), we have templ1 == H_UI and templ2 == H_UI.