Develop Reference

Monaco - Code Completion for CommonJS Modules without LSP

I'm integrating the Monaco editor into Eclipse Dirigible Web IDE.
This is how the editor is integrated as of now: ide-monaco/editor.html
In Dirigible we are using server-side JavaScript, based on Mozila Rhino, Nashorn, J2V8 or GraalVM (not NodeJS) as a target programming language.
To achieve modularization, we are loading the modules through require(...moduleName..) according to the CommonJS specification.
Here is an example of such module (API) that we have:
http/v4/response
Here is a sample usage of this API:
https://www.dirigible.io/api/http_response.html
Now going back to the Monaco topic, I'm trying to achieve code completion for the loaded modules e.g.:
var response = require("http/v4/response");
...
I found a sample on how to provide an external library:
monaco.languages.typescript.javascriptDefaults.addExtraLib('var response = {println: /** Prints the text in the response */ function(text) {}}', 'js:response.js');
Dirigible Monaco Code Completion with Extra Lib
But once var response is declared, it shadows the code completion options:
Dirigible Monaco Shadowed Code Completion Options
I found that there are several Monaco CompilerOptions available:
sourceRoot
module
moduleResolution
baseUrl
paths
rootDir
...
but I couldn't get the code completion of external modules work.
Is there a way to set some kind of "source provider" to the Monaco editor, so once the require(...) statement is found, then it triggers to load of this module and eventually will get the code completion working? We've managed to implement such approach approach for Orion and tern.js: ide-orion/editorBuild/commonjs-simplified

Reference implementation based on Acorn.js can be found here:
Server Side Integration:
Server side module for retrieving the JavaScript modules in the System and to provide code completion per imported module: dirigible/ide-monaco-extensions
Acorn.js Dirigible module: acorn/acornjs
Suggestions parser:
exports.parse = function(moduleName) {
var content = contentManager.getText(moduleName + ".js");
var comments = [];
var nodes = acorn.parse(content, {
onComment: comments,
ranges: true
});
var functionDeclarations = nodes.body
.filter(e => e.type === "FunctionDeclaration")
.map(function(element) {
let name = element.id.name;
let expression = element.expression
let functions = element.body.body
.filter(e => e.type === "ExpressionStatement")
.map(e => extractExpression(e, comments))
.filter(e => e !== null);
return {
name: name,
functions: functions
}
});
var result = nodes.body
.filter(e => e.type === "ExpressionStatement")
.map(function(element) {
return extractExpression(element, comments, functionDeclarations);
}).filter(e => e !== null);
return result;
}
function extractExpression(element, comments, functionDeclarations) {
let expression = element.expression;
if (expression && expression.type === "AssignmentExpression" && expression.operator === "=") {
let left = expression.left;
let right = expression.right;
if (right.type === "FunctionExpression") {
let properties = right.params.map(e => e.name);
let name = left.property.name + "(" + properties.join(", ") + ")";
let documentation = extractDocumentation(comments, element, name);
documentation = formatDocumentation(documentation, name, true);
let returnStatement = right.body.body.filter(e => e.type === "ReturnStatement")[0];
let returnType = null;
if (functionDeclarations && returnStatement && returnStatement.argument.type === "NewExpression") {
returnType = returnStatement.argument.callee.name;
returnType = functionDeclarations.filter(e => e.name === returnType)[0];
}
return {
name: name,
documentation: documentation,
returnType: returnType,
isFunction: true
};
} else if (right.type === "Literal") {
let name = left.property.name;
let documentation = extractDocumentation(comments, element, name);
documentation = formatDocumentation(documentation, name, false);
return {
name: name,
documentation: documentation,
isProperty: true
};
}
}
return null;
}
...
The complete file can be found here: suggestionsParser.js
Client Side Integration
Retrieve JavaScript modules available in the System: editor.html#L204-L212
Get suggestion per module: editor.html#L215-L225
Add extra lib for require(...): editor.html#L301
Register completion item provider for modules: editor.html#L302-L329
Register completion item provider for suggestions (functions/properties): editor.html#L330-L367

How can I register and use a Handlebars helper with Node?

I'm using Handlebars with Node, and that works fine:
require('handlebars');
var template = require('./templates/test-template.handlebars');
var markup = template({ 'some': 'data' });
console.log(markup);
That works fine. However, I need to register and use a custom helper in my template. So, now my code looks like this:
var Handlebars = require('handlebars');
Handlebars.registerHelper('ifEqual', function(attribute, value) {
if (attribute == value) {
return options.fn(this);
}
else {
return options.inverse(this);
}
});
var template = require('./templates/test-template.handlebars');
var markup = template({ 'some': 'data' });
console.log(markup);
But now when I run my script, I get
Error: Missing helper: 'ifEqual'
So: how can I define and use a custom helper in Node?

I figured it out. I needed to do this:
var Handlebars = require('handlebars/runtime')['default'];
One what's really cool is, this even works in the browser with Browserify.
However, I found that an even better way (and probably the "correct" way) is to precompile Handlebars templates via (shell command):
handlebars ./templates/ -c handlebars -f templates.js
Then I do this:
var Handlebars = require('handlebars');
require('./templates');
require('./helpers/logic');
module.exports.something = function() {
...
template = Handlebars.templates['template_name_here'];
...
};

Here is the way i did it.
I guess nowadays its a bit different.
const Handlebars = require('handlebars');
module.exports = function(){
Handlebars.registerHelper('stringify', function(stuff) {
return JSON.stringify(stuff);
});
};
then i made a little script to call require on all the helpers just so they get ran.
// Helpers Builder
let helpersPath = Path.join(__dirname, 'helpers');
fs.readdir(helpersPath, (err, files) => {
if (err) {throw err;}
files.filter((f) => {
return !!~f.indexOf('.js');
}).forEach((jsf) => {
require(Path.join(helpersPath, jsf))();
});
});
OR the simple way
require('./helpers/stringify')();
in fact you dont even have to export it as a function you can just not export anything at all and just call require from another js file with out the function params at the end.

Compiling an Ember template with specified values

I need to get a template from Ember.TEMPLATES, compile it with a specified object and get its raw HTML value.
Ember.TEMPLATES content (generated using gruntjs) returns a function and seems to be already passed through Handlebars.template() function so for example I would have this:
Ember.TEMPLATES["test"] = Ember.Handlebars.template(function anonymous(Handlebars,depth0,helpers,partials,data) {
this.compilerInfo = [4,'>= 1.0.0'];
helpers = this.merge(helpers, Ember.Handlebars.helpers); data = data || {};
var buffer = '', hashTypes, hashContexts, escapeExpression=this.escapeExpression;
data.buffer.push("<strong>hello world ");
hashTypes = {};
hashContexts = {};
data.buffer.push(escapeExpression(helpers._triageMustache.call(depth0, "test", {hash:{},contexts:[depth0],types:["ID"],hashContexts:hashContexts,hashTypes:hashTypes,data:data})));
data.buffer.push("</strong>\n");
return buffer;
});
and would like to compile that template with new values from a JSON object.
I tried something like that based on what I've seen in Ember code:
var test = Ember.TEMPLATES['test'];
var compiled = test({ test: 'value' });
I thought it might work but it doesn't actually.
Basically I'd like to do like with standard handlebars :
Handlebars.compile('<strong>{{hello}}</strong>', { hello: 'world' });
Is there any way to compile a template with specified values, and get the HTML result using Emberjs?

Ember do some modifications in handlebars compiler to enable the use of computed properties, make templates update when model changes etc.
If you see the view render method, it does more than template(context), it use the context and some private custom data. So Handlebars.compile is diferent of Ember.Handlebars.compile and I think that compiled templates from Ember.Handlebars.compile, is not intended to be used outside of a Ember.View.
Marking script types with text/x-raw-handlebars, instead of text/x-handlebars make the template be compiled with Handlebars.compile.
The following sample will work, but without the ember features:
Template
<script type="text/x-raw-handlebars" data-template-name="custom-template">
First name: {{firstName}} <br/>Last name: {{lastName}}
</script>
Javascript
App = Ember.Application.create({
ready: function() {
var template = Ember.TEMPLATES['custom-template'];
var html = template({ firstName: 'Tom', lastName: 'Dale' });
$('body').append(html);
}
});
You can see this sample here http://jsfiddle.net/marciojunior/MC8QB/

How do I load different partials dynamically using handlebars templates?

I'm loading a template with the following data:
"slides": [
{
"template": "video",
"data": {
"video": ""
}
},
{
"template": "image",
"data": {
"image": ""
}
}
]
in my template I want to loop over these slides and based on the configured template I want to load a partial
{{#each slides}}
{{> resources_templates_overlay_video }}
{{/each}}
How can I make this partial load dynamically (based on the configured template)?
I'm using the require-handlebars-plugin

As far as I can tell, hbs expects the partials to be known at compile time, which is way before you pass in your data. Let's work around that.
First, pull in your dynamic partials before rendering, something like:
// I required the main template here for simplicity, but it can be anywhere
var templates = ['hbs!resources/templates/maintemplate'], l = data.slides.length;
for (var i=0; i<l; i++ )
templates.push('hbs!resources/templates/overlay/'+data[i].template);
require(templates, function(template) {
var html = template(data);
});
And define a helper that will act as a dynamic partial
define(['Handlebars'], function (Handlebars) {
function dynamictemplate(template, context, opts) {
template = template.replace(/\//g, '_');
var f = Handlebars.partials[template];
if (!f) {
return "Partial not loaded";
}
return new Handlebars.SafeString(f(context));
}
Handlebars.registerHelper('dynamictemplate', dynamictemplate);
return dynamictemplate;
});
Finally, modify your main template to look like
{{#each slides}}
{{dynamictemplate this.template this.data}}
{{/each}}

I found the above answers a little hard to understand - they leak globals, have single character variables, and some odd naming. So here's my own answer, for my (and your) reference:
A dynamic partial using 'hbs', express.js default handlebars implementation:
I used this to make a simple blog making (article-name).md into /blog/(article-name), creating a dynamic partial:
// Create handlebars partials for each blog item
fs.readdirSync('blog').forEach(function(blogItem){
var slug = blogItem.replace('.md','')
var fileContents = fs.readFileSync('blog/'+blogItem, 'utf8')
var html = marked(fileContents)
var compiledTemplate = hbs.compile(html);
hbs.registerPartial(slug, compiledTemplate);
})
// Create 'showBlogItem' helper that acts as a dynamic partial
hbs.registerHelper('showBlogItem', function(slug, context, opts) {
var loadedPartial = hbs.handlebars.partials[slug];
return new hbs.handlebars.SafeString(loadedPartial(context));
});
Here's the route. It 404s if the partial doesn't exist, because the blog doesn't exist.
router.get('/blog/:slug', function(req, res){
var slug = req.param("slug")
var loadedPartial = hbs.handlebars.partials[slug];
if ( ! loadedPartial ) {
return res.status(404).json({ error: 'Article not found' })
}
res.render('blog', {
slug: slug
});
})
/views/blog.hbs looks like:
<div class="blog">
{{ showBlogItem slug }}
</div>

When Handlebars.partials[] returns a raw string it means the partial is not compiled.
I am not sure but my best guess is that Handlebars compiles the partial internally when it compiles the template which includes the partial. So when you use a helper to include a partial then Handlebars doesn't recognize it and it will not be compiled.
You can compile the partial yourself. Don't forget to register the compiled partial or you end up compiling every time the partial is required, which hurts performance. Something like this should work.
var template = Handlebars.partials['templatename'],
fnTemplate = null;
if (typeof template === 'function') {
fnTemplate = template;
} else {
// Compile the partial
fnTemplate = Handlebars.compile(partial);
// Register the compiled partial
Handlebars.registerPartial('templatename', fnTemplate);
}
return fnTemplate(context);

Best way to serialize/unserialize objects in JavaScript?

I have many JavaScript objects in my application, something like:
function Person(age) {
this.age = age;
this.isOld = function (){
return this.age > 60;
}
}
// before serialize, ok
var p1 = new Person(77);
alert("Is old: " + p1.isOld());
// after, got error Object #<Object> has no method 'isOld'
var serialize = JSON.stringify(p1);
var _p1 = JSON.parse(serialize);
alert("Is old: " + _p1.isOld());
See in JS Fiddle.
My question is: is there a best practice/pattern/tip to recover my object in same type it was before serialization (instances of class Person, in this case)?
Requirements that I have:
Optimize disk usage: I have a big tree of objects in memory. So, I don't want to store functions.
Solution can use jQuery and another library to serialize/unserialize.

JSON has no functions as data types. You can only serialize strings, numbers, objects, arrays, and booleans (and null)
You could create your own toJson method, only passing the data that really has to be serialized:
Person.prototype.toJson = function() {
return JSON.stringify({age: this.age});
};
Similar for deserializing:
Person.fromJson = function(json) {
var data = JSON.parse(json); // Parsing the json string.
return new Person(data.age);
};
The usage would be:
var serialize = p1.toJson();
var _p1 = Person.fromJson(serialize);
alert("Is old: " + _p1.isOld());
To reduce the amount of work, you could consider to store all the data that needs to be serialized in a special "data" property for each Person instance. For example:
function Person(age) {
this.data = {
age: age
};
this.isOld = function (){
return this.data.age > 60 ? true : false;
}
}
then serializing and deserializing is merely calling JSON.stringify(this.data) and setting the data of an instance would be instance.data = JSON.parse(json).
This would keep the toJson and fromJson methods simple but you'd have to adjust your other functions.
Side note:
You should add the isOld method to the prototype of the function:
Person.prototype.isOld = function() {}
Otherwise, every instance has it's own instance of that function which also increases memory.

I wrote serialijse because I faced the same problem as you.
you can find it at https://github.com/erossignon/serialijse
It can be used in nodejs or in a browser and can serve to serialize and deserialize a complex set of objects from one context (nodejs) to the other (browser) or vice-versa.
var s = require("serialijse");
var assert = require("assert");
// testing serialization of a simple javascript object with date
function testing_javascript_serialization_object_with_date() {
var o = {
date: new Date(),
name: "foo"
};
console.log(o.name, o.date.toISOString());
// JSON will fail as JSON doesn't preserve dates
try {
var jstr = JSON.stringify(o);
var jo = JSON.parse(jstr);
console.log(jo.name, jo.date.toISOString());
} catch (err) {
console.log(" JSON has failed to preserve Date during stringify/parse ");
console.log(" and has generated the following error message", err.message);
}
console.log("");
var str = s.serialize(o);
var so = s.deserialize(str);
console.log(" However Serialijse knows how to preserve date during serialization/deserialization :");
console.log(so.name, so.date.toISOString());
console.log("");
}
testing_javascript_serialization_object_with_date();
// serializing a instance of a class
function testing_javascript_serialization_instance_of_a_class() {
function Person() {
this.firstName = "Joe";
this.lastName = "Doe";
this.age = 42;
}
Person.prototype.fullName = function () {
return this.firstName + " " + this.lastName;
};
// testing serialization using JSON.stringify/JSON.parse
var o = new Person();
console.log(o.fullName(), " age=", o.age);
try {
var jstr = JSON.stringify(o);
var jo = JSON.parse(jstr);
console.log(jo.fullName(), " age=", jo.age);
} catch (err) {
console.log(" JSON has failed to preserve the object class ");
console.log(" and has generated the following error message", err.message);
}
console.log("");
// now testing serialization using serialijse serialize/deserialize
s.declarePersistable(Person);
var str = s.serialize(o);
var so = s.deserialize(str);
console.log(" However Serialijse knows how to preserve object classes serialization/deserialization :");
console.log(so.fullName(), " age=", so.age);
}
testing_javascript_serialization_instance_of_a_class();
// serializing an object with cyclic dependencies
function testing_javascript_serialization_objects_with_cyclic_dependencies() {
var Mary = { name: "Mary", friends: [] };
var Bob = { name: "Bob", friends: [] };
Mary.friends.push(Bob);
Bob.friends.push(Mary);
var group = [ Mary, Bob];
console.log(group);
// testing serialization using JSON.stringify/JSON.parse
try {
var jstr = JSON.stringify(group);
var jo = JSON.parse(jstr);
console.log(jo);
} catch (err) {
console.log(" JSON has failed to manage object with cyclic deps");
console.log(" and has generated the following error message", err.message);
}
// now testing serialization using serialijse serialize/deserialize
var str = s.serialize(group);
var so = s.deserialize(str);
console.log(" However Serialijse knows to manage object with cyclic deps !");
console.log(so);
assert(so[0].friends[0] == so[1]); // Mary's friend is Bob
}
testing_javascript_serialization_objects_with_cyclic_dependencies();

I am the author of https://github.com/joonhocho/seri.
Seri is JSON + custom (nested) class support.
You simply need to provide toJSON and fromJSON to serialize and deserialize any class instances.
Here's an example with nested class objects:
import seri from 'seri';
class Item {
static fromJSON = (name) => new Item(name)
constructor(name) {
this.name = name;
}
toJSON() {
return this.name;
}
}
class Bag {
static fromJSON = (itemsJson) => new Bag(seri.parse(itemsJson))
constructor(items) {
this.items = items;
}
toJSON() {
return seri.stringify(this.items);
}
}
// register classes
seri.addClass(Item);
seri.addClass(Bag);
const bag = new Bag([
new Item('apple'),
new Item('orange'),
]);
const bagClone = seri.parse(seri.stringify(bag));
// validate
bagClone instanceof Bag;
bagClone.items[0] instanceof Item;
bagClone.items[0].name === 'apple';
bagClone.items[1] instanceof Item;
bagClone.items[1].name === 'orange';
Hope it helps address your problem.

The browser's native JSON API may not give you back your idOld function after you call JSON.stringify, however, if can stringify your JSON yourself (maybe use Crockford's json2.js instead of browser's API), then if you have a string of JSON e.g.
var person_json = "{ \"age:\" : 20, \"isOld:\": false, isOld: function() { return this.age > 60; } }";
then you can call
eval("(" + person + ")")
, and you will get back your function in the json object.

I had the exact same problem, and written a small tool to do the mixing of data and model. See https://github.com/khayll/jsmix
This is how you would do it:
//model object (or whatever you'd like the implementation to be)
var Person = function() {}
Person.prototype.isOld = function() {
return this.age > RETIREMENT_AGE;
}
//then you could say:
var result = JSMix(jsonData).withObject(Person.prototype, "persons").build();
//and use
console.log(result.persons[3].isOld());
It can handle complex objects, like nested collections recursively as well.
As for serializing JS functions, I wouldn't do such thing because of security reasons.

I've added yet another JavaScript serializer repo to GitHub.
Rather than take the approach of serializing and deserializing JavaScript objects to an internal format the approach here is to serialize JavaScript objects out to native JavaScript. This has the advantage that the format is totally agnostic from the serializer, and the object can be recreated simply by calling eval().
https://github.com/iconico/JavaScript-Serializer

I had a similar problem and since I couldn't find a sufficient solution, I also created a serialization library for javascript: https://github.com/wavesoft/jbb (as a matter of fact it's a bit more, since it's mainly intended for bundling resources)
It is close to Binary-JSON but it adds a couple of additional features, such as metadata for the objects being encoded and some extra optimizations like data de-duplication, cross-referencing to other bundles and structure-level compression.
However there is a catch: In order to keep the bundle size small there are no type information in the bundle. Such information are provided in a separate "profile" that describes your objects for encoding and decoding. For optimization reasons this information is given in a form of script.
But you can make your life easier using the gulp-jbb-profile (https://github.com/wavesoft/gulp-jbb-profile) utility for generating the encodeing/decoding scripts from simple YAML object specifications like this:
# The 'Person' object has the 'age' and 'isOld'
# properties
Person:
properties:
- age
- isOld
For example you can have a look on the jbb-profile-three profile.
When you have your profile ready, you can use JBB like this:
var JBBEncoder = require('jbb/encode');
var MyEncodeProfile = require('profile/profile-encode');
// Create a new bundle
var bundle = new JBBEncoder( 'path/to/bundle.jbb' );
// Add one or more profile(s) in order for JBB
// to understand your custom objects
bundle.addProfile(MyEncodeProfile);
// Encode your object(s) - They can be any valid
// javascript object, or objects described in
// the profiles you added previously.
var p1 = new Person(77);
bundle.encode( p1, 'person' );
var people = [
new Person(45),
new Person(77),
...
];
bundle.encode( people, 'people' );
// Close the bundle when you are done
bundle.close();
And you can read it back like this:
var JBBDecoder = require('jbb/decode');
var MyDecodeProfile = require('profile/profile-decode');
// Instantiate a new binary decoder
var binaryLoader = new JBBDecoder( 'path/to/bundle' );
// Add your decoding profile
binaryLoader.addProfile( MyDecodeProfile );
// Add one or more bundles to load
binaryLoader.add( 'bundle.jbb' );
// Load and callback when ready
binaryLoader.load(function( error, database ) {
// Your objects are in the database
// and ready to use!
var people = database['people'];
});

You can create an empty instance of of your class and assign values to it with Object.assign.
let p1 = new Person(77);
let serialized = JSON.stringify(p1);
let deserialized = Object.assign(new Person(), JSON.parse(serialized))

I tried to do this with Date with native JSON...
function stringify (obj: any) {
return JSON.stringify(
obj,
function (k, v) {
if (this[k] instanceof Date) {
return ['$date', +this[k]]
}
return v
}
)
}
function clone<T> (obj: T): T {
return JSON.parse(
stringify(obj),
(_, v) => (Array.isArray(v) && v[0] === '$date') ? new Date(v[1]) : v
)
}
What does this say? It says
There needs to be a unique identifier, better than $date, if you want it more secure.
class Klass {
static fromRepr (repr: string): Klass {
return new Klass(...)
}
static guid = '__Klass__'
__repr__ (): string {
return '...'
}
}
This is a serializable Klass, with
function serialize (obj: any) {
return JSON.stringify(
obj,
function (k, v) { return this[k] instanceof Klass ? [Klass.guid, this[k].__repr__()] : v }
)
}
function deserialize (repr: string) {
return JSON.parse(
repr,
(_, v) => (Array.isArray(v) && v[0] === Klass.guid) ? Klass.fromRepr(v[1]) : v
)
}
I tried to do it with Mongo-style Object ({ $date }) as well, but it failed in JSON.parse. Supplying k doesn't matter anymore...
BTW, if you don't care about libraries, you can use yaml.dump / yaml.load from js-yaml. Just make sure you do it the dangerous way.

I've made an npm module named esserializer to solve this problem: save JavaScript class object values during serialization, in plain JSON format, without storing any functions. During the serialization, the only overhead incurred is saving the class names information. Thus, the disk usage is optimized.
Later on during the deserialization stage, esserializer can recursively deserialize object instance, with all types/functions information retained. It works in both browser and Node.js environment.
In the OP's case, the code would be pretty easy:
var ESSerializer = require('esserializer');
function Person(age) {
this.age = age;
this.isOld = function (){
return this.age > 60;
}
}
// before serialize, ok
var p1 = new Person(77);
alert("Is old: " + p1.isOld());
// serialize
var serializedText = ESSerializer.serialize(p1);
//...do something, or send the above serializedText to another JavaScript environment.
// deserialize
var deserializedObj = ESSerializer.deserialize(serializedText, [Person]);
alert("Is old: " + deserializedObj.isOld());
the deserializedObj is a Person instance, which contains all values/functions/superclasses information.
Wish it could help.