I know that converting a buffer to a json object is trivial with:
JSON.parse(buffer)
But what about converting a buffer that contains a javascript object with functions to a real javascript object ??
I have files that contact objects with functions and I need to read these files in and save them to another master object.
While the task of reading the files is not a problem, I cannot find any way of converting the buffer back into a real javascript object again.
This is a very simple example of a file 'test.js' that I am trying to load
{
get:function(){return 'hello'},
somevar: "xxx",
put: function(){return 'world'}
}
Reading this data in it is a buffer, I can't convert using JSON as this contains functions and I cannot read using utf8 encoding as it will become a string !
var funcs = {}
fs.readFile('test.js',function(err,buff){
funcs['test'] = buff;
})
Is it possible to read a file and convert it into a real javascript object ?
Edit
OK, I have found a solution but it is using the dreaded eval(), however this is backend code and as far as I can tell there's no way for anything to be injected by a user from the frontend, I would prefer not to use it but unless there's anything that will work without modifying the format of the files, I will use it:
var funcs = {}
fs.readFile('test.js','utf8',function(err,buff){
eval('var_='+buff);
funcs['test'] = _;
})
For what it's worth, you could use the Function constructor. It's slightly safer than eval because it doesn't access the local scope. But it can still access global variables.
var script = buffer.toString('utf8'); // assuming the file is in UTF-8
var returnObject = new Function('return ' + script);
var myObject = returnObject();
Depending on the complexity of the code you're dealing with, this approach may or may not suit your needs:
You can create a temporary file, say "test-module.js", which includes the object from "test.js" but with a module export prependend, for example:
module.exports = {
get: function() { return 'hello' },
somevar: 'xxx',
put: function() { return 'world' }
}
Then, from your main file, you can retrieve the file content already available as a Javascript object:
var funcs = {}
var buff = require('./test-module');
funcs['test'] = [buff.get, buff.put];
Would this solution work for you?
Related
fs.readFile(), fs.readFileSync() and also Amazon S3 return a Buffer object. How can I use this Buffer as actual JavaScript code that I can execute?
Here is a very basic example.
some-file.js:
module.exports = function () {
console.log('hello');
};
main.js
const data1 = fs.readFileSync('./some-file.js'); // This is a Buffer
const data2 = fs.readFileSync('./some-file.js', 'utf-8'); // This is a string
data1(); // Error: data1 is not a function
JSON.parse(data2)(); // Error: data2 is no valid JSON
If you want to execute javascript code, instead of using eval, which can be quite a security risk, you should use built-in module vm
const vm = require('vm');
const sandbox = { x: 2 }; // pass variables or some context if you want
vm.createContext(sandbox); // Contextify the sandbox.
const code = buffer.toString(); // your buffer
vm.runInContext(code, sandbox);
It sounds like you are after something like eval for the function scenario (not the evil version, this one leverages the vm module). The other option here is to save the contents of the buffer to a file and then let require do it's thing (not sure if that's an option for you).
The object scenario is simpler, if Buffer is a JSON string then you should simply be able to call JSON.parse(buffer.toString('utf8'))
I use a JSON file for common phrases so I don't have to type them and maybe in the future they can be translated. So for example in my main code I want to say You don't have the permission to use ${command_name}. This works perfectly fine hardcoded into my .js file but ultimately I want this to be in a JSON file, which does not allow any variables to be inserted.
Does anyone know a solution to my problem?
EDIT: Thanks for the suggestions. I guess string.replace would be my best option here. Wish there was some built in feature that'd convert variables in a JSON string to variables declared in that JS file.
You cannot treat template string literals in JSON files like in Javascript "code". You said it yourself. But: You could use a template engine for this - or just simple String.replace().
Example for a template engine: https://github.com/janl/mustache.js
With Mustache (as an example) your code will look like this
var trans = {
command_name: "dump"
};
var output = Mustache.render("You don't have the permission to use {{command_name}}", trans);
With simple String.replace():
var str = "You don't have the permission to use %command_name%";
console.log(str.replace('%command_name%', 'dump'));
You can simply use placeholders. The following function replaces the placeholders with user-defined values:
const messages = {
msgName: 'Foo is :foo: and bar is :bar:!'
}
function _(key, placeholders) {
return messages[key].replace(/:(\w+):/g, function(__, item) {
return placeholders[item] || item;
});
}
Usage:
_('msgName', { foo: 'one', bar: 'two' })
// "Foo is one and bar is two!"
It's just an example. You can change the placeholders style and the function behavior the way you want!
You can use config npm module and separate your JSON files according to your environment.
./name.json
{
command: "this is the output of 'command'"
}
./Node.js
cost names = require('./name.json');
console.log('name === ', name.command);
// name === this is the output of 'command'
So the main challenge is getting separated file with string constants when some of them being parametrizable, right?
JSON format itself operates on strings(numbers, booleans, lists and hashmap) and knows nothing about substitution and parameters.
You are also unable to use template strings like you don't have permission to do ${actionName} since template strings are interpolated immediately.
So what can you do?
Writing your own parser that takes config data from JSON file, parse a string, find a reference to variable and substitute it with value. Simple example:
const varPattern = /\${([^{}]+)}/g;
function replaceVarWithValue(templateStr, params) {
return templateStr.replace(varPattern, (fullMatch, varName) => params[varName] || fullMatch);
}
or you can use any npm package aimed on localization like i18n so it would handle templates for you
Basically you can implement a function parse which, given a text and a dictionary, it could replace any ocurrence of each dictionary key:
const parse = (template, textMap) => {
let output = template
for (let [id, text] of Object.entries(textMap)) {
output = output.replace(new RegExp(`\\$\{${id}}`, 'mg'), text)
}
return output
}
const textMap = {
commandName: 'grep',
foo: 'hello',
bar: 'world'
}
const parsed = parse('command "${commandName}" said "${foo} ${bar}"', textMap)
console.log(parsed)
BTW, I would suggest you that you should use some existing string templating engine like string-template to avoid reinventing the wheel.
I need to load untrusted modules, written by third parties. I'm using vm for the sandbox and I was thinking to use threads (from npm: here) in order to load the module asynchronously and avoid blocking code.
I have stripped down the code to the minimum, because I'm stuck and I dont' understand if what I'm trying to achieve is impossible or it's just me messing with scopes.
Here is a dummy module:
exports.dummy = function () {
console.log('Dummy');
};
exports.val = 5;
And here is a module where I try to load this dummy module using threads:
var spawn = require('threads').spawn;
var mod;
var other;
var t = spawn(function (input, done) {
var m = require(input.dir + '/dummyMod');
done({m: m, other: 'hey'})
});
t.send({dir: __dirname}).on('message', function (result) {
mod = result.m;
other = result.other;
console.log(mod);
console.log(other);
t.kill();
});
The logged output is:
{ val: 5 }
hey
As you can see, the function in the dummy module has been skipped. If I try to load the module in the main process and log it, then the function is, of course, part of the object.
You need to properly serialize and deserialize the function. JSON.stringify ignores functions, probably because json is a format for storing data, not scripts.
Serialize the function by calling toString() on it. Then you can send it along as a string.
done({m: m.toString(), other: 'hey'})
Converting m to a string will give you something like this:
"function m(){console.log(\'called m()\')}"
On the receiving end, you will need to deserialize the function.
var m = new Function("return " + result.m)()
I have a some JavaScript with a complex structure. Because I'm new comer to JavaScript (only understanding some basic concepts) I don't know how to use it properly.
I have two files : Circle.js and Line.js. In Circle.js, I want to use a class object defined in Line.js:
In file Circle.js :
Helper.using('py.Figures', function (ns) {
ns.Circle = function (params) {
// some additional methods and code here
}
}
And in Line.js is :
Helper.using('py.Figures', function (ns) {
ns.Line2Point = function (params) {
// some addition methods and code here
};
}
In Figures.Circle, in ns.Circle I want to use Line2Point but I don't know how.
I think it should be :
line = new ns.Line2Point(params);
But It seem doesn't work.
According to Helper Class, ns will point to helper.using, in this case py.Figures. Does it mean, ns is the same object/reference in both the files?
I don't think this is doable in Javascript directly across files. If they are part of the same namespace you could share some 'global' objects to achieve this have the line2points and circles attach themselves to that global object:
Ex:
var myShapesNameSpace = {};
Circle.js:
(function(){
var circle = {};
circle.method1 = function(){...}
circle.method2 = function(){...}
myShapesNameSpace.Circles = circle;
})(window.myShapesNameSpace = window.myShapesNameSpace || {}); //check if namespace object exists. Else create a new blank one.
Line.js:
(function(){
var line= {};
line.method1 = function(){...}
line.method2 = function(){...}
myShapesNameSpace.Lines= line;
})(window.myShapesNameSpace = window.myShapesNameSpace || {});
Now you can check for the existence of myShapesNameSpace.Circles or .Lines and call the corresponding methods accordingly.
You can include files in javascript and reference objects across files unless they are exported in some global form either via window or your define global
Welcome to Javascript, the shit parts. Require.js was designed precisely for this because the creators of JS, well, I guess thought that everyone would write every program in one file.
RequireJS
It was designed for web use but can be used elsewhere too (locally, with Node, etc.)
I have a server side Python script that returns a JSON string containing parameters for a client side JavaScript.
# Python
import simplejson as json
def server_script()
params = {'formatting_function': 'foobarfun'}
return json.dumps(params)
This foobarfun should refer to a JavaScript function. Here is my main client side script
// JavaScript
function client_script() {
var xhr = new XMLHttpRequest();
xhr.open("GET", url, async=true);
xhr.onreadystatechange = function() {
if (xhr.readyState == 4) {
options = JSON.parse(xhr.responseText);
options.formatting_function();
}
};
xhr.send(null);
}
function foobarfun() {
//do_something_funny_here...
}
Of course, options.formatting_function() will complain that "a string is not callable" or something to that effect.
Upon using Chrome's Inspect Element, under the Resources tab, and navigating the left sidebar for XHR > query, I find that client_script interprets options as below. foobarfun is seen as a string.
// JavaScript
options = {"formatting_function": "foobarfun"}
I would have liked client_script to see options as
// JavaScript
options = {"formatting function": foobarfun}
Of course, doing the following within Python will have it complaining that it doesn't know anything about foobarfun
# Python
params = {'formatting_function': foobarfun}
QUESTION:
How should I prepare my JSON string from the server side so that the client script can interpret it correctly? In this case, I want foobarfun to be interpreted as a function object, not as a string.
Or maybe it's something I should do on the client side?
There's nothing you can do in the JSON to get the result you want because JSON has no concept of functions, it's purely a data notation. But there are things you can do client-side.
If your foobarfun function is a global function (which I would recommend against, we'll come to that), then you can call it like this:
window[options.formatting_function]();
That works because global functions are properties of the window object, and you can access properties either by using dotted notation and literals (window.foobarfun), or by using bracketed notation and strings (window["foobarfun"]). In the latter case, of course, the string doesn't have to be a string literal, it can be a string from a property -- your options.formatting_function property, for instance.
But I don't recommend using global functions, the window object is already very crowded. Instead, I keep all of my functions (or as many as possible, in some edge cases) within a master scoping function so I don't add anything to the global namespace:
(function() {
function foobarfun() {
}
})();
Now, if you do that, you can't access foobarfun on window because the whole point of doing it is to avoid having it be on window. Instead, you can create your own object and make it a property of that:
(function() {
var myStuff = {};
myStuff.foobarfun = foobarfun;
function foobarfun() {
}
function client_script() {
var xhr = new XMLHttpRequest();
xhr.open("GET", url, async=true);
xhr.onreadystatechange = function() {
if (xhr.readyState == 4) {
options = JSON.parse(xhr.responseText);
myStuff[options.formatting_function](); // <== using it
}
};
xhr.send(null);
}
})();
Frequently, rather than this:
myStuff.foobarfun = foobarfun;
function foobarfun() {
}
you'll see people write:
myStuff.foobarfun = function() {
};
I don't recommend that, either, because then your function is anonymous (the property on myStuff that refers to the function has a name, but the function doesn't). Giving your functions names is a good thing, it helps your tools help you (showing you the names in call stacks, error messages, etc.).
You might also see:
myStuff.foobarfun = function foobarfun() {
};
and that should be valid, it's correct JavaScript. But unfortunately, various JavaScript implementations have various bugs around that (which is called a named function expression), most especially Internet Explorer prior to IE9, which will create two completely different functions at two different times.
All of that said, passing the names of functions around between the client and server usually suggests that you want to step back and look at the design again. Data should drive logic, but not in quite such a literal way. That said, though, there are definitely valid use cases for doing this, you may well have one in your situation.
This question seems like it may be helpful for you:
How to execute a JavaScript function when I have its name as a string
I think what I would do is store references to these methods in an object literal, and then access them through properties.
For example, if I wanted to call foobarfun, among other functions
var my_functions = {
foobarfun: function(){
},
...
};
...
var my_fn = my_functions[options.formatting_function];
my_fn();
may be you can think the return string as javascript not json, by set MIME type text/javascript
There is a trick you can do. Analyzing the code of the json module in python, I notice that it is possible to make the serializer believe that it is an int what is serializing that way __str__ method will be executed.
import json
class RawJS(int):
def __init__(self):
self.code = "null"
def __repr__(self):
return self.code
def __str__(self):
return self.__repr__()
#classmethod
def create(cls, code):
o = RawJS()
o.code = code
return o
js = RawJS.create("""function() {
alert("hello world!");
}""")
x = {
"func": js,
"other": 10
}
print json.dumps(x)
the output is:
{"other": 10, "func": function() {
alert("hello world!");
}}
The disadvantage of this method is that the output is not a valid JSON in python so it can't be deserialized, but it is a valid javascript.
I don't know whether it's possible to trick Python's JSON serializer to do what you want, but you can use eval() function to execute any JavaScript code stored in a string variable.
Well I am not a Python expert but I know some java script. From server you can pass the information to client only as string. If in any case you want to pass any java script function then you can pass that function name as string and evaluate that string on client side.
Ex. If you passes xyz as string from server and on client side you call
var funcName = "xyz"; // pursuing that variable would pass here some how
eval (funcName); // This line would make a call to java script function xyz
Hint: You can think of java script eval utility as a reflection utility in java.
Thanks
shaILU
I'm in the same situation (python backend, needing to pass a JS function through JSON to frontend).
And while the answers here are honestly mind-blowing, I'm asking myself whether passing a javascript function that needs to designed in python and snuck into a JSON is the right answer, from a design perspective.
My specific scenario I need to tell JS how to format a string (is it a percentage, or does it need unitary breakdown such as thousands, millions etc)
And I'm finding it's cleaner to just do:
python:
chart = {
y_axis: {
"formatter" : "percent"
}
}
JS:
format_type = chart["y_axis"]["formatter"]
switch(format_type) {
case "percent":
format_func = (value) => value.toFixed(0) +'%';
break;
}
chart["y_axis"]["formatter"] = format_func
I find this is overall cleaner than attempting to define your JS functions in Python. It's also more decoupled than passing a specific function name from python to JS
Which I guess it's quite similar to #jkeesh's solution