I'm trying to wrap my head around Node.js streams, not that I'm pretty new to JavaScript and node, the last languages I really got were Perl and PHP :-D
I've read the Buffer/Streams documentation # nodejs.org, watched James Halliday # LXJS, read his stream-handbook and Thorsten Lorenz event-stream post. I start to understand the basics :)
I process data which is serialized in RDF (which is neither JSON nor XML). I manage to fetch the data (in real code via request) and parse it into a JS object using rdfstore module.
So far I do this:
s.createReadStream('myRDFdata.ttl').pipe(serialize()).pipe(process.stdout);
Where serialize()does the job of parsing an serializing the code at the same time right now. I use through module to interface to the stream.
Now I have some more methods (not the real function declaration but I hope you get the point):
getRecipe(parsedRDF) -> takes the parsed RDF (as a JavaScript object) and tells me how to use it
createMeal(parsedRDF, recipe) -> takes the parsed RDF and the recipe from above and creates a new RDF object out of it
this new object needs to get serialized and sent to the browser
(In the real world getRecipe will have to do a user interaction in the browser)
I like the idea of chaining this together via pipes for higher flexibility when I enhance the code later. But I don't want to serialize it to a RDF serialization every time but just send around the JS object. From what I've read in the documentation I could use the stringify module to get a string out of each step for piping it to the next step. But:
does this actually make sense? In terms of do I add unnecessary overhead or is this negligible?
I don't see how I could give the parsedRDF to both methods with the dependency that getRecipe would have to be called first and the output is input for createMeal as well. Are there modules which help me on that?
It might be that I have to ask the user for the final recipe selection so I might need to send stuff to the browser there to get the final answer. Can I do something like this over sockets while the pipe is "waiting"?
I hope this shows what I'm trying to do, if not I will try to give more details/rephrase.
Update: After sleeping over it I figured out some more things:
It probably doesn't make sense to serialize a format like RDF into something non-standard if there are official serialization formats. So instead of using stringify I will simply pass an official RDF serialization between the steps
This does imply that I parse/serialize the objects in each step and this surely does add overhead. Question is do I care? I could extend the RDF module I use to parse from stream and serialize into one
I can solve the problem with the dependency between getRecipe and createMeal by simply adding some information from getRecipe to parseRDF, this can be done very easily with RDF without breaking the original data model. But I would still be interested to know if I could handle dependencies like this with pipes
yes, It's okay to make a stream of js objects,
you just have to remember to pipe it through something that will serialize the stream again after before writing it to IO.
I'd recomend writing a module called rdfStream that parses and serializes rdf, you would use it like this
var rdf = require('rdf-stream')
fs.createReadStream(file) //get a text stream
.pipe(rdf.parse()) //turn it into objects
.pipe(transform) //optional, do something with the objects
.pipe(rdf.stringify()) //turn back into text
.pipe(process.stdout) //write to IO.
and it could also be used by other people working with rdf in node, awesome!
Related
As seen here: https://anidiots.guide/coding-guides/storing-data-in-a-json-file.html
It shows you how to create a point system in discord.js. But what caught my eye is how they used let points = JSON.parse(fs.readFileSync("./points.json", "utf8"));
to read the file. So i am trying to learn how to make a database where i get the points plus money that can be redeemed daily and shared. kinda like a bank. but i don't know how to do that. If anyone could help me with a hastebin link or anywhere i can learn in depth how to use the JSON.parse(fs.readFileSync("./points.json", "utf8")); thing.
and if you want to see my bot in action don't hesitate to use https://discord.me/knut
The line you're asking about is made of two call to the functions JSON.parse and fs.readFileSync.
JSON.parse. This function receives a bunch of text and transform it (parse it) into a javascript object. It can be very useful when you want to, for example, build something dynamically based on the content of a file. Maybe w3school is a good place to start looking for info about it.
Example
var string = "{id: 4, name:'Volley'}"
var parseObject = JSON.parse(string)
console.log(parseObject.id); //4
console.log(parseObject.name); //Volley
fs.readFileSync. As you probably know, most of the functions in javascript and node.js are asynchronous, that is, instead of calling and get the returned value, you have to define a callback within which you would use the value you want. fs.readFileSync is just the synchronous version of fs.readFile(callback), which returns the content of the read file. Here you have the docs about that function.
These functions are actually simple to use, you should struggle in finding some examples or trying them by yourself.
If you want to imitate what the tutorial said, then you would need to define another file, with the money of each point, or edit the first file if you can, so you could an object like
var point_and_money = {
points : [...],
money : [....]
}
or two objects with the separate information
var points = JSON.parse(fs.readFileSync("./points.json", "utf8"));
var money = JSON.parse(fs.readFileSync("./money.json", "utf8"));
Hope I gave you a hint about what you asked
not really sure what you are trying to achieve?
JSON.parse(fs.readFileSync("./points.json", "utf8"));
This line reads a json file and parse it to a Javascript-Method. Nothing less and nothing more. this can also be done in Nodejs via
var points = require('./points.json');
You mentioned something like how to do a database? Basically I am not sure if you want to develop a database or better use an existing one. Look for MongoDB, SQLLite,IndexedDB, etc. There a tons of database for almost every use case.
Remember that your line of code reads synchronous in a blocking way when the file gets large.
And when multiple users would access the file at the same time you need to handle this somehow. So definitely would suggest to look for some existing database solution and have more time to focus on your business logic.
I hope I understand your question correct and my answer helps.
Maybe this one is also a good question to start: Lightweight Javascript DB for use in Node.js
I have been using a DOMParser object to parse a text string to an XML tree. However it is not available in the context of a Web Worker (and neither is, of course, document.ELEMENT_NODE or the various other constants that would be needed). Is there any other way to do that?
Please note that I do not want to manipulate the DOM of the current page. The XML file won't contain HTML elements or anything of the sort. In fact, I do not want to touch the document object at all. I simply want to provide a text string like the following:
<car color="blue"><driver/></car>
...and get back a suitable tree structure and a way to traverse it. I also do not care about schema validation or anything fancy. I know about XML for <SCRIPT>, which many may find useful (hence I'm linking to it here), however its licensing is not really suitable for me. I'm not sure if jQuery includes an XML parser (I'm fairly new to this stuff), but even if it does (and it is usable inside a Worker), I would not include an extra ~50K lines of code just for this function.
I suppose I could write a simple XML parser in JavaScript, I'm just wondering if I'm missing a quicker option.
according to the spec
The DOM APIs (Node objects, Document objects, etc) are not available to workers in this version of this specification.
I guess thats why DOMParser is not availlable, but I don't really understand why that decision was made. (fetching and processing an XML document in a WebWorker does not seems unreasonnable)
but you can import other tools available: a "Cross Platform XML Parsing in JavaScript"
At this point I like to share my parser: https://github.com/tobiasnickel/tXml
with its tXml() method you can parse a string into an object and it takes only 0.5kb minified + gzipped
A psuedo controller method
#RequestMapping("/foo")
public String getFoo(Model model) {
model.add("foo", repo.findFoo());
model.add("bar", repo.findBar());
model.add("barOptions", repo.findBarOptions(bar));
return "fooView";
}
Let's say the client uses Expression language to render foo and bar; but we use JavaScript to render barOptions.
<html>
<script>
var options = <mytag:toJSON object="${barOptions}"/>;
$("#options").renderOptions( options );
</script>
<body>
<mytag:renderFoo foo="${foo}"/>
<mytag:renderBar foo="${bar}"/>
<ul id="options"></ul>
</body>
</html>
Common conventions tells me this is bad. But the essence of MVC, where the controller sends data and the view determines how to use it, tells me this is good. Is there a better way to do the same thing? Is there any reason why this isn't commonly done? I could request the JSON using a separate call, but then I have to make more requests for the page to load, and there may be logic to determine barOptions in the controller method getFoo() based on other input at the time of the page load.
At first glance I can not say that I see anything blatantly wrong with approach. The only aspect that initially took me off guard was your need to convert data in the model object to json.
For me, JSON usually implies that there is some sort of server side object that needs to be converted so that a client side javascript can access or manipulate its structure in a javascript way. I guess without knowing more of the purpose of the options list, I can't see a reason why json serialization is required here.
By using a Tag to convert a model object to JSON, we avoid an
additional request made by the client
But if we assume that JSON is a requirement (perhaps for some third party jquery plugin), then I absolutely do not see anything wrong with approach.
What is special or different about the barOptions unordered list, why does it have be rendered with json? Why not just use a for loop to build the list items? Or you can have a custom tag that builds out the ul entirely.
Aside from that, I missing the point as how one may perceive this as being bad code.
Normally JSON is requested by JavaScript with an Ajax call, but if in your case it's available at page rendering time, I see nothing wrong with your solution. It's clean, compact code and easy to read. Looks perfectly OK to me, the alternative would be to loop on the options array with a forEach, but this approach looks better.
I basically want to be able to:
Write a few functions in python (with the minimum amount of extra meta data)
Turn these functions into a web service (with the minimum of effort / boiler plate)
Automatically generate some javascript functions / objects for rpc (this should prevent me from doing as many stupid things as possible like mistyping method names, forgetting the names of methods, passing the wrong number of arguments)
Example
python:
def hello_world():
return "Hello world"
javascript:
...
<!-- This file is automatically generated (either dynamically or statically) -->
<script src="http://myurl.com/webservice/client_side_javascript"> </script>
...
<script>
$('#button').click(function () {
hello_world(function (data){ $('#label').text(data)))
}
</script>
A bit of research has shown me some approaches that come close to this:
Automatic generation of json-rpc services from functions with a little boiler plate code in python and then using jquery and json to do the calls (still easy to make mistakes with method names - still need to be aware of urls when calling, very irritating to write these calls yourself in the firebug shell)
Using a library like soaplib to generate wsdl from python (by adding copious type information). And then somehow convert this into javascript (not sure if there is even a library to do this)
But are there any approaches closer to what I want?
Yes there is, there is Pyjamas. Some people bill this as the "GWT for Python"
It looks like using a javascript XML RPC client (there is jquery plugin for this) together with an XML RPC server is a good way to go.
The jquery plugin will introspect your rpc service and will populate method names make it impossible to mis type the name of a method call without getting early warning. It will not however test the number of arguments that you pass, or their type.
There doesn't seem to be the same support for introspection on json rpc (or rather there doesn't seem to be a consistent standard). This approach can also be used with django.
I've put together some example code and uploaded it here (I hope that linking to one's blog posts isn't considered terrible form - a brief search of the internet didn't seem to suggest it was)...
I'm using Django and Apache to serve webpages. My JavaScript code currently includes a data object with values to be displayed in various HTML widgets based on the user's selection from a menu of choices. I want to derive these data from a Python dictionary. I think I know how to embed the JavaScript code in the HTML, but how do I embed the data object in that script (on the fly) so the script's functions can use it?
Put another way, I want to create a JavaScript object or array from a Python dictionary, then insert that object into the JavaScript code, and then insert that JavaScript code into the HTML.
I suppose this structure (e.g., data embedded in variables in the JavaScript code) is suboptimal, but as a newbie I don't know the alternatives. I've seen write-ups of Django serialization functions, but these don't help me until I can get the data into my JavaScript code in the first place.
I'm not (yet) using a JavaScript library like jQuery.
n.b. see 2018 update at the bottom
I recommend against putting much JavaScript in your Django templates - it tends to be hard to write and debug, particularly as your project expands. Instead, try writing all of your JavaScript in a separate script file which your template loads and simply including just a JSON data object in the template. This allows you to do things like run your entire JavaScript app through something like JSLint, minify it, etc. and you can test it with a static HTML file without any dependencies on your Django app. Using a library like simplejson also saves you the time spent writing tedious serialization code.
If you aren't assuming that you're building an AJAX app this might simply be done like this:
In the view:
from django.utils import simplejson
def view(request, …):
js_data = simplejson.dumps(my_dict)
…
render_template_to_response("my_template.html", {"my_data": js_data, …})
In the template:
<script type="text/javascript">
data_from_django = {{ my_data }};
widget.init(data_from_django);
</script>
Note that the type of data matters: if my_data is a simple number or a string from a controlled source which doesn't contain HTML, such as a formatted date, no special handling is required. If it's possible to have untrusted data provided by a user you will need to sanitize it using something like the escape or escapejs filters and ensure that your JavaScript handles the data safely to avoid cross-site scripting attacks.
As far as dates go, you might also want to think about how you pass dates around. I've almost always found it easiest to pass them as Unix timestamps:
In Django:
time_t = time.mktime(my_date.timetuple())
In JavaScript, assuming you've done something like time_t = {{ time_t }} with the results of the snippet above:
my_date = new Date();
my_date.setTime(time_t*1000);
Finally, pay attention to UTC - you'll want to have the Python and Django date functions exchange data in UTC to avoid embarrassing shifts from the user's local time.
EDIT : Note that the setTime in javascript is in millisecond whereas the output of time.mktime is seconds. That's why we need to multiply by 1000
2018 Update: I still like JSON for complex values but in the intervening decade the HTML5 data API has attained near universal browser support and it's very convenient for passing simple (non-list/dict) values around, especially if you might want to have CSS rules apply based on those values and you don't care about unsupported versions of Internet Explorer.
<div id="my-widget" data-view-mode="tabular">…</div>
let myWidget = document.getElementById("my-widget");
console.log(myWidget.dataset.viewMode); // Prints tabular
somethingElse.addEventListener('click', evt => {
myWidget.dataset.viewMode = "list";
});
This is a neat way to expose data to CSS if you want to set the initial view state in your Django template and have it automatically update when JavaScript updates the data- attribute. I use this for things like hiding a progress widget until the user selects something to process or to conditionally show/hide errors based on fetch outcomes or even something like displaying an active record count using CSS like #some-element::after { content: attr(data-active-transfers); }.
For anyone who might be having a problems with this, be sure you are rendering your json object under safe mode in the template. You can manually set this like this
<script type="text/javascript">
data_from_django = {{ my_data|safe }};
widget.init(data_from_django);
</script>
As of mid-2018 the simplest approach is to use Python's JSON module, simplejson is now deprecated. Beware, that as #wilblack mentions you need to prevent Django's autoescaping either using safe filter or autoescape tag with an off option. In both cases in the view you add the contents of the dictionary to the context
viewset.py
import json
def get_context_data(self, **kwargs):
context['my_dictionary'] = json.dumps(self.object.mydict)
and then in the template you add as #wilblack suggested:
template.html
<script>
my_data = {{ my_dictionary|safe }};
</script>
Security warning:
json.dumps does not escape forward slashes: an attack is {'</script><script>alert(123);</script>': ''}. Same issue as in other answers. Added another answer hopefully fixing it.
You can include <script> tags inside your .html templates, and then build your data structures however is convenient for you. The template language isn't only for HTML, it can also do Javascript object literals.
And Paul is right: it might be best to use a json module to create a JSON string, then insert that string into the template. That will handle the quoting issues best, and deal with deep structures with ease.
It is suboptimal. Have you considered passing your data as JSON using django's built in serializer for that?
See the related response to this question. One option is to use jsonpickle to serialize between Python objects and JSON/Javascript objects. It wraps simplejson and handles things that are typically not accepted by simplejson.
Putting Java Script embedded into Django template is rather always bad idea.
Rather, because there are some exceptions from this rule.
Everything depends on the your Java Script code site and functionality.
It is better to have seperately static files, like JS, but the problem is that every seperate file needs another connect/GET/request/response mechanism. Sometimes for small one, two liners code os JS to put this into template, bun then use django templatetags mechanism - you can use is in other templates ;)
About objects - the same. If your site has AJAX construction/web2.0 like favour - you can achieve very good effect putting some count/math operation onto client side. If objects are small - embedded into template, if large - response them in another connection to avoid hangind page for user.
Fixing the security hole in the answers by #willblack and #Daniel_Kislyuk.
If the data is untrusted, you cannot just do
viewset.py
def get_context_data(self, **kwargs):
context['my_dictionary'] = json.dumps(self.object.mydict)
template.html
<script>
my_data = {{ my_dictionary|safe }};
</script>
because the data could be something like
{"</script><script>alert(123);</script>":""}
and forward slashes aren't escaped by default. Clearly the escaping by json.dumps may not 100% match the escaping in Javascript, which is where the problems come from.
Fixed solution
As far as I can tell, the following fixes the problem:
<script>
my_data = JSON.parse("{{ my_dictionary|escapejs }}");
</script>
If there are still issues, please post in the comments.