Develop Reference

Parsing and Modifying local HTML with Node.js

I kind of have a weird question that's probably very basic. I'm pretty new to web development and front-end stuff in general.
Premise
First of all, my only real other experience with JS is using node.js to make a discord bot, as well as use D3 to make some basic GUI stuff.
I'm trying to make a GitHub pages wiki for a game I play. In the game, there are units, which I've encoded as JSON files, as well as a folder of images. All I really want is a basic index.html page with links to HTML pages for all the units. But since units are constantly being added, I am constantly updating the JSON so I want to be able to generate the HTML files for the units as well as updating index.html automatically.
The Method
Since all I really know is node.js, I've been trying to use that to make the HTML files. What I'm trying to do right now is this hacky thing of using fs to read in the HTML files, then use jsdom to parse them into something I can use jquery to edit. After that I just want to serialize the jsdom into a string and use fs to write to the file. Right now I can successfully deserialize with jsdom and serialize the html back with fs. But I'm having trouble editing the actual HTML.
Question
First of all, I have no idea whether this is a good way to do what I've described at all, I'm still in the very early stages, so I'm welcome to suggestions on other libraries or just completely redoing the project. The closest thing I've found that's similar to what I want to do is this GitHub snippet from 10 years ago, unfortunately, I'm sure a lot of it is outdated and my puny knowledge is not able to adapt to this to modern js: https://gist.github.com/clarkdave/959811. I was wondering whether someone could do that for me, or suggest a better way of doing what I'm trying to do. You can see what I'm currently trying to do here: https://github.com/aayu3/ATBotJSONDependencies

"In order to be able to generate HTML, and Auto-Update webpages based on a configuration, input, &/or data, you need a template engine. You might not know it, but that is what your building already, well sorta. You have the right idea, using fs to read your HTML, but your going to need to go a few steps further."
Read your HTML files, and store the read HTML as a string inside your Node.js code-base.
"sounds like you have done this already."
Build a Parsing-Engine
"You can build one, or implement a pre-built parser via a Node.js module, you can probably find a few different parsers on NPM."
Create a Library that Manipulates Strings
"You need to implement a library that can alter the HTML after it has been read. You can implement a simple library, custom for a single webpage, or you can build something that is a bit more dynamic in its use, something that you can re implement for a later purpose, which, semantically, would actually be closer to an API, rather than a library."
Build/Implement a Compiler
"The last thing you need to implement is a "compiler". This takes the HTML, that you have altered — which should be in the form of a string at this point — and compiles it back to HTML. Like the parser, you can implement a pre-built compiler, or write it yourself."
"Most people do not build their own engines, because its not financially feasible. To build something quickly, and easily, its far more efficient to use a tool that has been pre-built, and a lot of the tools that already have template engines built into them, have been built by very good opensource developers, or companies with a lot of money (e.g. React is built by Facebook)."
There's really no shortage to the options you have.
I have listed some alternate options you can choose below:
"Including a tutorial about building your own template engine, which is considered by most to be the hard way. The hard-way isn't always the worst way though. In this case, choosing the hard-way — building your own template engine — offers an insight to the mechanics of the popular tools that contemporary web developers use,"
The first is the obvious HTML Template Engine.
Right now Handlebars and Pug are really popular. A template engine is essentially a rendering tool that that coverts a special variable syntax used in your html to data that is stored, usually, in JSON format somewhere, and if that data is dynamic, the variable in your page will change depending on the state of your data. It might sound a bit confusing at first, but I promise this is quite simple. Bellow is the syntax commonly used.
// Your Data.
{
animal: "My little dog, Sophie.",
place: "Santa Rosa, Ca.",
}
// Your HTML template, or your input.
<h3>{{place}}</h3>
<p>{{animal}}</p>
// Your HTML output, that is compile and rendered by your template engine.
<h3>Santa Rosa, Ca.</h3>
<p>My little dog, Sophie.</p>
Here is a link to an online tool built by Handlebars, for experimenting with their template engine. In-fact I used that tool to create the example above.
You have two other options
For the second option, you could use a full blown framework like React or Vue. Personally I have never used React, but I have spent sometime with Vue. Vue implements the same Double Curly Bracket Syntax that the template engines use (which is demonstrated in the above example). Vue takes things a step further though, or it might be more correct to say Vue takes things 5-thousand and 42 steps further which is why some people choose it over a template engine, on the flip side it is also why many people choose to not use it (same goes for React). I am a Vanilla JS Guy myself...
...and as a Vanilla lover the 3rd Option you have, is what I do. Build a simple yet extremely powerful template engine for yourself. Though this may sound the most ludicrous and time consuming, its not. The frameworks will take you far more time to learn, and building a template engine for your self only takes a few functions, though it would be quicker to implement one that's already made for you from Option #1 above, however; Building your own template engine ensures you understand whats going on under that hood of yours, and not only that, but you become a mechanic to your own engine under your hood. You will be able to swap parts and make adjustments as you please. Bellow is a small bit of code that looks simple, but has a lot going on in it.
var render = (template, data) => {
return template.replace(/{{(.*?)}}/g, (match) => {
return data[match.split(/{{|}}/).filter(Boolean)[0]]
})
}
The snippet above was written by ShadowTime2000 at Hackernoon.com.
The snippet above is a great example for demonstrating the render function that is at the heart of most every JavaScript template engine (for the sake of correctness there are exceptions, but that is irrelevant here). The snippet above came from an entire guide that is a wonderful free resource that I suggest reading. Link is below.
Shadowtime2000's Tutorial on Template Engines
By the way I don't know shadow-time, and I am not trying to market his stuff. I really like the way he thinks. IMHO, Shadowtime2000 writes, very useful tutorials/guides, which is becoming increasingly harder to find the more I learn.
I hope something above helps you mate,
CHEERS!

Migrating Javascript from the database

I am working on an old enterprise solution with these properties:
The solution has a MVC web application
The solution has a WCF service layer
The solution has javascript in the database, in the form of functions in a database column
The web application retrieves said javascript through the service layer and plugs it into certain pages
My team cannot modify the web application, nor the service layer
My team must write javascript by inserting functions into said database columns
This architecture leads to:
A very inefficient development loop
Very poor source control
I'd like to propose a solution for them, how to upgrade this, but here's where I fall a bit short on experience. My suggestion would be:
Migrate the javascript from the database to javascript files
Make some sort of hook in the web application for other teams' javascript files
My questions are:
Has anyone had this kind of problem and how did they solve it?
Is there an effective way to do this kind of javascript migration into files? My idea would be to write a small console program to do the migration
How would they make a hook to import our javascript files? My idea is to make a script bundle with some naming convention, so we can add scripts without them needing to change their code. Are there problems with this approach?
Any kind of input would be invaluable.
Edit:
Additional explanation:
The mechanism maps the javascript function names to a certain DOM elements' event attributes and inlines the code right after the element
The functions are standalone functions, depending only on libraries already in the web application
The functions are grouped by a common form
So I suppose it would be better to group them into files bearing the form names.

If these are just simple, static function definitions being inlined into the web page, then I suppose it might be possible to serialize/aggregate them all into a giant file and run something like prettier on it to make it readable.
That wouldn't be ideal to gain traction in your proposed migration, though. If the code has any volume to it at all, it would be nice to give some structure and order to maintain it.
It's already kind of a huge assumption that this javascript is just pure functions without any complex dependencies on each other, but it's possible that these pieces of Javascript work in isolation already if they are being pulled out of a database. It's hard to know without knowing more context. It seems unlikely that your life will be that easy.
If you managed to extract this monolithic Javascript file, the easiest thing to do would be include it in a script tag for the entire site and be done with it. This could be a bad idea if the file is getting to the ~MB size and slows your initial page load time.
Then again, the point at which you have a bunch of functions in one file, you could probably do a lot there to optimize and reduce duplication of code.
This is still all conjecture because I don't know the mechanism by which your web application imports the javascript once it retrieves it from the database.

Javascript C++ binding?

I have some C++ code that I want to expose to client side of a web app. Ideally, I want to write Javascript wrapper objects for my C++ classes so that I can use them clientside.
Has this been done before?. Does anyone have a link to show how this may be achieved?

There is a library to convert C++ code to javascript, it might help:
emscripten

Libjspp C++ template based wrapper for embedding and extending Javascript engine spidermonkey 1 . 8 . 5 and more
SpiderMonkey? is Mozilla Project's Javascript/ECMAScript engine.
Libjspp allows C++ developers to embed SpiderMonkey? simply and easily into their applications. Libjspp allows to run multiple Javascript Engines within same process which suits one engine per thread para dime which is helpful in achieving true parallisim. Also Libjspp no way stops user from running multiple threads within engine.
http://code.google.com/p/libjspp/

I guess that RPC is what you want. You'll need to wrap your functions on the server side using some sort of framework. I've not yet used it, but this one looks promising.
On the client side you use proxy objects to dispatch the function calls. The communication is handled usually either via XML-RPC or JSON-RPC. I used this client side framework and was quite content but I'm sure you'll find many others.

This is an old topi, however, I was in the exact situation right now, and all of the solutions I found on the net complicated or outdated.
Recently, I ran across a library which supports V8 engine (including the new isolation API, which makes 90% of the libraries I found outdated) and provides great exposure and interaction API.
https://github.com/QuartzTechnologies/v8bridge
I hope that my solution will help anybody.

There's a relatively new library for doing this called nbind. Maybe that would suit you? It looks very good to me, and I'm just about to start using it.

I think you want a C++ JSON parser. You should be able to find one here http://www.json.org/. It may not do all you want because it just serializes and deserializes C++ objects without any behavior, but it should be good enough. See https://stackoverflow.com/questions/245973/whats-the-best-c-json-parser for some discussion.

If the C++ code has to be on the client, then there is no simple way to do this for a web app. A solution may involve coding plugins for the browsers you want to support, which may then be accessed from javascript code.
If, for example, you need this for a client application, that is another case. Such a thing has been done and involves linking your application to (or running from outside) with for example chromium library, or any other javascript execution engine. That way you can create bindings to C++ classes and use such objects from javascript and vice-versa. Note that this is also not a trivial solution and may be a big effort to implement (also requires additional resources).

You could for example wrap the C++ classes in PHP or Python, and then implement an API over HTTP to access the required functions.
Or if you insist on exposing the functions as JavaScript you could try using Node.js, and create an C++ add-on to wrap you classes. See the Node.js documentation here: http://nodejs.org/api/addons.html#addons_wrapping_c_objects
But either way, I don't think avoid creating some sort of API (HTTP SOAP, XML RPC) to access the functions on your server.

Though QML is not exactly Javascript, Qt is not plain C++, but what they do together seem just like what you need

using haml/jade in kanso couchapp

following this post, I took a look at kanso.
From this I learnt that people are not afraid to load to the database context complicated modules if they need them, and that encouraged me a lot.
So I tried kanso. It gave me some trouble that implied immaturity - however, it demonstrateds a great potential.
(mostly compatibiliy view with npm, with node 0.6.x, and some open edge-cases)
So I looked a little deeper.
I saw it comes with a templates engine of it's own.
But what if I want to reuse templates that are written already by another standard?
(for example - haml, or even better - jade that has also a nice text-to-text JS implementation, and a well growing jQuery plugin - same link - see end of document ).
Does anybody here know how coupled the templates engine with the kanso types mechanism, and how simple should it be to use other template engines instead?
Or, what are the limitations I take upon myself when trying to use my own templates?

As far as I can see in the source, DustJS (the template engine of kanso) is not pluggable (like in expressjs for example). That being said, it probably won't be that hard to plug in a different templating engine, the code doesn't seem very complicated.
You might want to add something to this issue on the GitHub page and request for a pluggable templating mechanism.

How do you manage a big script?

I'm building a JavaScript engine. I used a simple function as a class, added subfunctions, strings... and used prototypes to add other objects...
Everything is fine. The problem now, is that I'm using one class (or main function), because I need it to be so. It's becoming huge, to the point that I can't control it any more and debugging the code is becoming as hard as hell.
I'm a C# developer, who used Visual Studio. I never come across this problem, because I have different files/classes/forms...
I wonder how developers here deal with large JavaScript files. What tools/strategies/techniques do you use to solve this issue.
Thanks

You can still create multiple objects analogous to your classes and objects in C#.
I suggest you go look at some JavaScript frameworks like YUI3 ( http://developer.yahoo.com/yui/3/ )
These provide nice mechanisms for structuring your JavaScript code and behaviours. For example YUI3 provides the Y.extend method to allow you to extend one object from another much like you do in C#. There's a whole suite of other mechanisms you can use in JavaScript that are actually a lot more powerful than what you've learnt in C#.
Go look up prototypal inheritance, and maybe watch some of the videos by Douglas Crockford on the YUI Theater ( http://developer.yahoo.com/yui/theater/ ). All really excellent stuff that'll show you how you can do this sort of thing in JavaScript without the major headache of giant scripts.
To answer you question more specifically, I use the module pattern in YUI3 to allow me to split my code up into mulitple files. It allows a mechanism where you can define modules then use them from other files, while auto-resolving the required dependancies. So you define multiple JavaScript files containing your behaviours and code defining various modules, which can then be 'used' in other files.
YUI().use('my-module', 'my-other-module', function(Y) {
Y.MyModule.doSomething();
Y.MyOtherModule.doSomethingElse();
});
'my-module' and 'my-other-module' can be defined in completely different JS files, which I then tell the YUI loader where to find, and it automatically includes them into my pages.
This is really powerful, and lets you break up your code for maximum reuse and simplicity. Lots of other developers are also putting their code up on the YUI Gallery so you can mix and match behaviours into your projects.

jQuery. It's a great tool to help you write less and better code. It's a simple JS framework/library you link to on every html page. It should be quite simple to learn, and there's a lot of support, manuals, books.

Google created GWT to solve this problem.