Develop Reference

Parsing and Modifying local HTML with Node.js

I kind of have a weird question that's probably very basic. I'm pretty new to web development and front-end stuff in general.
Premise
First of all, my only real other experience with JS is using node.js to make a discord bot, as well as use D3 to make some basic GUI stuff.
I'm trying to make a GitHub pages wiki for a game I play. In the game, there are units, which I've encoded as JSON files, as well as a folder of images. All I really want is a basic index.html page with links to HTML pages for all the units. But since units are constantly being added, I am constantly updating the JSON so I want to be able to generate the HTML files for the units as well as updating index.html automatically.
The Method
Since all I really know is node.js, I've been trying to use that to make the HTML files. What I'm trying to do right now is this hacky thing of using fs to read in the HTML files, then use jsdom to parse them into something I can use jquery to edit. After that I just want to serialize the jsdom into a string and use fs to write to the file. Right now I can successfully deserialize with jsdom and serialize the html back with fs. But I'm having trouble editing the actual HTML.
Question
First of all, I have no idea whether this is a good way to do what I've described at all, I'm still in the very early stages, so I'm welcome to suggestions on other libraries or just completely redoing the project. The closest thing I've found that's similar to what I want to do is this GitHub snippet from 10 years ago, unfortunately, I'm sure a lot of it is outdated and my puny knowledge is not able to adapt to this to modern js: https://gist.github.com/clarkdave/959811. I was wondering whether someone could do that for me, or suggest a better way of doing what I'm trying to do. You can see what I'm currently trying to do here: https://github.com/aayu3/ATBotJSONDependencies

"In order to be able to generate HTML, and Auto-Update webpages based on a configuration, input, &/or data, you need a template engine. You might not know it, but that is what your building already, well sorta. You have the right idea, using fs to read your HTML, but your going to need to go a few steps further."
Read your HTML files, and store the read HTML as a string inside your Node.js code-base.
"sounds like you have done this already."
Build a Parsing-Engine
"You can build one, or implement a pre-built parser via a Node.js module, you can probably find a few different parsers on NPM."
Create a Library that Manipulates Strings
"You need to implement a library that can alter the HTML after it has been read. You can implement a simple library, custom for a single webpage, or you can build something that is a bit more dynamic in its use, something that you can re implement for a later purpose, which, semantically, would actually be closer to an API, rather than a library."
Build/Implement a Compiler
"The last thing you need to implement is a "compiler". This takes the HTML, that you have altered — which should be in the form of a string at this point — and compiles it back to HTML. Like the parser, you can implement a pre-built compiler, or write it yourself."
"Most people do not build their own engines, because its not financially feasible. To build something quickly, and easily, its far more efficient to use a tool that has been pre-built, and a lot of the tools that already have template engines built into them, have been built by very good opensource developers, or companies with a lot of money (e.g. React is built by Facebook)."
There's really no shortage to the options you have.
I have listed some alternate options you can choose below:
"Including a tutorial about building your own template engine, which is considered by most to be the hard way. The hard-way isn't always the worst way though. In this case, choosing the hard-way — building your own template engine — offers an insight to the mechanics of the popular tools that contemporary web developers use,"
The first is the obvious HTML Template Engine.
Right now Handlebars and Pug are really popular. A template engine is essentially a rendering tool that that coverts a special variable syntax used in your html to data that is stored, usually, in JSON format somewhere, and if that data is dynamic, the variable in your page will change depending on the state of your data. It might sound a bit confusing at first, but I promise this is quite simple. Bellow is the syntax commonly used.
// Your Data.
{
animal: "My little dog, Sophie.",
place: "Santa Rosa, Ca.",
}
// Your HTML template, or your input.
<h3>{{place}}</h3>
<p>{{animal}}</p>
// Your HTML output, that is compile and rendered by your template engine.
<h3>Santa Rosa, Ca.</h3>
<p>My little dog, Sophie.</p>
Here is a link to an online tool built by Handlebars, for experimenting with their template engine. In-fact I used that tool to create the example above.
You have two other options
For the second option, you could use a full blown framework like React or Vue. Personally I have never used React, but I have spent sometime with Vue. Vue implements the same Double Curly Bracket Syntax that the template engines use (which is demonstrated in the above example). Vue takes things a step further though, or it might be more correct to say Vue takes things 5-thousand and 42 steps further which is why some people choose it over a template engine, on the flip side it is also why many people choose to not use it (same goes for React). I am a Vanilla JS Guy myself...
...and as a Vanilla lover the 3rd Option you have, is what I do. Build a simple yet extremely powerful template engine for yourself. Though this may sound the most ludicrous and time consuming, its not. The frameworks will take you far more time to learn, and building a template engine for your self only takes a few functions, though it would be quicker to implement one that's already made for you from Option #1 above, however; Building your own template engine ensures you understand whats going on under that hood of yours, and not only that, but you become a mechanic to your own engine under your hood. You will be able to swap parts and make adjustments as you please. Bellow is a small bit of code that looks simple, but has a lot going on in it.
var render = (template, data) => {
return template.replace(/{{(.*?)}}/g, (match) => {
return data[match.split(/{{|}}/).filter(Boolean)[0]]
})
}
The snippet above was written by ShadowTime2000 at Hackernoon.com.
The snippet above is a great example for demonstrating the render function that is at the heart of most every JavaScript template engine (for the sake of correctness there are exceptions, but that is irrelevant here). The snippet above came from an entire guide that is a wonderful free resource that I suggest reading. Link is below.
Shadowtime2000's Tutorial on Template Engines
By the way I don't know shadow-time, and I am not trying to market his stuff. I really like the way he thinks. IMHO, Shadowtime2000 writes, very useful tutorials/guides, which is becoming increasingly harder to find the more I learn.
I hope something above helps you mate,
CHEERS!

What problems does javascript frameworks like React/Angular solve?

I am new to React js and I have been trying to understand what real problem does js frameworks like React,Angular solve and why one need to learn them.
Of Course when i see advantages of framework everybody talk about "Shadow/Virtual DOM, Router, Reusable Components etc" , but then I stumbled upon below links which talk something different
https://medium.com/dailyjs/the-deepest-reason-why-modern-javascript-frameworks-exist-933b86ebc445
https://www.codementor.io/binarymuse/react-components-from-a-jquery-programmer-background-du107k2lz
but they were not clear to me
So i am meaning to understand how difficult it was to maintain js/jquery code before these framework came by.
Everybody is just asking to learn these frameworks because they are popular but none of them clearly explains why they are popular and what problem they solve and how they make our life easier when it comes to creating web-app from scratch.

Others have already mentioned the advantages and what these frameworks resolve.
I wanted to add what problems they can create:
I have seen many people that use a framework see it as a hammer and every website as a nail. In other words they either don't or can't figure out that in many cases a framework is not needed. I have seen web pages that are nearly 90% static data still download an entire framework just to provide drop menus or something else trivial.
Frameworks force you into a lifecycle and development style that, in some ways, make some things easier and other things much, much more complicated. AngularJS had a nasty digest cycle that often led to complications. Redux forces you to write much more code than needed just to handle state that on many pages is not needed. Vue, React, Angular all force you to either use their data creation functions, their way of rendering or their way of handling inter element communication. And none of these are faster than vanillaJS. Even virtual DOM has its issues.
The size of a framework is often much more than needed. I have several small libraries that do the majority of the work that a framework does but my code is in the sub 5K range if I load it all.
Some people that learn a framework first can not figure out how to write in raw JavaScript. They also tend to learn sloppy ways to write code. No, not everyone, but a large number people that did not start with vanilla JS write code that tends to be much larger and sloppier than needed.
It is best to avoid JavaScript except where really needed. Often I can use raw HTML and CSS to do everything I need to do. I find that it is rare that I need to use much JavaScript even in projects that exceed several thousand files. It is faster to allow the browser to process HTML and CSS, which all processes at compiled speeds and only use JavaScript for things you can not do in HTML and CSS.

You don't "need" to learn them, but many people and companies use them. The main advantage is being able to have your entire site on a single webpage, where you just modify the data and it handles changing the elements on the page for you. For instance, you can just think about "put the user's name here, and when they click the button, add another list field there", but you no longer have to think about "get this DOM element, set its innerHTML to the user's name, and when they click the button, create a new DOM element for input, another one for the label, get the container element, append the new children". It also allows you to more easily bind data to DOM elements, so you can just say "this field is for the variable 'numberOfItems'" instead of worrying about grabbing the DOM element and its value and storing it into numberOfItems when you need it; with React/Angular/etc. that variable will always be up to date with the user's input without you needing to update it.
TL;DR: The frameworks are not necessary, but they help automatically handle a lot of common, tedious DOM manipulation for you in a way more optimized than you'd likely end up implementing yourself, with less code for you, and all on a single page instead of having to reload the page every time you want to display new data or a new layout.

What is React JS?
ReactJS is an open-source JavaScript library that is used for building user interfaces specifically for single-page applications. It’s used for handling the view layer for web and mobile apps. React also allows us to create reusable UI components.
React allows developers to create large web applications that can change data, without reloading the page. The main purpose of React is to be fast, scalable, and simple. It works only on user interfaces in the application. This corresponds to view in the MVC template. It can be used with a combination of other JavaScript libraries or frameworks, such as Angular JS in MVC.
https://www.c-sharpcorner.com/article/what-and-why-reactjs/

This is a wide discussion about frameworks in general. Is easy to say React, Angular and so on makes easier to manipulate DOM elements and reuse components, but to apply frameworks usually are about if solve your problem.

in javascript we have load the webpage by using location.href but by using angular/react we the page updates automatically.

Implementing HTML DOM rewriting logic on the server

My application does massive rewriting of the DOM on the client side at load time. It traverses the page scanning for special markup (think Markdown) or other patterns, replacing them with sometimes rather complicated DOM structures (using DOM calls such as createElement) to style text but also create diagrams and graphics.
I adopted this architecture in order to avoid any build or preprocessing steps. It works fine in a desktop browser, but is noticeably slow on mobile devices (several seconds, even after relentless optimization). So I would like to rearchitect the system to pre-scan the page and pre-build the DOM. I'm having a bit of a mental block figuring out how to do this. Obviously, I would prefer not to rewrite all the Javascript in some other server-side language. Also, I would like to preserve the option to do the building at load time as I do now, with the basic rewriting logic sharing the same code.
The most likely-sounding option is to build this as a node app, although I am a node beginner. using jsdom both to parse the input and modify the DOM. Or, since I am an a fan of XSLT, and intrigued by Saxon-CE, even though it would mean rewriting everything, I also considered implementing the scanning/rewriting logic in XSLT, to be invoked either from node (for the pre-building case--do people use Saxon from node?) or the browser (for the load-time building case).
Can anyone comment on this approach or throw out alternative ideas?

Not sure what specific use cases you are tackling with the massive DOM rewriting, nor am I sure what your throughput requirements are. That said, one alternate approach to the node/jsdom route could be to run a farm of headless Webkit browsers, and run your current JavaScript as-is in that live rendering context. That would allow you to offload processing from those pokey mobile CPU's into arbitrarily scalable cloud resources (assuming this might be affordable for your project/business), and skirt the need to rewrite or tweak your current, working code at all.

Sounds like you want Node. If you already know JavaScript it really is a cinch to pick up.
I would recommend a tutorial like this one: http://www.nodebeginner.org/
It will take you an hour-ish to get through but gives a solid overview of Node as you build a small but functional app along with the author.

How to conciliate DRY and Loose Coupling in Javascript Libraries?

I am building my own JS library;
The idea is that the library should be comprised of small, independent modules, and some slightly larger utilities, that serve mainly to iron out browser differences.
I am having trouble getting anything done, because I am not being able to decide between staying dry or being loosely coupled.
An example? Given:
A small library that takes care of generating dom elements from a template
Another one that takes care of duck-typing issues (is_function, is_array...)
And a last one that creates modal boxes. That last one needs:
some type checking
will be creating the modals using only one function from the templating library
My options for the modal box library:
Be 100% dry, and dependant on the two other libraries. But that means if you are a user wanting to download only the modal box lib, you'll have to make with the two others
Allow users to pass an object of options on initiation that would allow them to specify the functions needed; defaulting to the ones of the libraries. This is better, but in practice, it still means, in 90% cases, using the provided libraries, as creating functions with the same signature might be cumbersome. Furthermore, it adds complexity to the modal box code.
Be 100% loose, and reproduce the functions needed in my modal box library; possibly more efficient because more targeted and there is no need to check for edge cases; but: any bug will have to be fixed in two places, and my download size increases.
So I am wasting time oscillating between the two extremes, refactoring a million times and never being satisfied.
I was going for a more generic question, but then I realized it is really pertaining to JS, because of the size & performance concern as well as the widespread usage.
Is there any known pattern to follow in such cases? What's the accepted way to go about this? Any thoughts are welcome.
[edit:]
This is the only article I found that spells out my concerns. Like the article says,
DRY is important, but so are [...] low coupling and high cohesion. [...] You have to take all [principles] into account and weigh their relative value in each situation
I guess I am not able to weigh their value in this situation.

Personally, I've always taken the view that Loose Coupling refers to creating seams in your code. In classical languages, such as Java, this is achieved by creating Interfaces which hide the concrete implementation. This is a powerful concept as it allows developers to 'unpick the seams' in your application and replace the concrete implementations with mocks and test doubles (or indeed, their own implementation). As JavaScript is a dynamic language developers rely on duck-typing instead of Interfaces: as nothing is frozen, every object becomes a seam in your code and can be replaced.
In direct answer to your question I think it pays dividends to always aim to decompose and modularize your code into smaller libraries. Not only do your avoid repeating code (not a good idea for a host of reasons) but you encourage re-use by other developers who only want to re-use parts of your library.
Instead of re-inventing the wheel, why not leverage some of the more popular JavaScript libraries that are out there; for example, underscore.js is a lightweight library which provides a rich toolkit for duck-type checks and Mustache.js may well take care of your templating needs.
Many existing projects already use this approach, for example, Backbone.js depends on underscore.js and jQuery Mobile depends on jQuery. Tools such as RequireJS make it easy to list and resolve your application's javascript dependencies and can even be used to combine all the separate.js files into a single, minified resource.

I like the concept of DRY, but your right it has a couple of problems.
Your end-user-developers will need to know that they need to download the dependencies of components.
Your end-user-developers will need to know that they need to configure the dependencies (i.e. the options to pass in).
To help solve 1. your project website could customise the download on the fly, so the core code is downloaded along with optional components. Like the modernizer download page.
To help solve 2. Rather than allowing users to pass in options, use sensible defaults to detect what parts of your packages have been loaded in the browser and automatically tie them up.
This loose coupling could also give you the great advantage that could also rely on 3rd party frameworks if the user already has them installed. For example selectivizr allows you to use jquery or dojo etc etc depending on what the browser has already loaded.
Perhaps you could use requirejs to help solve dependency management. I get the impression it's not really meant for libraries to use directly, but instead the end-user-developer... but it could be a nice fit.
I realise my answer doesn't answer your question directly, but perhaps it could help balance out some of the negative points of DRY.

Writing a JavaScript abstraction abstraction - is this sane?

I've written a JavaScript application that runs off Prototype & Scriptaculous. I'm thinking of rolling it out in as an open source product & would like it to be able to run on jQuery as well — I typically use jQuery for most of my other apps, except the site this app was originally built for.
I was originally thinking about building two separate applications, but maintaining them would be time consuming. Instead I'm considering building a library abstraction layer that detects if the page is running jQuery or Prototype and then calls the appropriate methods. I'm not going to abstract the whole libraries, just the functionality applicable to my application — namely selectors, events & effect. The core of my app is under 500 lines of code, so there isn't too much I need to worry about.
So instead of calling $('id') I would call LA.$('id') (LA for Library Abstraction) which would call $('id') in prototype and $('#id') in query etc…
Does this sound sane? I can't think of any technical hurdles, although I would have expected someone to have attempted this before. I couldn't find anything similar in my searches.

I expect that if you support the libraries only partly then no one will choose to use it, as they would have to finish the support, and you may find that maintaining it will be a headache, as there will be requests to add more functionality.
If your application is so small, why not just switch to jQuery for it, and standardize on that, as MS has done.
You may run into problems with versions, as, if someone uses it, and they are using an older version of a library, and there was some API change, then they will be wanting you to add support for that library.

I believe Ext.Js does something similar. They have a concept of "adapters" which allow you to sit Ext.JS on top of any of the underlying libraries and it will just work. The key difference is that they are using a 1-to-1 model where you say which library you want to use and it joins the dots, whereas I believe you're trying to say "I don't know which one will be available but whatever one you find, go use it".
I don't think it's insane, but you might have some fun trying to work out which library to use, particularly if both are available.

Web development frameworks (Prototype, jQuery, etc.) themselves are designed to be abstractions on top of the various browsers that exist. You ask the framework to do one thing and it has the same result (ideally) regardless of the browser. So, here, you are proposing an abstraction on top of abstractions. Presumably because you want people to be able to use your tool regardless of what framework they have chosen for their site. While it sounds like an interesting idea, I would personally have to guess that it would not work out in the long run. Largely because you do not know what the future holds. Prototype used to be the number one use framework, now jQuery has surpassed it. Perhaps another one will be very popular in a year, what if you then want to support that framework? That can be a lot of conditional code that you have to add, and in addition, force being loaded into the browsers of those using your tool.
I say either pick a single framework to support and stick with it, or maintain separate libraries. Ideally, it would be really cool if you could write some sort of builder script. This would allow you to set framework rules in some list and the script would build separate scripts for each framework based on some core script and the rules list. I honestly am not sure how to best accomplish something like this but it would effectively give you that abstraction on abstraction power that you are looking for without it being visible to the end user.