I am using Javascript wrapped in HTML to simplify the task of one of my students. Her task is to create a text file for a research project, which will act as a configuration file for the analysis software.
I decided on Javascript, because I wanted portable, transparent code, with zero dependencies (no libs, no server, no installation), yet a familiar feel from the web that is easy to get started with. However, client side Javascript appears to have its limits when it comes to handling file output.
There are multiple questions and answers on Stackoverflow that address the issue by appealing to server-side solutions, external dependencies, and the newer HTML5 download element.
I have considered if I should use a complementary script or batch file that reads the output, but I am not sure about how to best implement such a layer. The file is complex to generate and this is achieved using form elements.
Another idea would be to package the script as an executable. For example, a browser could be called from Java, or the HTML/JS could be converted somehow. Perhaps there is a wrapper that I am not familiar with.
This is one of those side projects that is fast to code and so I would assume that there is a go-to solution among programmers for this type of problem. On the one hand, this is a packaging problem. On the other hand, it is about some of the limitations with Javascript for projects that run without a server backbone.
How can I deliver a no-bells-and-whistles Javascript application that is local only and capable of handling file I/O?
Related
I am working on an old enterprise solution with these properties:
The solution has a MVC web application
The solution has a WCF service layer
The solution has javascript in the database, in the form of functions in a database column
The web application retrieves said javascript through the service layer and plugs it into certain pages
My team cannot modify the web application, nor the service layer
My team must write javascript by inserting functions into said database columns
This architecture leads to:
A very inefficient development loop
Very poor source control
I'd like to propose a solution for them, how to upgrade this, but here's where I fall a bit short on experience. My suggestion would be:
Migrate the javascript from the database to javascript files
Make some sort of hook in the web application for other teams' javascript files
My questions are:
Has anyone had this kind of problem and how did they solve it?
Is there an effective way to do this kind of javascript migration into files? My idea would be to write a small console program to do the migration
How would they make a hook to import our javascript files? My idea is to make a script bundle with some naming convention, so we can add scripts without them needing to change their code. Are there problems with this approach?
Any kind of input would be invaluable.
Edit:
Additional explanation:
The mechanism maps the javascript function names to a certain DOM elements' event attributes and inlines the code right after the element
The functions are standalone functions, depending only on libraries already in the web application
The functions are grouped by a common form
So I suppose it would be better to group them into files bearing the form names.
If these are just simple, static function definitions being inlined into the web page, then I suppose it might be possible to serialize/aggregate them all into a giant file and run something like prettier on it to make it readable.
That wouldn't be ideal to gain traction in your proposed migration, though. If the code has any volume to it at all, it would be nice to give some structure and order to maintain it.
It's already kind of a huge assumption that this javascript is just pure functions without any complex dependencies on each other, but it's possible that these pieces of Javascript work in isolation already if they are being pulled out of a database. It's hard to know without knowing more context. It seems unlikely that your life will be that easy.
If you managed to extract this monolithic Javascript file, the easiest thing to do would be include it in a script tag for the entire site and be done with it. This could be a bad idea if the file is getting to the ~MB size and slows your initial page load time.
Then again, the point at which you have a bunch of functions in one file, you could probably do a lot there to optimize and reduce duplication of code.
This is still all conjecture because I don't know the mechanism by which your web application imports the javascript once it retrieves it from the database.
One big benefit I've always perceived with using NodeJS on the server is the potential for sharing bits of code between the server and client side (ex. input validation). Now that I'm actually developing using NodeJS one difficulty that I've found is determining the responsibility and context in which each body of code is executed. Below I'll list a few of the difficulties I've had in hopes gain some enlightenment on conventions or guidance that I may be overlooking that could help elevate these issues.
Build-Time Code
Build time code for projects that use Gulp, Grunt, or vanilla NPM in a way that follow the basic documentation are generally pretty easy to follow. Most smaller projects tend to keep all of the code within a single file and the file tends to be named a conventional name like gulpfile.js, however with bigger projects I've seen these scripts begin to be split out. I've seen some cases where the gulp file is split into multiple files and placed under a separate directory. Even worse I've found cases where the gulpfile.js file isn't even named as such causing new developers to hunt around to find where the gulpfile is located and once it is located the gulp command always has to be run with the specific --gulpfile option.
Run-Time Server-Side Code
The entry point for basic node applications appear to simply require pointing out a specific JavaScript file when running the node command (ex. node script.js). For web server applications, such as those using Express, I've noticed that by convention the entry point file is often called server.js and can usually be found in the root directory of the application. In some other cases however such as when running the web server in a developer environment I've seen gulp tasks take on the responsibility of launching Node. In these cases there seems to be multiple ways to include the entry point but one example I've found is just starting up the webpack complier followed by a require statement for the entry point script. Figuring out how to incorporate normal guidance on how to accomplish a typical node debug command is non-trivial in this type of setup. Besides the entry point of the application, there doesn't seem to be any general guidance on directory structure for NodeJS/Express applications that keeps server-side specific code in it's place to help locate it and to keep it separate from build-time and client-side code.
The server-side story becomes even more complex in cases where the server side code is used both for the purpose for serving up static content, server-side generated views (such as with MVC), as well as for providing an API to the client side. My preference is to separate API from the application project but I get the feeling from others that there is a sense of overcomplexity involved in doing so where I see it as a reasonable separation of concerns.
Run-Time Client-Side Code
Since client-side code can often have various entry points based on the first page that is requested this can be tricky. However because of the general transparency of URLs with how they map to resources for typical cases it as well as how powerful the debugging tools have become in modern browsers, it isn't too much trouble following the follow of the scripts. The difficult instead for the client side code comes more for typical build processes which generally end up copying the files around and placing them into a production like structure under a different name. An example would be a project that has a folder called src or js that holds client-side and server-side code intermingled except for that only a portion of the files happen to be included in a build task which transforms and often concatenates the files and places them in a distribution folder. Common names of these distribution folders that I've seen are dist, public, www, and wwwroot. Often if not always these directories are at the root of the project which at least makes it a bit easier to locate without having to interrogate the build scripts.
My hope is that there is some general guidance on how to put all of this together in a sane way perhaps by an authoritative source mainly to give guidance to those like myself who may want start off on the right foot. As a side effect perhaps being able to reference some sort of standard even if it is a loose one may also reduce the amount of boilerplate a team has to invent and discuss as they get started. Within each of the contexts listed above there will obviously be some technology specific conventions such as those followed for AngularJS, Meteor, or ReactJS on the client-side. The conventions I'm looking for are more specific to separating the main highlevel contexts in end-to-end JavaScript applications where the language and platform no longer become obvious way to differentiate between each.
Build-Time Code
IMHO if you have so much build-time code that it's more than say 1000 lines and requires more than a handful of files, something has gone off the rails. Either you don't know how to make good use of existing packages from npm or you don't understand how to refactor generic code and publish as independent npm packages. If you feel like you need guidance about a project's build-time code because it is so large and complex, my suggestion is to modularize and split out into separate projects - each published independently to npm. Also just check your overall approach. What are you doing that is so custom and requires so much machinery?
Run-Time Server Side Code
Please see my other answer to ExpressJS How to structure an application?
Generally I'd rather see client side code and server side code either completely separate npm packages (separate git repos, separate package.json files, published independently) (if they are large enough) or otherwise co-mingled in the same module and grouped by coupling (all the code relating to a feature kept together including front and back end code), especially if your code base has a substantial amount of code that works in both environments.
I have an open-source full-stack node/JS application called mjournal that keeps browser code and node code alongside each other. You can have a look and see if it feels logical to you and easy to understand where code lives. It is by no means a popular approach, so many folks will dislike it, but it feels good to me personally since I've embraced "group by coupling" as a general principle.
Figuring out how to incorporate normal guidance on how to accomplish a typical node debug command is non-trivial in this type of setup
Yeah, that's nonsense. Your app should start with npm start or something like node server.js. Elaborate gulp/grunt setups that confuse new developers are unnecessary complexity you should just eliminate.
The server-side story becomes even more complex in cases where the server side code is used both for the purpose for serving up static content
In my experience, the code to serve up static content boils down to 5 lines or less, so no big deal. If you have a non-microscopic amount of code dealing with serving static content, again something has gone way off the rails.
Run-Time Client Side Code
My hope is that there is that there is some general guidance on how to put all of this together in a sane way perhaps by an authoritative source mainly to give guidance to those like myself who may want start off on the right foot.
There are some people in the node community that have adopted the "convention over configuration" approach in use by Ruby on Rails, EmberJS, and some other large projects. If you like that approach, check out tools that use it like SailsJS, EmberJS, Yeoman generators, etc.
But in general, looking for a "standard" is not how the node.js/javascript/web community rolls. Small npm packages. File layouts that are forced into obviousness due to smallness. I feel your frustration here as front-end toolchains are so complex, but ultimately it's because JavaScript in the browser took too many decades to create a reasonable module system. Things may start to standardize in the next few years now that ES6 modules are an official spec, but with so much code already written in CommonJS and it's terrible precursors like RequireJS/AMD, we'll be dealing with them probably for the foreseeable future.
I have some html files that I need to do automated processing on, basically regex replaces, but also some more complex actions like copying select blocks of text from one file to another.
I want to create a series of scripts that will let me do this processing (it will need to be done more than once on different batches of files). It would be trivial to use Go for this (read the file into memory, regex, save to disk) but I am the only member of the project that's familiar with Go.
Javascript is a tad more ubiquitous, and I do have project members who are familiar with the language, so it's a better fit in that respect. If I'm not around later, someone else could edit the scripts.
Is there a simple way to write some JS scripts to do on-disk text processing? I'm looking for a cross-platform solution (OSX, Windows). Ideally, once the scripts are written, they can be executed by double-clicking an icon--there will be "not computer people" involved at some point.
Also, I'd like to be able to do some kind of alert/message box to inform the user of the success/failure of the script. (This may be a tall order, and is of secondary importance.)
What I've looked at:
Node.js was the first thing that popped into my head, because I know that it has file system access tools, and obviously regex capacity. But I've never used Node before, and based on the tutorials I've read, it seems like overkill for something this simple.
There's a whole slew of "javascript compiling" tools that you can find by googling around. Some are not cross-platform, some seem old or not actively maintained, etc. None of them caught my eye as easy to pick up and just write some JS scripts with.
Any thoughts?
Node.js is a simple solution and with it's framework you can create or later modify your script to your needs. This way you will not be locked down by someone else's code. And it is not that difficult to to use.
Here is a quick tutorial on accesing files using node.js
http://www.sitepoint.com/accessing-the-file-system-in-node-js/
And here is a quick tutorial on using a node module called Cheerio. It allow you to access html files using "jquery like syntax". You don't need to use regex.
http://maxogden.com/scraping-with-node.html
I worked on a project for a client once and it required parsing thru hundreds of html files to check and replace certain image files based on certain criterias. I wasn't familiar with node at the time so I read some tutorials and wrote the script in about an hour.
And as long as Nodejs' path is set, you can run it on the command line.
Some tips:
You need any kind of DOM HTML parser, not only JS nor specifically JS.
You can do that thing with Java with use of jTidy or jSoup libraries (I've used second one few times). It's pretty simple language to learn if you know JS and IDE like Netbeans helps a lot. So can be made quickly with that.
You can use PhantomJS to create some job files and create shell/batch code to run them on some files. You might need to write a generator for job files (like taking a list of files, creating job files for each and running them).
You can use Node.js which isn't much overkill, I'm sure any solution won't be trivial.
You can create an ETL for processing with for example Pentaho ETL (which has JS embedded as one of two scripting languages... but without DOM parser - for that one you would need to use a bit of Java there and some library in way similar to this article).
You can also do that with PHP with Simple HTML DOM Parser - so you can make a service online (or on local server) that takes those html files and throws out processed ones.
First I think you underestimate the complexity. The statement
"It would be trivial to use Go for this (read the file into memory,
regex, save to disk) but I am the only member of the project that's
familiar with Go."
is probably false. Parsing HTML with RegExp is just a bad idea. (Google it and you will see why)
Second, if you can trivially write the code using RegExps in Go, you can just as easily write the same thing in Javascript. They both support RegExp and file operations. If you are unsure about the Javascript/Node.js details, I suggest writing the trivial solution in Go and then translate the thing into Javascript with a colleague.
Since Javascript is a script language, writing command line utilities in Node.js is straight forward.
Some pointers to get you started
RegExp in Javascript
Building command line apps in Node.js
how to zip a folder using javascript
knowing that i am making a javascript win8 store app , i have access to MyDocumnets on the client PC ,
my task now is to zip a folder (saved data of the app) and upload it to my server ??
There is no direct way to zip a folder, or even a file for that matter, using plain JavaScript (no custom libraries). This is because JavaScript is generally limited to operating in the DOM. It would be really bad for example if some malicious website was able to get you to open a page with their JavaScript on it which say, deleted random files on your machine.
However, you do have an option or two. The closest thing that's available right now (again, not considering custom libraries) is the HTML 5 File API. Support for it across major browsers varies, but most newer builds have at least limited, if not full support for it.
With the HTML 5 File API you can access, edit, and write files in a sandboxed environment in the browser. Based on this, what you could do is:
Get access to all the files you want to put in the zip using the HTML 5 File API
Write all the data from those files into a new file, using the .zip format
Save that file as myfile.zip
Part 2 there is going to be your struggle. Parts 1 and 3 are pretty straight forward. Check out this guide on getting started with HTML 5 File API.
The other, likely better option is to zip the files on your server and then just send the .zip to the client machine. Likely will be a lot easier than using the HTML 5 File API.
An alternative to finding a straight-up JS library (which is difficult), is to find a good C/C++ library and wrap it inside a Windows Runtime component. To give a little quick background, the entirety of the WinRT API is written in a way that makes it possible to project that API surface area into multiple languages such as C++, C#, VB, and JavaScript. This model is extensible, which means that you can write your own APIs that behave in exactly the same way. Visual Studio even has templates for this in the C#, VB, and C++ language options.
For an app written in JavaScript, what this makes possible is that you can access a whole host of additional APIs that are not native to JavaScript, simply by creating a small WinRT component (it's just a DLL with some metadata) that's implemented in a language like C++ that does have access to many more APIs, e.g. those in Win32, COM, and .NET. The whitelist of such APIs for Windows Store apps can be found on http://msdn.microsoft.com/en-us/library/windows/apps/br205757.aspx.
Generally speaking, for a JS app you want to implement the component in C++ (which is how WinRT is implemented). Although can also do it a bit more easily in C# or VB, you end up loading the whole CLR and taking something on the order of a 45MB memory hit.
So if you can find a good ZIP library for C++, and it uses the whitelisted APIs on the previous link, then it's straightforward to create a WinRT component that presents an interface usable from JavaScript. I'd recommend having the JS app just pass something like the StorageFolder you're looking to compress and have it return the StorageFile to upload. Then you can simply pass that StorageFile to the BackgroundTransfer API (http://msdn.microsoft.com/en-us/library/windows/apps/windows.networking.backgroundtransfer.aspx), which you should be using for any significant transfer operation as it will make sure it continues if your app is suspended.
For more details on writing WinRT components, refer to Chapter 18 of my free ebook, Programming Windows Store Apps with HTML, CSS, and JavaScript, 2nd Edition. It also details how to create async APIs which is probably what you want in this case because a zipping operation could take a long time and you don't want to block UI responsiveness.
Html can contain little bits of Javascript embedded in it (e.g. defined in onclick event handlers).
If I were writing an Html browser using a dotNet language like C#, what technologies or APIs could I use to run such Javascript fragments, given that I don't receive it until run-time (and receive it as string data, not as executable code)?
Is it any easier or harder if the code to be run were C# snippets rather than Javascript?
Is there any technique which doesn't require my code to have unusual priviledges? For example, a method like CodeCompiler.FromSource requires SecurityPermissionFlag.UnmanagedCode (which seems to me excessive: I don't see why it's so risky to compile code).
If I controlled the server-side as well as the client-side code, I could also consider compiling such script fragments on the server instead of on the client, and then sending it as precompiled code to the client side to be executed. Is there a way to send such code (a dotNet assembly, presumably) over the network to the client, have client-side code receive it from the network into client-side RAM, and invoke it on the client side without storing it as a file on a client-side disk drive?
Edit
I have answer to the first three questions: I've resigned myself to the fact that compiling takes high privileges. I don't see why; maybe (although I don't find this a very convincing reason) it's because the compiler is implemented using unmanaged code. Maybe this will change when they reimplement the compiler using managed code, in maybe the "C# version 5" timeframe. In any case, whatever the reason, that seems to be the way it is, and there are no work-arounds (other similar APIs but which require fewer privileges).
My remaining question then is how to get an Assembly instance from one machine to another. When I have time I'll find out whether untrusted code can run the Assembly.Load(byte[] rawAssembly) method.
Server side Javascript is one of the languages supported by the .NET platform. I used it many times in the scenrios when you need to insert small code snippets into existing code. Runtime it can be loaded from i.e. database and compiled, so there is no preformance penalty.
From the standpoint of making the plumbing work (retrieveing the source, compiling it, etc.) there is no difference. With strongly typed languages though it is much more difficult to assemble code snippets into a compilable compilation unit.
Permissions is certanly a challenge. I am not sure about the specific permission you mentioned, but security is a concern, after all the source you compile can be anything and if you are not careful about the source of your code it can become the backdoor into your system
The answer to this one is - yes of course. You can load an assembly from anywhere, not necessarily from a file, you can also compile in memory - that's what I do. There is no dll file in this case.
You're asking several questions, sort of, so I'll give you an idea on one of them.
There's a very good article and some code samples from:
http://www.west-wind.com/presentations/dynamicCode/DynamicCode.htm
which talks about compiling and executing C# code at runtime. I found it very useful and I am using this in a standard c# application. Seems like it would be usable for your problem as well.