Scenario:
I am currently working on a desktop-based application involving multiple independent production sites, each having its own server, database and local clients.
The software undergoes continuous automated updates, as part of these updates each production site's database (postgresql) will need to undergo schema migrations (while maintaining persistence/transformation on the existing data during the migration).
Accessing the client machines manually (e.g. SSH / Remote Desktop) to run scripts/perform the update is not an option. The client would need to be self-sufficient in checking, downloading and installing the updates.
Problem:
I have been looking around for a few months and have not yet suitable option apart from writing SQL scripts and using a separate service to run it one by one or using the updated app to run it.
Question:
What tools would you suggest would be suitable for this scenario to ease the migration process?
It is a generally an accepted practice to run SQL scripts when doing database migrations. Tools like ActiveRecord (in the ruby world) provide an intermediate layer (a DSL), but are essentially wrappers around SQL.
There is a good reason for that - a migration is what is being done on the SQL server and the only strictly defined interface for manipulating tables in SQL servers is the PL/SQL language language flavor that the particular RDBMS server is using.
This means, that if SQL scripts are an acceptable option for migrations (acceptable in the sense that the team knows it, nobody objects to this, etc.) you should use them.
To make the task easier (the way ruby does it) you could take a look at: http://migrate4j.sourceforge.net/
They describe the tool as The initial intent of migrate4j was to make a Java version of Ruby's db:migrate. But this is not a fundamental requirement.
Related
I need to create a web interface for a Python numerical simulation code, and host both (interface and python code).
When a user will run a simulation, it could take hours, or even days for the simulation to finish (and of course it has to run many simulations in parallel for multiple users).
So I am wondering what are my options to achieve this.
Use a frontend framework for the interface and a web server for the python code ?
Or use a framework like Django (which I never used)
And also where can I host such a code (the Python code), is there any provider like AWS or GCP which have a good option for it ?
You can use any stack for frontend with any framework for backend in Python (e.g. Flask or FastAPI), Django is a great option too (even without experience, you can quickly create a project)!
But I guess whatever option you choose, the part of "code simulation" that takes a lot of time should work separately. For example, you can use something like Celery to queue the tasks. All in all, the part of simulating should be independent from backend and maybe even written in another language, it should only take tasks and return results. You can even host it on another more powerful machine.
What about hosting, I'm not an expert, but I think it depends on the budget, and maybe country. For hosting backend and frontend I think Heroku is a good option, but I'm not sure is it suitable for hosting simulation related part. You can consider options like Google Cloud, Digital Ocean or maybe you can find cheaper VPS providers.
I asked this (voted to be too broad) Question while working my way through a starter book on node. Reading this book, I'm sure I'll learn the answer to this later, but I'd be more comfortable if I knew this up front:
My Question: Can I (efficiently) continue using a usual webhost such as iPage or GoDaddy to host my web application, building and hosting the front end in a simple, traditional manner through an Apache web server, and communicate with a separate Node.js server (my application back-end) via AJAX for queries and other things that I can more efficiently process via Node?
More specifically, would this be a bad programming practice in terms of efficiency and organization? In other words, would it be likely that a large scale commercial application would ever be handled via this method?
Yes, you can separate the front-end of your web application and the APIs that power it. In fact, this is a very common configuration, especially for "large scale commercial applications".
Where you draw the separation line between the two specifically depends on what you are doing, how you're doing it, and what your needs are. Also, in terms of hosting, remember that if you're accessing something server-side across the internet, you're adding extra latency to everything. Consider getting off Go Daddy and using another host that gives you more flexibility, such as a VPS provider.
It's ok. Actually, this is how things shoud be done. You have a backend API on a separate server and lots of apps which are using the API. Just go with Nginx server, check this Apache vs Nginx.
Yes you can use node js as a part of some big application. It depends on wich type of interaction you would like to get. Is it comfortable to you to mix technologies? Then node is pretty good thing to work over web. I've finished a part of big nodejs-ruby-php-c-flash application (my part was nodejs) for very large data mounts. This application have different levels of interaction. Sometimes I use 2 languages at one time to create each part of my application the best for task I'm working on. There is applications that initiate, run and destroy mutiple OS instances. So using of multi environmental application not so hard.
While experimenting with some data indexing using node.js objects (arrays, maps...) that takes some time to populate (from DB data) at every startup of the script, I wished my node.js objects could be automatically and transparently persisted in my database.
Having used MongoDB from node.js as well as other databases (including SQL-based) for some time now, I'm quite aware that the query/update mechanisms are very different between javascript objects (synchronous access, no queries) and database records (asynchronous access, queries, ...). However, I'm still hoping that a solution to make a javascript var persisted, at least for indices, can exist and be helpful.
Basically, I'm thinking of something like HTML5's LocalStorage, but for node.js servers.
Do you think this idea is interesting, feasible, or maybe it already exists?
EDIT: Work in progress: https://github.com/adrienjoly/persistent-harmony
A first thing to make clear, is that databases serve two purposes: persistence and convenient/efficient querying.
If you only need the first because you absolutely know up front that no other program is going to access your persistent data, you could take a look at the literature on orthogonal persistence, which is exactly the concept that you're suggesting here. An example is the KEN protocol that was successfully implemented in the WaterKen Java server.
There is some work to integrate the protocol into Google's V8 JavaScript runtime, which could lead to Nodeken, a Node.js with orthogonal persistence.
One of the difficulties of getting orthogonal persistence right is to map transactional semantics to a f.e. object-oriented programming system. The approach taken by V8-ken is to treat a single event loop execution of your JavaScript runtime as a transaction. In other words, the state of the virtual machine is persisted at the end of each "turn" in response of some event (incoming web request, server reply, user interface event, all asynchronous operations (IO), etc.). This however requires a modified runtime such as V8-ken, but evolutions in ECMAScript, such as proxies, look promising to be able to implement such features more conveniently.
In many cases (think web applications) though, persistent data needs to be accessible by different programs, requiring a "real" database for data to be easily exported, migrated, queried, etc.
Hybrid approaches could of course be possible...
%> npm search persistent storage
closet JSON persistent storage with methods chainability and callbacks for asynchronous use. =ganglio 2013-01-29 18:41 0.0.7 json persistent storag
ewdDOM Persistent lightweight DOM using Mumps Global Storage =robtweed 2013-02-02 14:39 0.0.4
fs-persistent-object Tiny Node library for persisting small runtime objects on filesystem =oleksiyk 2013-04-09 09:13 0.0.1 persistent tiny storage
headstorage A persistent storage for Node.js =headhsu2568 2012-11-20 13:41 0.0.0 storage
level-store A streaming storage engine based on LevelDB. =juliangruber 2013-06-21 19:55 3.3.2 leveldb levelup stream persistent
node-persist Super-easy (and fast) persistent data structures in Node.js, modeled after HTML5 localStorage =benmonro 2013-04-09 17:33 0.0.1 node persist
persistent-hash-trie Pure string:val storage, using structural sharing =hughfdjackson 2013-05-24 19:24 0.4.1 persistent hash trie pure functional d
perstore Perstore is a cross-platform JavaScript object store interface for mapping persistent objects to various different storage mediums using an in
shelf.js A modular, powerful wrapper library for persistent objects in the browser and Node.js =shakty 2013-05-24 08:10 0.4.7 persistance localStorag
stay Persistent scuttlebutt instances for browser and node =juliangruber 2012-12-11 21:54 0.1.0 persistent scuttlebutt persistence loc
Looks like the closest match would be node-persist
=)
EDIT: Here may be a better alternative solution...
#adrienjoly You know prototyping is still fairly high level and may not be (in the long run) as efficient as you are thinking.
You may be better off developing a module in C/C++ exposing a high level API for node.js to take advantage of.
I think I have a post about getting your feet wet with this type of node.js development (it stemmed from an original tutorial I followed here)
I do believe that method is however outdated and a newer method is to use the node-gyp tool. Some additional resources and examples: node-gyp projects, uRSA (I have a small pull request with this one), bcrypt etc..
My assumption in this is that you may bind the module extension to a db api such as oracle or postgres etc., and by writing a low level module linking to a low level API while exposing a high level API for developers to implement the persistent configuration options with API calls for slicing, indices, etc the performance would be optimal vs. trying to have node.js interpret your prototyping shim
Maybe this is what you're looking for?
https://github.com/yangli1990/flydb
or
npm install flydb
Now in javascript
var flydb = require('flydb');
flydb.test = "hello world"; //flydb.test will now persist
To run this
node --harmony-proxies <your commands>
e.g
node --harmony-proxies app
I come from a "traditional web application" background: think Java, .NET, PHP, ColdFusion, etc.
In evaluating NodeJS for use as the primary server-side technology for non-trivial applications, I'm wondering what complexities, problems, challenges a team of developers and operations folks might expect to face which are unique to NodeJS. In short, I'd like to reduce my unknowns. Some (not all) examples:
How well does it lend itself to large-team development? What unique challenges exist, for Node, in a team of 20 or 50 or 200 developers?
What unique challenges exist with respect to Database access? "Enterprisey" data access concerns are handled mostly trivially in Java (connection pools, security, etc via Spring). Is this the case with Node?
Reporting-heavy applications often require Excel, PDF, even PNG export... How does Node fare in this type of application?
Does Node present any unique challenges with respect to development-time debugging?
What unique operations challenges exist? Everything from server restarts / hot code swap / load balancing to tools for monitoring and managing a production cluster.
And so forth. What lessons exist for developing, maintaining, and production-managing a 100+K LoC codebase, deployed across a farm of servers, touched by dozens of developers?
I'll try to answer some of your questions
How well does it lend itself to large-team development? What unique challenges exist, for Node, in a team of 20 or 50 or 200 developers?
It does the same, as every other language does; nothing! Teams of programmers normally use versioning systems such as git, svn, mercurial etc. to deal with co-workers working on the same files
What unique challenges exist with respect to Database access? "Enterprisey" data access concerns are handled mostly trivially in Java (connection pools, security, etc via Spring). Is this the case with Node?
Node is Database agnostic. You can use any database with node where drivers/wrappers exist for it (same as with PHP). There are many drivers/wrappers for relational Databases (MySQL, SQLite) and NoSQL (MongoDB)
Reporting-heavy applications often require Excel, PDF, even PNG export... How does Node fare in this type of application?
Node can, as php and others do, access the normal shell. So if you handle an Image with php, you are using a wrapper for ImageMagick or GD. GD needs to be installed on the WebServer for it to work, because php sends the commands down to the command line. Same with node; find a wrapper (http://search.npmjs.org) and use the feature you desire.
Does Node present any unique challenges with respect to development-time debugging?
Node is Javascript, so it doesn't compile, but does JIT. So a failure will only be detected as soon as it is executed. You'd want a local env for every developer next to your staging/dev machine and live servers so they can test the code prior to commiting it to the staging server
What unique operations challenges exist? Everything from server restarts / hot code swap / load balancing to tools for monitoring and managing a production cluster.
No "Unique" challanges I am aware of. You'll deal with monitoring/heartbeats as with all other languages (yeah, there are languages that do this, like Erlang). You'll need a service like forever or supervisord to startup node on server restarts, but thats the same with Apache+PHP/Perl.
(Node has a Cluster module which helps you handle multiple workers
(for multi-core servers))
look at Git for managing your code. And choose the language based on what you want to be doing (high concurrency, scalability)
I will comment on the things for which I am qualified:
Connection Pooling to data sources.
The standard Node HTTP(S) server tools implement pooling by default, and there are knobs that you can configure to improve or control performance. The community being very active, there are lots of other projects (like poolee) that implement either general-purpose or specialized connection pooling. You will need to look around. In fact, given your traditional Webdev background...
1.1 Sidenote: Third-party libraries
When developing with Node, you may find yourself using a lot of third-party libraries. depending on your particular background, this may seem strange. To understand why, consider .NET or Java: the standard libraries for those platforms are gargantuan. As a result, when using those platforms you might pick very few third-party tools or plugins to help you get your work done. Node, by comparison, has an intentionally narrow and strictly controlled standard library. All the necessary APIs to unify "the way things are written" in the interest of server performance are there, but no more.
To help manage third-party libraries, the npm package manager was designed and included very early with node. It's known to be of high quality. The authors clearly anticipated a lot of third-party use.
Compute Power
You mentioned image export. Javascript libraries for all of these things exist, so as far as "it can be done easily", it can. Keep in mind that if you have a compute-heavy task, Javascript may not be the most effective language to implement the core computation. The v8 engine allows you to write modules in C, if you need to, but forwarding a request to a specialized backend server is the thing that Node does extremely well.
Operations Challenges
Node.js will not scale up to your hardware. If your server has multiple cores (which by now it most certainly does), you will need to run multiple server processes on the same physical hardware to achieve high utilization. This may make a different picture for operations: lots more processes than you would normally see, those processes groupable by physical or virtual machine.
Breaking your server into processes is not a bad thing for reliability, by the way: One large traditional process will crash on an unhandled exception, taking down with it eight (or whatever) cores worth of active sessions. Multiple Node processes will still crash individually, but with a proportionally smaller impact. There's probably room to explore the point, "how many processes is too many?", taking into account both performance and monitorability. It may actually be worth it to use more Node processes per server than there are available cores.
I looking at using Javascript server side and took a look at persevere/pintura but for a novice like me, I need more documentation and tutorials and CouchDB seems to have a lot of that but is it used as a server side js tool?
Kind Regards
Yes (seconding what #Pointy said above). The JavaScript in CouchDB does run server-side. :)
Beyond that, though, JavaScript in CouchDB can do far more than just map/reduce queries. It's also used for document validation, update handling (for processing XML POSTed to CouchDB for example), as well as HTML or other markup output from either a single JSON document (using _show) or from the results of a map/reduce query (using _list).
CouchDB does not, however, offer filesystem access or TCP/IP listeners as Node.js does. Right now, the JavaScript is focused on the "shipping and handling" for the database contents (things like validation, formatting, and querying).
If you're interested in looking into options for handling things that CouchDB doesn't do "internally" (image processing, sending e-mails, etc), you might checkout my reply about CouchApp architecture options.
CouchApp's are definitely a viable option for web apps. Coupling those with replication, it's hard to beat.
http://nodejs.org/ is a great example for server-side javascript. If you don't want to write your complete webapp on NodeJS you can trigger it via the node command.
CouchDB is a database with an interresting feature, it allows you to interact via a RestFull api making it ideally suited to access over the web, either in javascript or other client environments. If you are looking for a server-side javascript development environment, try Servoy. It allows full featured server-side application development in javascript including easy database access to all major databases.
Short Answer: Yes.
Longer Answer:
There is a Javascript Engine shipped with CouchDB, which is SpiderMonkey
(the very first engine, open source, written by Brendan Eich, creator of javascript)
CouchDB's View function is JavaScript function running server side that acts as the Map half of a map/reduce operation to construct view.
Then there is client side javascript API, which can be used to communicate with the REST layer to fetch/update documents, which is also where jQuery is used by CouchDB.
More on CouchDB's both server-side & client side javascript capability:
The default engine is capable of doing many features, while package availability is limited compared to Node.js. Kan.so can be considered "NPM for CouchDB, where some packages are server side.
For example, there are frameworks like duality that try to make use of both server side & client side javascript capability to reuse code for rendering with couchDB.
More on the engine:
(The selection of spider Monkey over v8 is because for couchDB's use case the map reduce is not v8 designed for. Also include packaging, multi-threading and also historical reason v8 did not exist that time when spiderMoney is chosen) see [the mailing list] and tweet2
There is even project to port the whole CouchDB to Node.js (i.e. instead of Erlang), Chesterfield
I am glad if someone can add how Erlang talk with the javscript engine.
Thanks for everyone's answer, this is an important thing to make clear for people like me messing up couchDB with some Node.js packages
No, CouchDB is a document-oriented database that can be queried and indexed in a MapReduce fashion using JavaScript. CouchDB also offers incremental replication with bi-directional conflict detection and resolution.
http://couchdb.apache.org/
CommonJS can be used for server side JavaScript, see here: http://www.commonjs.org/
NodeJS is another: http://nodejs.org/