Develop Reference

Is Node.js suitable for my project given how I'm using it?

I'm about to start writing an application and I want to include newer technologies to improve my knowledge and gain a skill set.
I'm using mysql to store user data (no choice), Angular.js to display/template content on the dom, and I wanted to use node.js to return results from the database in json format to be used by Angular.js.
My assumption was I could use node.js (along with the node-mysql module and maybe express)to accomplish my database queries that feed into my Angular.js templates.
Additionally, the application will live on a different server than the database itself and may reside in a mobile framework.
Am I way off base when it come to how to use node.js? Am I just using the wrong tool? Am I on the right track?
Any help, explanation, and points in the right direction would be great. Most of the info I've seen are copy/paste from the node.js site/wiki and don't really answer my question.

This is largely an opinion-based question and likely to be closed, but you're not way off base. What you want to do is absolutely (and commonly) done using Node.js.
As a complete aside, you might like to check out Platform-as-a-Service providers such as Heroku.com and Nodejitsu.com; both offer great and easy Node.js hosting and addons (such as NoSQL and RDBMS data stores) which have free tiers.
For that matter, MongoLab and MongoHQ both offer free tiers independent of Heroku, which you could use from your MySQL-only hosting provider if you like.

First bigger javascript / jquery website nearly finished - what to do before going live?

I just finished my first single site application project, most lines of code are in JS / Jquery with a backend (called with Ajax ofc) of Asp.net / C#.Net and a MSSQL Server.
I have around 10000 lines of code in total, 8K of that iin JS / JQuery.
I am going live this week, so what is to do to clean up such a big single site app?
What I have done:
comment methods / classes
make hard and complex code more readable
explain unclear code with comments
delete up out-commented code (source safe will save it for later then)
Is there something in geenreal I miss here?
Additional to your ideas, I have a idea to do, but I don't know how... In C#/Asp.net my Visual Studio gives me a warning or an error for syntax check, variable checks (e.g. unused variables, double used variables etc.) - is there a tool or a way to do this for my js files?
Recommend every help, thanks :)

None of this is specific to javascript or ASP.NET but for web applications in general. Basically the first time you expose your applications to real users, you have:
silly users, who don't do things the way you want them to.
naughty users who try to bring your site down or steal/destroy your data
lots of users, who may be individually good, but collectively overload your site and expose performance issues
So to deal with this you need to:
Test that everything works. Ideally you have automated end-to-end tests but if not, test manually. Note your tests, so you can automate them for next time.
Run through your application with erratic users who enter incorrect stuff, change workflow, etc. Does anything break?
Security test. Can I login as someone else? SQL injection? XSS? etc. If you deal directly with money or credit card numbers, I would get a professional penetration test done.
Performance and load test. Use a tool like gatling or jmeter, or something equivalent on the .NET side and hit your commonly used server functions with N users at once. N could be 10, 100, 1000000, depending on your requirements.
Once the application is released, monitor your logs carefully and fire fight :)

Here are few things you may check(in case you have'nt already)
Minify all your 8000 lines of jquery code and use the minified version in your production site.
https://developer.yahoo.com/performance/rules.html
Ensure you set the compilation node debug property to false. (Something developers tend to forget)
<compilation debug="false"/>
Ensure compression is turned on
Remove the unnecessary headers. You may refer to this link
Ensure your connection settings are encrypted from config files
Perform DB maintenance activities : rebuild indexes, statistics

Best practices of building a website using Node.js

This question's answers are a community effort. Edit existing answers to improve this post. It is not currently accepting new answers or interactions.
I was wondering how one would go about developing a website from scratch with Node.js.
I understand how I could possibly do it, but I am interested in the best design practice.
I need this theoretical website to:
Do a lot of AJAX
Be very straightforward
Be relatively small
Connect to... let's say a MySQL server
In PHP, building a pretty small website was very straightforward - I set up PHP on Apache and a MySQL server and then do something like:
includes/db/ which has connect.php for connecting to the db, a file with common db related functions and so on
includes/layout/ which had stuff like footer.php, header.php, and other layout related stuff
includes/users/ to handle user related actions
Then PHP just let you build pages and include these files together to form a website - I could go something like:
<?php
require_once('inclues/users/user_session.php');
require_once('inclues/db/connect.php');
require_once('inclues/design/header.php')
?>
// Other php or html or related content relating to the page
<?php
require_once('inclues/.../footer.php');
?>
I was wondering what might be similar in Node.js - I am looking for a way to accomplish this which is as simple, fast and straightforward as possible.
If the answer is not simple, I would love a book recommendation, I don't mind reading.
I love event based programming, I really love JavaScript's abilities and I'm really excited about Node.js. I want to learn how to develop this sort of stuff with it the right way from the start.

To start with the bad news: As Node.js is a pretty young technique, I think you'll find that the proces of creating a full fledged website and maintaining/operating it will be very different than what you're currently used to.
Josh3736 adds: Once you figure out how Node.js and its various packages (Connect, Express) work, I found that you can develop new sites very quickly.
The rough edges that currently exist in Node.js, combined with the fast pace of its development and all modules involved can complicate things though, and make things less simple, fast and straightforward than you'd like.
Having that out of the way, here's the good news:
The Node Package Manager, NPM has a lot of good tools and frameworks to expand Node.js's bare bones functionality, making it suitable to create a webserver.
Most notably would be the Express Framework which contains almost everything you need to run a webserver (including cookies, sessions and path routing). Additionally Express supports partials, which take care of your header and footer includes.
Express is built on top of Sencha's Connect. Cookies and sessions are actually powered by Connect. Express is what simplifies your routing and handles views/partials. So if you don't need all bells and whistles that come with Express you could just go for Connect instead.
If you like to use templates for these partials, the Jade Template Engine can speed things up for you. Though Josh3736 points out that Jade is slow and whitespace-significant. A more complete overview can be found here, which includes his favourite, doT. (I personally use Node.js for socket.io based applications only, so he's a better source than me when it comes to templating).
You can connect to MySQL from Node.js using the db-mysql module, but if you don't need that because you're accessing data connected to an already present system, I'd advise to use a more... 'modern' approach, which is to use a NoSQL database as most Node.js projects seem to do. MongoDB via Mongoose is the popular way to go.
Or if it's just storing objects you're interested in, just go for Redis instead (which you're probably going to need at some point anyway).
Once your website is complete, you'll have to deploy it and make sure it keeps running. There are many ways to do so, like using built-in cluster support or use the more feature-friendly forever npm module. See this SO question of mine for more information.
Conclusion:
What I'm trying to get at is this:
Asking what the best practice for building a website in Node.js is, is about the same as asking what the best way to build a website in PHP is: 100 developers will give you 100 different answers.
NPM is blessed with a variety of excellent frameworks that greatly simplify a lot of tasks involved, but it's all based on preference which one is the way to go really.
As I've said, Node.js is still a pretty young technique, so none of the frameworks or additional tools have emerged as 'defacto standard' yet; for most things you're trying to do there are probably various alternatives, and expect your code to break when using most of them during updates, because development of Node.js itself and most modules is fast paced. You'll have to keep up.
Putting it all together:
As I've said, my main production use for Node.js is to be able to use socket.io, so I don't have any good production examples present (And as I'm about to leave on a well-deserved vacation I don't have the time to put one together either). There are some good examples though:
Setup and deployment using Express and Jade
A very complete blog example using Express, Jade and MongoDB
Combining Restify (an extension of Express), Backbone.js and Mongoose
Again, the way to go (and subsequently the example to follow) depends greatly on your ultimate goals and the techniques chosen, but luckily there are plenty of resources available for all of the choices available. Most modules use well documented GitHub repositories and include examples in combination with the most popular modules (See the /examples/ dir that seems to be present in most repositories).
(thanks to Josh3736 for rectifying my errors.)

Is it okay to use plaintext files for authentication purposes?

I tried to implement a simple login system with Mongodb and Node.js, but I found it to be too much of a hassle (everyone is building things on a hundred of frameworks and not explaining half of the code). I probably could do it, but it would require too much work and so on, as I only want to make a simple multiplayer game.
My alternative to databases are plaintext files. I do plan to hash the passwords, of course. The first problem that came to my mind was that the file could potentially get too big, and that the server would slow down completely trying to read it (in a hypothetical situations, I doubt my game will have that many users). The solution would be to create a users/ folder on the server, and split the big file into many smaller ones. Their names would correspond with the usernames, and there hashed passwords and emails would be held. This solves the speed issue, and I can't see how I still need databases!
Am I missing something? Are there other advantages to using databases? Also, I would be very pleased if someone answered with a login Mongodb example, that would solve all my problems. Just make it as simple as possible.

There are a ton of reasons to use a real database instead of some hacked, half-baked file implementation. Some big ones:
ACID (this is really 4 reasons)
Performance & stability (databases are some of the most mature pieces of software still in use)
Standardization
If you want "lightweight," I'd recommend that you start with SQLite.
Related:
Why use SQL database?
Why use MySQL over flatfiles?

Let's say you have an authentication information as: UserName(10), Password(32). These makes 42 characters. Let's say each character is 4 bytes (consisting all characters in the universe), each user consumes 42 * 4 = 168 bytes. Even if you have 100.000 users, it would only require 16 MBs of memory. It can fit the whole RAM, you only need to wirte changes to disk, and read them all in the boot of the program. Of course it would have a litle overhead, but not a big deal.
However, handling files may become an issue, you do not have to re invent the wheel. This is what databases are for. I agree, using Mongo, MySQL can be a overkill sometiems, however If you only plan to use database for authentication, you may go with more lightweight solution, you can use SQLite, which has only a file, a ~270KB client and a nice NodeJS library. Have a look at this nice post

If you are working with mongo I would strongly recommend working with the Mongoose library. It provides an excellent ORM for Mongo. If you do end up deciding to work with that ORM there is an established authentication library called mongoose-auth. I used the default password example for my own site.

If you are using NodeJS mongoDB would work well for what you want, there is also the option of redis,couch etc... MongoDB is as good a place as any to start and f you have been playing with it a little might as well stick with it.
Have a look to this repo which using a cloudfoundry or Mongolab (both free) mongoDb server.
https://github.com/daithiw44/Node_Authenticated_MongoDB
The demo has a button which when clicks hits the db to get a value. Substitute the hardcoded value for values in username and password textfields and one could authenticated a user and set up an expressjs session.
This was written with an older version of node but a tweak to the package.json should give you a good place to get started.
Hope that helps.

Reflective Web Application (WebIDE)

Preamble
So, this question has already been answered, but as it was my first question for this project, I'm going to continue to reference it in other questions I ask for this project.
For anyone who came from another question, here is the basic idea: Create a web app that can make it much easier to create other web applications or websites. To do this, you would basically create a modular site with "widgets" and then combine them into the final display pages. Each widget would likely have its own set of functions combined in a Class if you use Prototype or .prototype.fn otherwise.
Currently
I am working on getting the basics down: editing CSS, creating user JavaScript functions and dynamically finding their names/inputs, and other critical technical aspects of the project. Soon I will create a rough timeline of the features I wish to create. Soon after I do this, I intent to create a Blog of sorts to keep everyone informed of the project's status.
Original Question
Hello all, I am currently trying to formalize an idea I have for a personal project (which may turn into a professional one later on). The concept is a reflective web application. In other words, a web application that can build other web applications and is actively used to build and improve itself. Think of it as sort of a webapp IDE for creating webapps.
So before I start explaining it further, my question to all of you is this: What do you think would be some of the hardest challenges along the way and where would be the best place to start?
Now let me try to explain some of the aspects of this concept briefly here. I want this application to be as close to a WYSIWYG as possible, in that you have a display area which shows all or part of the website as it would appear. You should be free to browse it to get to the areas you want to work on and use a JavaScript debugger/console to ask "what would happen if...?" questions.
I intend for the webapps to be built up via components. In other words, the result would be a very modular webapp so that you can tweak things on a small or large scale with a fair amount of ease (generally it should be better than hand coding everything in <insert editor of choice>).
Once the website/webapp is done, this webapp should be able to produce all the code necessary to install and run the created website/webapp (so CSS, JavaScript, PHP, and PHP installer for the database).
Here are the few major challenges I've come up with so far:
Changing CSS on the fly
Implementing reflection in JavaScript
Accurate and brief DOM tree viewer
Allowing users to choose JavaScript libraries (i.e. Prototype, jQuery, Dojo, extJS, etc.)
Any other comments and suggestions are also welcome.
Edit 1: I really like the idea of AppJet and I will check it out in detail when I get the time this weekend. However, my only concern is that this is supposed to create code that can go onto others webservers, so while AppJet might be a great way for me to develop this app more rapidly, I still think I will have to generate PHP code for my users to put on their servers.
Also, when I feel this is ready for beta testers, I will certainly release it for free for everyone on this site. But I was thinking that out of beta I should follow a scheme similar to that of git: Free for open source apps, costs money for private/proprietary apps.

Conceptually, you would be building widgets, a widget factory, and a factory making factory.
So, you would have to find all the different types of interactions that could be possible in making a widget, between widgets, within a factory, and between multiple widget making factories to get an idea.
Something to keep on top of how far would be too far to abstract?
**I think you would need to be able to abstract a few layers completely for the application space itself. Then you'd have to build some management tool for it all. **
- Presentation, Workflow and the Data tier.
Presentation: You are either receiving feedback, or putting in input. Usually as a result of clicking, or entering something. A simple example is making dynamic web forms in a database. What would you have to store in a database about where it comes/goes from? This would probably make up the presentation layer. This would probably be the best exercise to start with to get a feel for what you may need to go with.
Workflow: it would be wise to build a simple workflow engine. I built one modeled on Windows Workflow that I had up and running in 2 days. It could set the initial event that should be run, etc. From a designer perspective, I would imagine a visio type program to link these events. The events in the workflow would then drive the presentation tier.
Data: You would have to store the data about the application as much as the data in the application. So, form, event, data structures could possibly be done by storing xml docs depending on whether you need to work with any of the data in the forms or not. The data of the application could also be stored in empty xml templates that you fill in, or in actual tables. At that point you'd have to create a table creation routine that would maintain a table for an app to the spec. Google has something like this with their google DB online.
Hope that helps. Share what you end up coming up with.

Why use PHP?
Appjet does something really similar using 100% Javascript on the client and server side with rhino.
This makes it easier for programmers to use your service, and easier for you to deploy. In fact even their data storage technique uses Javascript (simple native objects), which is a really powerful idea.