Develop Reference

How to implement firebase server side security

I'm currently working on a new google polymer web application and wondered if I should use firebase as the backend/db. I took a look at the project, made some test applications and really liked it! But to fully convince me, that firebase is the way to go I need the following questions answered:
I'm a little bit concerned about security: So, I know, that firebase uses read, write and validate to implement server side security. From the samples, I noticed that the validation basically is a one-line JS script, that represents a 'if'. As I'm planning to build a web e-commerce application I need to validate quite some inputs. Is there a possibility, to outsource the validation in a separate file, to make it more readable? Also I wondered, if there is a possibility, to test these server side validations, with for example unit tests?
I'm not 100% sure at the moment, that firebase can cover all of our use cases. Would it be possible/a good solution to use a "normal" backend for some critical functions and then persist the data from the backend in firebase?
I saw some nice polymer elements for firebase. Is firebase 100% supported in polymer/web components?
Is there an other way (like Java approach) to implement server business logic?
Is there a way, to define update scripts, so that new releases can easily be pushed to production?
Thanks & kind regards
Marc

So, I asked the firebase supprt and got the following answer:
Great to meet you.
I'm a little bit concerned about security: So, I know, that firebase uses read, write and validate to implement server side security. From the samples, I noticed that the validation basically is a one-line JS script, that represents a 'if'. As I'm planning to build a web e-commerce application I need to validate quite some inputs. Is there a possibility, to outsource the validation in a separate file, to make it more readable? Also I wondered, if there is a possibility, to test these server side validations, with for example unit tests?
You can implement extremely complex and effective rules using our security rules language. You can deploy security rules as part of your hosting deploy process, or via the REST API. It's not possible to break the contents into multiple files on the server, but you could certainly build your own process for merging multiple files into a single JSON result.
I'm not 100% sure at the moment, that firebase can cover all of our use cases. Would it be possible/a good solution to use a "normal" backend for some critical functions and then persist the data from the backend in firebase?
Generally speaking, synchronizing Firebase and a SQL back end is not very practical and they don't translate well. It's probably entirely redundant as well.
I saw some nice polymer elements for firebase. Is firebase 100% supported in polymer/web components?
I don't know what 100% supported means in this context. We offer a JavaScript SDK so they should play fine together.
Is there an other way (like Java approach) to implement server business logic?
We offer official SDKs in Java, Objective-C/Swift, Android, Node.js, JavaScript, and a REST API for use with other languages.
Is there a way, to define update scripts, so that new releases can easily be pushed to production?
I'm not sure what this means. Most likely the answer is no, since we don't provide a build process or any tools to release your software.
I hope that helps!
I responded:
Thank you for the information, it helped me very much! After reading your response on question number 5 one further question popped into my mind:
…
5. Is there a way, to define update scripts, so that new releases can easily be pushed to production?
I'm not sure what this means. Most likely the answer is no, since we don't provide a build process or any tools to release your software.
Is there like a best practice on how to handle the database schema? I only have one web application (without apps, etc.) in my case... I expect, that the database will change drastically over time and releases. Should I write JS logic, that checks the current database version and update it, if it's necessary? Maybe this would make a nice feature...
For example: I deployed Version 1.0 of my application and everything works fine. After 3 months of programming I notice, that the user data needs a further attribute: address, which is a 'not null' attribute. I now deploy Version 2.0 of my application and every new registered user has a address, but the old users (from Version 1.0) do not have this field or a value.
How should I handle this?
Support responded:
Hi Marc,
There’s no best practice here, but your ideas seem fairly sound. You probably don’t need to check in your JavaScript. You can probably store a version number in the user’s profiles, and when they upgrade to the latest software, you can upgrade that in their profile data.
Then your validation rules could use something like the following:
{
"user": {
".write": "newData.hasChild('address') || newData.child('appVersion') < 4",
"address": {
".validate": "newData.isString() && newData.val().length < 1000"
}
}
}
So if you are concerned about versioning, this could be used to deal with legacy releases.
Another popular approach I’ve seen from devs is to do intermediate upgrades by duplicating data. Thus, you release an intermediate version that writes to the old path and to the new path with the updated data structure (which keeps the app working for old users till they upgrade). Once a reasonable percent of clients are upgraded, then release a final version that no longer does a dual write to the old structure and newer structure.
Of course, flattening data, while it makes joining and fetching data bit more of a pain, will make upgrades much easier as the modular data structure adapts more easily to changes. And, naturally, a pragmatic design where you wrap the various records in a class (e.g. the UserProfile class with getter/setter methods) makes transitions simpler as you can easily hack in versioning at one place.
Hope this helps someone :)

On-Disk Text Processing With Javascript

I have some html files that I need to do automated processing on, basically regex replaces, but also some more complex actions like copying select blocks of text from one file to another.
I want to create a series of scripts that will let me do this processing (it will need to be done more than once on different batches of files). It would be trivial to use Go for this (read the file into memory, regex, save to disk) but I am the only member of the project that's familiar with Go.
Javascript is a tad more ubiquitous, and I do have project members who are familiar with the language, so it's a better fit in that respect. If I'm not around later, someone else could edit the scripts.
Is there a simple way to write some JS scripts to do on-disk text processing? I'm looking for a cross-platform solution (OSX, Windows). Ideally, once the scripts are written, they can be executed by double-clicking an icon--there will be "not computer people" involved at some point.
Also, I'd like to be able to do some kind of alert/message box to inform the user of the success/failure of the script. (This may be a tall order, and is of secondary importance.)
What I've looked at:
Node.js was the first thing that popped into my head, because I know that it has file system access tools, and obviously regex capacity. But I've never used Node before, and based on the tutorials I've read, it seems like overkill for something this simple.
There's a whole slew of "javascript compiling" tools that you can find by googling around. Some are not cross-platform, some seem old or not actively maintained, etc. None of them caught my eye as easy to pick up and just write some JS scripts with.
Any thoughts?

Node.js is a simple solution and with it's framework you can create or later modify your script to your needs. This way you will not be locked down by someone else's code. And it is not that difficult to to use.
Here is a quick tutorial on accesing files using node.js
http://www.sitepoint.com/accessing-the-file-system-in-node-js/
And here is a quick tutorial on using a node module called Cheerio. It allow you to access html files using "jquery like syntax". You don't need to use regex.
http://maxogden.com/scraping-with-node.html
I worked on a project for a client once and it required parsing thru hundreds of html files to check and replace certain image files based on certain criterias. I wasn't familiar with node at the time so I read some tutorials and wrote the script in about an hour.
And as long as Nodejs' path is set, you can run it on the command line.

Some tips:
You need any kind of DOM HTML parser, not only JS nor specifically JS.
You can do that thing with Java with use of jTidy or jSoup libraries (I've used second one few times). It's pretty simple language to learn if you know JS and IDE like Netbeans helps a lot. So can be made quickly with that.
You can use PhantomJS to create some job files and create shell/batch code to run them on some files. You might need to write a generator for job files (like taking a list of files, creating job files for each and running them).
You can use Node.js which isn't much overkill, I'm sure any solution won't be trivial.
You can create an ETL for processing with for example Pentaho ETL (which has JS embedded as one of two scripting languages... but without DOM parser - for that one you would need to use a bit of Java there and some library in way similar to this article).
You can also do that with PHP with Simple HTML DOM Parser - so you can make a service online (or on local server) that takes those html files and throws out processed ones.

First I think you underestimate the complexity. The statement
"It would be trivial to use Go for this (read the file into memory,
regex, save to disk) but I am the only member of the project that's
familiar with Go."
is probably false. Parsing HTML with RegExp is just a bad idea. (Google it and you will see why)
Second, if you can trivially write the code using RegExps in Go, you can just as easily write the same thing in Javascript. They both support RegExp and file operations. If you are unsure about the Javascript/Node.js details, I suggest writing the trivial solution in Go and then translate the thing into Javascript with a colleague.
Since Javascript is a script language, writing command line utilities in Node.js is straight forward.
Some pointers to get you started
RegExp in Javascript
Building command line apps in Node.js

How to integrate Golang backend and Javascript (three.js) frontend?

I would like to write a 3D application using Golang, my favorite programming language. However, I would prefer not to use native OpenGL directly as the graphical frontend as this would entail a larger learning curve than I'm willing to tolerate. Additionally, I don't really want to use a Golang game engine like go:ngine.
After doing some research I found out about the amazing WebGL Javascript library three.js and I was so intrigued that I completed an introductory Javascript course in a few hours. I'm not really a web developer at all, so I'm wondering how practical it would be for me to write my application primarily in Go but with a three.js frontend.
Some specific questions:
Is it worthwhile / practical to use Javascript/three.js as a frontend to a Golang server like this?
If it is worthwhile, where can I look to learn how to integrate the two cleanly?
If it isn't really worthwhile, which alternatives do I have?
Thanks, any response appreciated.
Edit:
Do you plan to make operations on geometry in Go, possibly in realtime and communicate with your Javascript layer using Websockets?
Yes, this is what I'm thinking now. I'm intending my (Golang) program to generate streams of vertices and triangles based on a few parameters from the user. Each "structure" is generated all at once after the users supply their parameters (which I see as slider bars + input boxes on a Javascript frontend).
Here's an example of something the Go program might generate, plotted in GNUPlot:
(source: alexozer.com)
If this is the case you underestimate Javascript a bit. You should write most of your application in Javascript and use Go as a classical backend service layer like user accounts, persisting state, etc.
I'd be fine with doing that, except for these reasons:
The core generating process fundamentally depends on Goroutines and channels
I know close to nothing about web development, including frontend and backend
So I suppose I'm asking about the implementation details of one of these three possibilities:
Writing most of the program, including the generator, in JS, and a minimal backend in Go
Writing most of the program in Go and using JS just as a graphical frontend
Something else?

I recommend leaving out the whole Websocket aspect because it's only a possible optimization (and maybe not even that).
If you need to depend on Go's concurrency model then go for it, write your component in Go, then a Webserver in Go that takes parameters coming from an HTML request, uses them to compute the result and sends back the data in a JSON format.
On the frontend you will then only focus on sending this request when the user changed a parameter and for displaying of the JSON data you can use ThreeJS right away.
You'll still have to learn a bit of web development and Javascript though. But hey, they say the web is the future ;-)
I think it's the way to go for your case because your application operates in a quite strict request-result way.

The answer to your question depends a lot on where exactly you plan to draw the line between your Go component and the Javascript component. From the way you describe the problem I get the feeling that you want to write most of your application in Go and use JS only as a display layer? I wouldn't recommend doing that.
To rephrase my question: Do you plan to make operations on geometry in Go, possibly in realtime and communicate with your Javascript layer using Websockets? If this is the case you underestimate Javascript a bit. You should write most of your application in Javascript and use Go as a classical backend service layer like user accounts, persisting state, etc.
I might be able to give a more specific answer if I know what exactly you plan to do.

R & Javascript callbacks

I'm writing a UI to my R script, which asks the user some names of organisms and the location of a folder, using javascript/html that will be local (not hosted, ever).
At the moment, I have just that: a couple of text boxes that take input and pass an executable R script. Originally this UI was being written as a very user friendly option, but slowly I've realized that some nifty tricks can be added such as a textbox that completes the word for the user (so if the user misspells the name of the organism, the UI will correct the input based on the files uploaded. And this would come from a list of organisms text file that R would generate immediately once the files have been added).
Is there a way to make this more efficient? For example, retrieving plots from R (as .pngs) and updating my local webpage and being able to share a log file between R and the UI (mind you, I am aware of the potential File I/O errors)..but for the sake of brainstorming.
I'm aware of Shiny, but what I would like is a simple local UI, as I will be dealing with big data (average ~ 1 gigabyte worth of files that my script will process).
Another way to ask my question that is more to the point:
Here's an example of integrating PHP and R: http://www.r-bloggers.com/integrating-php-and-r/
I am looking to create something similar with javascript/css/html/jquery etc.
Thanks

You could definitely use nodejs (nodejs.org) for that. Take a look at https://github.com/elijah/r-node and r-node. Confusingly enough, this is two different projects with the same name. More info on the latter here: squirelove.net/r-node/doku.php
In recent years JavaScript has become one of the fastest programming languages. In one case I know of, JavaScript is faster than C++. See: benchmarksgame.alioth.debian.org/u32/performance.php?test=regexdna
Bear in mind, though, that memory is very difficult to manage in JavaScript, so you should run some sort of memory leak detection program on your code, if you plan to create long running processes.
E.I: memwatch (npmjs.org/package/memwatch) or nodeheap (npmjs.org/package/memwatch)
Good luck with your endeavors!
PS. sorry for the lack of real links. I'm apparently not allowed to post more than 2 links.

Why wouldn't you be able to use Shiny locally? You design your app on your computer and run it locally with runApp('myapp') from an R-prompt. Unless you are experienced with javascript I would give shiny another look: http://www.rstudio.com/shiny/
The example you linked to can be very easily implemented using Shiny. See link below for a tutorial on how to write the app:
http://rstudio.github.com/shiny/tutorial/#hello-shiny
To run that example locally:
install.packages('shiny')
shiny::runExample('01_hello')

I have a similar case, and shiny looked like a good idea to me. However, after I did a few first steps, I am no longer sure about this. Note that most of the examples use shiny to display results. When you get into editing some fields and using a database, things can become messy; the reactive-ness gets in the way once fields can be change by program and by the user.
As an example see https://gist.github.com/dmenne/4721235/edit. The main problem for the current state of shiny is that you must use the dynamic UI for this type of work, which kills any separation of ui and server because you have to create the ui elements in the server.
shiny is a great idea, but for anything larger with interaction it is too early now. Knowing that the amazing RStudio team is behind it, I am sure the stress should be on now.
What else is there around to make user interfaces for R? TclTk makes me shudder. I working in c# a lot, and I had been using R(D)COM for interfacing some years ago, but gave up after installation and licensing problems. There is R.DOTNet which works better now; it is the most hazzle-free installation-wise, but it is not a very active project, and tends to crash. Interfacing via RServe/RServeCLI is stable, but is too difficult to install on Windows, for example on hospital computers with their strict security issues.
And there is Qt. With the active RInside community, it would be a good choice and the interface is great. I wish however my programming skills were at the level of the RStudio-guys. The fact that even Dirk is one the proof-of-concept level (using rinside with qt in windows) is not encouraging.

How should I go about writing a node.js web application with both server and client side code?

I'm planning on writing a spine/backbone.js style web application which basically just transfers a large application.js file to the client's browser that communicates with the node.js backend using ajax. The problem is that I don't know how to structure such a project, since I've never seen examples of such an application. I can picture some pros and cons with different ways of doing this
Keep everything in one project folder. Both the server side and client side code resides in the same folders which means they can share resources such as form input validation and language files. This seems like a good solution, but I have no clue how I would bundle only the code that the client needs, and not the server code. Just in general I don't know how to accomplish this. If it has been done before, I would like to see some sample code, perhaps even a git repo
Create two separate projects. One for the client and one for the server. This seems a lot more simple and straight forward, but not as elegant when it comes to sharing resources. I would have to write code such as form input validation twice.
Any thoughts?

Your first situation is a very tricky scenario and I would suggest that we're not quite there yet. Some would argue that there's little reason to try to get there, as front/back ends will always be tasked with slightly and sometimes drastically different tasks. Libraries like derby show promise, but aren't quite there yet.
I discussed this recently with a friend and we came to the conclusion that perhaps the best bet for now would be to serialize models over websockets, and then ensure that the node server and client app stay in sync.
I may work on such a library, but for now I'm still developing with 2 folders and copies of models on both sides. Layout mark-up gets sent from the server, with all other content rendered client-side after receiving JSON from the server. Frankly, the amount of duplication isn't really that substantial. A little irritating but also maintains greater flexibility to grow in different directions.

This won't be a complete answer to your question, but one library that might help if you choose to pursue such an endeavour might be Browserify.
It's designed so you can use a similar require() function with a preprocessed, or on-the-fly generated from module source, js file containing many different modules. These modules can be shared with the server side through the same require() mechanism.
I don't know explicity of a Backbone implemented on the server side as a server side counter part for model sync, that would seem to be the first goal you are looking for, aloowing code that makes sense to be shared, such as models and validation, to be usefully shared.
Another thing to look at is requirejs, which uses more traditional script tag asynchronous loading f js modules, but also works within node.js aloowing the same AMD modules to a be shared between node and client code.

Was realtime required? Otherwise the Derby approach might be a little too heavy. Express.js proposes a structure where client js is separated in public folder, and provides methods to get a quick RESTful API running, which you can then access with your application.js.
I guess you could load "classic" js files from public into node via eval() too.

Things have moved much ahead now, and things like
browserify influenced coding can help us achieve this easily
there will always be some uncommon code between server and client sides, But the goal shall always be to keep all the logic code in different modules(which are later used from both environments). This is better from TDD point of view as well, also keeps your keyboard press count to lesser.
Have a look at things like this stack -
http://mindthecode.com/lets-build-an-angularjs-app-with-browserify-and-gulp/
Having said that your option1 did not seem that manageable to me, if you had the right coders coding the right code.