Develop Reference

Handling database objects in node js

In general how does javascript interpret a Database {} object? I am writing some back end scripts to handle a registration form verification. In particular I need to ensure that the username and email used to register is not currently in use. To do this I use the sqlite3 package and use a few db.get calls to determine if there are existing entries in my database for the username and email used on the registration form. I want to use the return of db.get to check if it is empty or not and use this conditional to perform the necessary task. However the db.get returns a Database {} object which I am unfamiliar of how to work with.
Hopefully the following pseudo describes the issue better. Here uname returns Database {} and so never fails the if statement.
function existance(username, email) {
let uname = db.get(sql, username, callback(err, throw));
if (uname) {
let errors = {username: 'Username already in use.'};
return errors;
}
};
EDIT
I have since used Gajiu's recommendation but still having issues. So I have two files:
registrant_existence.js
// necessary requirements and initialisation
function registrant_existence(username) {
let uname;
let sql = 'SELECT username FROM table WHERE username=?';
db.get(sql, username, function(err, row) {
if (err) {
throw err;
} else {
uname = row;
console.log(uname);
}
});
console.log(uname);
if (uname !== undefined) {
return {username: 'Username already in use.'};
} else {
return 'DNE';
}
};
module.exports = registrant_existence;
register.js
let registrant_existence = require("path to registrant_existence.js");
// necessary requirements and initialisation
router.post('/', function(req, res) {
let existence = registrant_existence(req.body.username, req.body.email);
if (existence != 'DNE') {
// render registration page notifying the user
// that the username is already in use
} else {
// register the new user details
}
});
The uname variable is undefined always. I placed the console.log(uname) in two spots in registrant_existence.js as is seen above to see what is happening. Two strange things occur.
The first is that the console.log(uname) outside the db.get() displays undefined in the console and the console.log(uname) inside the db.get() displays the expected string (a username I know is in my database).
The second is that the console.log(uname) outside the db.get() is displayed before the console.log(uname) _inside the db.get() in my console.
I have no idea why these things are happening. Does anyone have any suggestions?

You should try something like that:
db.get(sql, username, (err, data) => {
// process the data here
if (err) {
return console.error(err.message);
}
return data
? console.log(data.id, data.userName)
: console.log('No username found');
});

I guess you are looking for a wrapper around your Database object, something like an Object Relational Mapper (ORM), this one is used regularly https://sequelize.org/master/manual/getting-started.html
On the other hand for your specific use case you might want to get a look at this https://www.sqlite.org/lang_createtable.html#unique_constraints: unicity is usually via constraints in the data store.

sails.js node.js Parse JSON on controller

In my controller called MapController I'm doing a function to do a parse of remote json files, and from an if-else structure add some values in an array called "parsewebservice", apparently everything is working fine but console.log ( parsewebservice); is not returning the values that were passed to the array "parsewebservice" in the place where it is returning it empty. But when I put it inside the forEach it returns, but everything cluttered and repeated then is not the right way.
I wanted to know why the values that were passed to the array "parsewebservice" are not going along with the variable after populada and what would be the correct way to do it?
Here is my code below:
/**
* MapController
*
* #description :: Server-side logic for managing Maps
* #help :: See http://sailsjs.org/#!/documentation/concepts/Controllers
*/
module.exports = {
index: function(req, res, next) {
Data.find(function foundData(err, datas) {
if (err) return next(err);
var parsewebservice = [];
datas.forEach(function(data, index) {
var req = require("request");
var url = data.address + "?f=pjson";
req(url, function(err, res, retorno) {
if (err) {
console.log(err);
} else {
var camadas = JSON.parse(retorno);
if (camadas.mapName) {
camadas.layers.forEach(function(campo, i) {
if (campo.subLayerIds != null) {
} else if (campo.subLayerIds == null) {
parsewebservice.push([i, "dynamicMapLayer", campo.name, data.address]);
}
});
} else if (camadas.serviceDataType) {
parsewebservice.push([null, "imageMapLayer", camadas.name, data.address]);
} else if (camadas.type) {
parsewebservice.push([null, "featureLayer", camadas.name, data.address]);
}
}
});
});
console.log(parsewebservice);
});
},
};

My first comment has to be that you should not combine function(req, res) with var req = require('request')... you lose your access to the original req object!
So, you need to run a list of async tasks, and do something when they are all complete. That will never be entirely easy, and no matter what, you will have to get used to the idea that your code does not run from top to bottom as you've written it. Your console.log at the bottom runs before any of the callbacks (functions you pass in) you pass to your external requests.
The right way to do this is to use promises. It looks like you are using this request library, whose returned requests can only accept callbacks, not be returned as promises. You can create your own promise wrapper for them, or use an alternative library (several are recommended on the page).
I don't want to write a whole intro-to-promises right here, so what I will do is give you a less pretty, but maybe more understandable way to run some code at the completion of all your requests.
Data.find(function foundData(err, datas) {
if (err) return next(err);
var parsewebservice = [];
// here we will write some code that we will run once per returned data
var processResponse = function(resp) {
parsewebservice.push(resp);
if(parsewebservice.length >= datas.length) {
// we are done, that was the final request
console.log(parsewebservice);
return res.send({data: parsewebservice)}); // or whatever
}
};
datas.forEach(function(data, index) {
var request = require("request");
var url = data.address + "?f=pjson";
request(url, function(err, res, retorno) {
// do some processing of retorno...
// call our function to handle the result
processResponse(retorno);
});
});
console.log(parsewebservice); // still an empty array here
});

I solved the problem.
the "request" module is asynchronous so we need to wait for it to respond and then send the response to the view.
To do this we created a function called "foo" to contain the foreach and the request, we made a callback of that function and finally we made the response (res.view) within that function, so that the controller response would only be sent after the response of the "foo" function to the callback. So we were able to parse.json the data from the "data" collection using foreach and the "request" module and send the objects to the view.
Many thanks to all who have helped me, my sincere thanks.

Javascript function doesn't return query result

I am trying to figure out why one of my queries won't return the value from a query...my code looks like this:
var client = new pg.Client(conString);
client.connect();
var query = client.query("SELECT count(*) as count FROM sat_scores")
// Don't use demo key in production. Get a key from https://api.nasa.gov/index.html#apply-for-an-api-key
function getNEO(callback) {
var data = '';
query.on('rows', function(rows) {
console.log("Row count is: %s", rows[0].count)
data += rows[0].count;
});
query.on('end', function() {
callback(data);
});
}
with that, getNEO returns a blank...but if I set var data = '4', then getNEO returns 4....the query should return 128 but it just returns a blank...

First of all, getNEO() doesn't return anything - I'm operating on the assumption that you call getNEO() exactly once for your query, and pass in a callback to handle the data, and that callback is what's not getting the appropriate data?
My typical recommendation for troubleshooting things like this is to simplify your code, and try and get really close to any example code given (for instance):
var client = new pg.Client(conString);
// define your callback here, in theory
client.connect(function (err) {
if (err) throw err;
var query = client.query("SELECT count(*) as count FROM sat_scores"),
function(err, result) {
if (err) throw err;
console.log(result.rows.length);
}
);
});
... I'm doing a couple things here you'll want to note:
It looks like the client.connect() method is asynchronous - you can't just connect and then go run your query, you have to wait until the connection is completed, hence the callback. Looking through the code, it looks like it may emit a connect event when it's ready to send queries, so you don't have to use a callback on the connect() method directly.
I don't see a data event in the documentation for the query object nor do I see one in the code. You could use the row event, or you could use a callback directly on the query as in the example on the main page - that's what I've done here in the interest of simplicity.
I don't see the count property you're using, and row[0] is only going to be the first result - I think you want the length property on the whole rows array if you're looking for the number of rows returned.
I don't know if you have a good reason to use the getNEO() function as opposed to putting the code directly in procedurally, but I think you can get a closer approximation of what you're after like this:
var client = new pg.Client(conString);
// define your callback here, in theory
client.connect();
function getNEO(callback) {
client.on('connect', function () {
var query = client.query("SELECT count(*) as count FROM sat_scores"));
query.on('end', function(result) {
callback(result.rowCount);
});
});
}
... so, you can call your getNEO() function whenever you like, it'll appropriately wait for the connection to be completed, and then you can skip tracking each row as it comes; the end event receives the result object which will give you all the rows and the row count to do with what you wish.

so here is how I was able to resolve the issue....I moved the var query inside of the function
function getNEO(state, callback) {
var conString = "postgres://alexa:al#alexadb2.cgh3p2.us-east-1.redshift.amazonaws.com:5439/alexa";
var client = new pg.Client(conString);
client.connect();
var data = '';
var query = client.query("SELECT avg(Math) as math, avg(Reading) as reading FROM sat_scores WHERE State = '" + state + "'");
console.log("query is: %s", query);
query.on('row', function(row) {
console.log("Row cnt is: %s", row.math);
console.log("row is: " + row)
data += row;
});
console.log("made it");
query.on('end', function() {
callback(data);
});
}

Node JS How to get the query data outside

I am new from Node js , just i am trying implementing this functionality last few days but i am unable to fix
exports.get_exercises_for_trainer = function(req, res)
{
connection.query('SELECT * FROM ag_exercise', function(err, exercise)
{
console.log('------------------------------before add fields ----------------------------------');
console.log(exercise);
for (var i in exercise)
{
fields(exercise[i].wt_id, function(result1) {
exercise[i].wt_fields = result1; //here i am adding result set of other query but i am not geting this fields data
console.log(result1) //but i printed here working fine but i need this result1 data out side query
});
}
console.log('------------------------------after add fields ----------------------------------');
console.log(exercise);
res.render('pages/trainer_home.ejs',{page_title:"Exercise Create",exercise:exercise});
});
}
function fields(wt_id,callback)
{
connection.query('SELECT * FROM ag_workout_type_fields WHERE wt_id = "'+wt_id+'"', function( err1, result1){
callback(result1);
});
}
I have one more query ? in node js : If table having users , users having different relation tables like orders , profiles , address
How to implement this
First i am getting users
User loop
getting every user profiles ,address,orders
end user loop
but above scenario i am unable to implement in node js but in php very simple like this
$get_users = ... //users data
foreach($getusers as $usr)
{
$usr->orders = //orders data
.... like etc
}

There are three main questions here, I will adress each seperately.
Question 1: When making an async function, how do I then access my data outside that function?
All data from async calls are accessed via callback, event listeners or promises (a fancy callback and event listener handler). For the most part, you are going to just be using callbacks. So, instead of :
get_user = function(user_id){
//Do some stuff to get the user
return the_user;
};
var user = get_user('1234');
//do whatever you want with user
You will see :
get_user = function(user_id,callback){
//Do some stuff to get the user
callback(null,the_user);
}
get_user('1234',function(err,user){
//do whatever you want with user
});
When we get to Question 3, you will see the more complicated use case you were speaking of.
Question 2: How do I loop through my data, perform a subsiquent query on each row, and append that data to my current data?
There are a couple of issues here.
Every time you query the database, you are performing an asynchronous function, so you need to manage all of those callbacks accordingly. Fortunately there are some great tools to do that for you, and we will be using async.
Every time you call an asynchronous function in a for loop, the for loop continues, thus your iterator is overwritten, but your asynchronous function is not done with it yet, so you will get all sorts of unexpected results like vanishing variables, or missmapped results. You can handle this with JavaScript closures, or, you can rely again on libraries like async which handle it all for you.
Instead of running a for loop over your queries results, we're going to pass it to async.forEachOf, which we will use to modify the existing array and append the results of the subsequent queries to the primary query's rows. It is important to note that forEachOf will run the subsequent queries in parallel, so you should not be using a single database connection, but a pool. If you MUST use a single database connection, use forEachOfSeries instead.
async = require('async');
exports.get_exercises_for_trainer = function(req, res){
connection.query('SELECT * FROM ag_exercise', function(err, exercises)
{
console.log('------------------------------before add fields ----------------------------------');
console.log(exercises);
async.forEachOf(exercises,function(exercise,index,callback){
connection.query('SELECT * FROM ag_workout_type_fields WHERE wt_id = "' + exercise.wt_id + '"', function( err, result1){
if(err)return callback(err1);
exercises[index].wt_fields = result1; //Modify original array
return callback();
});
},function(err){
if(err){return;} //do some error handling
console.log('------------------------------after add fields ----------------------------------');
console.log(exercises);
res.render('pages/trainer_home.ejs',{page_title:"Exercise Create",exercise:exercises});
});
});
};
Question 3: How do I perform many related but different queries so that I can populate information about my object?
This is another great usage of the async library. In this case since the queries are all different, we'll use parallel instead of forEachOf.
async = require('async');
populate_user = function(user,_callback){
async.paralell({
profile: function(callback){
var sql = "SELECT * FROM profiles WHERE user_id = " + user.id + " LIMIT 1 ";
var connection.query(sql,function(err,rows,fields){
if(err)return callback(err);
if(rows.length === 1)return callback(null,rows[0]);
return callback(null,[]);
});
},
address: function(callback){
var sql = "SELECT * FROM addresses WHERE user_id = " + user.id + " LIMIT 1 ";
var connection.query(sql,function(err,rows,fields){
if(err)return callback(err);
if(rows.length === 1)return callback(null,rows[0]);
return callback(null,[]);
});
},
orders: function(callback){
var sql = "SELECT * FROM orders WHERE user_id = " + user.id;
var connection.query(sql,function(err,rows,fields){
if(err)return callback(err);
if(rows.length > 0)return callback(null,rows); //notice how this one could have multiple results so we're returning them all
return callback(null,[]);
});
},
},
function(err,result){
if(err)return _callback(err);
for(var att in result){user[att] = result[att];}
callback(null,user);
}
}
user = {id:1234};
populate_user(user,function(err,populated_user)){
console.log(user); //wow notice how it's populated too!
console.log(populated_user); //this is really just a new handle for the same object
});
I want to note that NONE of this was tested, not even for syntax, so it may take a little reworking.

How can I use arr.forEach to call async JavaScript redis calls?

I'm working with node.js and redis. I've got a redis database with a bunch of keys. Something like this:
user/chris/potion
user/pete/potion
user/chris/race
user/pete/race
user/chris/weapon
user/pete/weapon
I want to do a redis call which retrieves all user stats, puts the stats into a JS object, then passes it to the client for displaying character stats in the browser. Using javascript I inject the username chris at u into the redis call like this:
KEYS user/u/*
which returns:
1) "user/chris/weapon"
2) "user/chris/race"
3) "user/chris/potion"
Now I can iterate through those results, get the value of each key with GET, and make a javascript object. Seems super simple so I write the code. I quickly run into problems using forEach:
var redis = require('redis');
var client = redis.createClient();
exports.getUserObject = function(requesteduser, callback) {
var userstats = {}; // the object to hold the user stats once retrieved from the db
client.KEYS('user/' + requesteduser + '/*', function(err, replies) {
replies.forEach(function (reply, i) {
client.GET(reply, function(err, value) {
// get the key name so we can populate a js object
var n = reply.lastIndexOf('/');
var key = reply.substring(n + 1, reply.length);
userstats[key] = value;
console.dir(userstats);
callback(null, userstats); // parent expects (err, userstats)
});
});
});
}
When ran, output is like this:
{ weapon: 'twosword' }
{ weapon: 'twosword', race: 'elf' }
{ weapon: 'twosword', race: 'elf', potion: 'mana' }
callback(null, userstats) is called three times. Calling callback more than once will be problematic, since eventually callback will trigger data being sent data to the client.
I think I know what is happening. client.GET is ran three times because I asked it to. The replies array has three elements, so each time the result comes in for client.GET, the callback is called.
What I need to happen is for the forEach loop to finish iterating through all the redis keys, and once that's done, call the callback once. I tried solving this problem first with promises, then with async, because async has async.each. I got stuck solving the problem with both. I'm back to square one now, I'm convinced I have to do something different with forEach to make progress.
How can I accomplish this?

Since you're iterating over replies, you can check when you've reached the last element and only call callback in that instance.
client.KEYS('user/' + requesteduser + '/*', function(err, replies) {
replies.forEach(function (reply, i) {
client.GET(reply, function(err, value) {
// get the key name so we can populate a js object
var n = reply.lastIndexOf('/');
var key = reply.substring(n + 1, reply.length);
userstats[key] = value;
console.dir(userstats);
if (i == replies.length-1) callback(null, userstats); // parent expects (err, userstats)
});
});
});

I know it's a little late to help #Grimtech but I'll leave my opinion for other people that arrives here eventually:
First of all I rather to model #Grimtech's problem differently. Something like this:
client.hmset("user:chris", "weapon", "some weapon", "race", "some race", "potion", "some potion");
client.hmset("user:pete", "weapon", "same weapon", "race", "other race", "potion", "other potion");
Then I would have a method in Node returning a json like the following, assuming that I can have more than one key starting with "user:chris":
router.get('/users/:user_id', function(req, res, next) {
var response = [];
var client = redis.createClient();
client.keys("user:" + req.params.user_id + "*", function (err, users) {
users.forEach(function (key, pos) {
client.hgetall(key, function (err, user) {
response.push(user);
if (pos === users.length - 1) {
res.json(response);
client.quit();
}
});
});
});
});
The if (pos === users.length - 1) solves the async issue.
The json returned will have all the "user:chris" attributes so yo can do whatever you want in the client browser.