Develop Reference

How to get return the id in buffer form in nodejs

I have a test db with only 1 record and I've checked that the record is in the DB using the following code:
const prod = await Product.findOne({_id: params.ProductID}).exec();
console.log(prod)
My codebase has a lot of these types of queries. I cannot rewrite every query in the db so I need to make my testing work with the code that is already in place.
Product.findByIdAndUpdate(new Buffer(params.productID, 'base64').toString('hex')
I am having difficulty testing this because I am getting record not found. I feel like the record id has already been converted before I pass it in and the query is manipulating it which is why I can't find the record.
How can I send the id that this query is looking for?

Firebase concurrency issue: how to prevent 2 users from getting the same Game Key?

DATABASE:
SITUATION:
My website sells keys for a game.
A key is a randomly generated string of 20 characters whose uniqueness is guaranteed (not created by me).
When someone buys a key, NTWKeysLeft is read to find it's first element. That element is then copied, deleted from NTWKeysLeft and pasted to NTWUsedKeys.
Said key is then displayed on the buyer's screen.
PROBLEM:
How can I prevent the following problem :
1) 2 users buy the game at the exact same time.
2) They both get the same key read from NTWKeysLeft (first element in list)
3) And thus both get the same key
I know about Firebase Transactions already. I am looking for a pseudo-code/code answer that will point me in the right direction.
CURRENT CODE:
Would something like this work ? Can I put a transaction inside another transaction ?
var keyRef = admin.database().ref("NTWKeysLeft");
keyRef.limitToFirst(1).transaction(function (keySnapshot) {
keySnapshot.forEach(function(childKeySnapshot) {
// Key is read here:
var key = childKeySnapshot.val();
// How can I prevent two concurrent read requests from reading the same key ? Using a transaction to change a boolean could only happen after the read happens since I first need to read in order to know which key boolean to change.
var selectedKeyRef = admin.database().ref("NTWKeysLeft/"+key);
var usedKeyRef = admin.database().ref("NTWUsedKeys/"+key);
var keysLeftRef = admin.database().ref("keysLeft");
selectedKeyRef.remove();
usedKeyRef.set(true);
keysLeftRef.transaction(function (keysLeft) {
if (!keysLeft) {
keysLeft = 0;
}
keysLeft = keysLeft - 1;
return keysLeft;
});
res.render("bought", {key:key});
});
});
Just to be clear: keyRef.limitToFirst(1).transaction(function (keySnapshot) { does not work, but I would like to accomplish something to that effect.

Most depends on how you generate the keys, since that determines how likely collisions are. I recommend reading about Firebase's push IDs to get an idea how unique those are and compare that to your keys. If you can't statistically guarantee uniqueness of your keys or if statistical uniqueness isn't good enough, you'll have to use transactions to prevent conflicting updates.

The OP has changed the question a bit so, i will update the answer as follows: I will leave the bottom part about transactions as it was and will put the new update on top.
I can see two ways to proceed:
1) handle the lock system on your own and use JavaScript callbacks or other mechanisms for preventing simultaneous access to a portion of the code.
or
2) Use transactions/fireBase. On this case, i don't have the setup ready to share code other than sample/pseudo code provided at the bottom of this page.
With respect to option 1 above:
I have coded a use-case and put in on plunker. It uses JavaScript callbacks to queue users as they try to access the part of the code under lock.
I. user comes in and he is placed in queue
II. It then calls the callback function which pops users as
first come first out bases. I have the keys on top of the page to
be shared by the functions.
I have a button click event to this and when you click the button twice quickly, you will see keys assigned and they're different keys.
To read this code, click on the script.js file on the left and read starting from the bottom of the page where it calls the functions.
Here is the sample code in plunker. After clicking it, click on Run on top of the page and then click on the button on right hand side. Alert will pop up to show which key is given (note, there are two calls back to back to show two users coming in at same time)
https://plnkr.co/edit/GVFfvqQrlLeMaKlo5FCj?p=info
The fireBase transactions:
Use fireBase transactions to prevent concurrent read/write issues - below is the transaction() method signiture
transaction(dataToBeWritten, onComplete, applyLocally) returns fireBase.promise containing {
committed: boolean, nullable fireBase.database.snapshot }
Note, transaction needs writeOperation as first parameter and in your case looks like you’re removing a key upon success! hence the following function to be called in place of write
Try this pseudo code :
//first, get reference to your db
var selectedKeyRef = admin.database().ref("NTWKeysLeft/"+key);
// needed by transaction as first parameter
function writeOperation() {
selectedKeyRef.remove();
}
selectedKeyRef.transaction(function(writeOperation) , function(error,
committed, snapshot) {
  if (error) {
    console.log('Transaction failed abnormally!', error);
  } else if (!committed) {
    console.log('We aborted the transaction (because xyz).’);
  } else {
    console.log(‘keyRemoved!’);
  }
  console.log(“showKey: ", snapshot.val());
}); // end of the transaction() method call
Docs + to see parameters/return objects of the transaction() method see:
https://firebase.google.com/docs/reference/js/firebase.database.Reference#transaction
In the Docs.... If another client writes to the location before your new value is successfully written, your update function is called again with the new current value, and the write is retried.
https://firebase.google.com/docs/database/web/read-and-write#save_data_as_transactions

I don't think the problem you're worried about can happen. JavaScript, including Node, is single-threaded and can only do one thing at a time. If you had a big server infrastructure with more than one server running this code, then it would be possible, but for a single Node program, there's no problem.

Since none of the previous answers discussing the scope of Transactions worked out, I would suggest a different workaround.
Is it possible to trigger the unique code generation when someone buys a code? If yes, you could generate the unique string if the "buy" button is clicked, display the ID and save the ID to your database.
Later the user enters the key in your game, which checks if the ID is written in your database. This might probably also save a bit of data, since you do not need to keep track of the unique IDs before they get bought and you will also not run out of IDs, since they will always get generated when necessary.

Caching select query data on server side

I am writing an express app, where I'm pushing data from my views to a database. But most of the data is mapped to some other data in database tables.
For example, is a choose student name drop down- once you choose the student by his name , a drop down below - will show all roles that he is allowed for.
So I'm following this pattern of
app.post('\action1', function(req,res){
function querySomething(){
var defered = Q.defer();
connection.query(some_select_query,defered.makeNodeResolver());
return defered.promise;
}
function querySomethingElse(){
var defered = Q.defer();
connection.query(some_other_select_query,defered.makeNodeResolver());
return defered.promise;
}
Q.all([querySomething(), querySomethingElse()]).then((results,err) => {
connection.release()
if(results){
res.render('some_view.ejs', {
result1:results[0][0],
result2:results[1][0]
});
}
else{
res.render('error.ejs',{});
}
})
})
Now the problem is that I have to follow this pattern of selecting something from multiple tables, pass all these function to a promise- and when the results is passed back, goto my view with all those result objects - so that I can use them in my view - as a means of doing drop downs dependent on one another.
Sometimes I have to re-write this multiple times.
Doing a select query like this would be performance intensive especially if all views are using the result of the same query.
Is there any way I can build a cached data store on my express server side code and query that instead of the actual database??
If there is an insert or an update - i will refresh this store and just do a new select * that one time.
What libraries are there on top of express which will help me do this??
Does mysql-cache does the same thing?? I'm also using connection pooling with createPool.
How do I achieve this - or do I just restore to using big mvc's like sails to rewrite my app?

You can try apiCache npm module.
"Sometimes I have to re-write this multiple times."
Based on the business need, you may want to handle each use case separately and this scenario doesn't deal with caching.
Doing a select query like this would be performance intensive especially if all views are using the result of the same query.
This is a classic example for the need of server-side caching.

What's remove() and save() mean in mongodb node.js when initializing one database

I am newly using node.js. I am reading the code of one app. The code below is to initialize the db, to load some question into the survey system I can't understand what's remove() and save() means here. Because I can't find any explanation about these two method. It seems mongoose isn't used after being connected. Could any one explain the usage of these methods?
Well, this is my understanding of this code, not sure to be correct. My TA tell me it should be run before server.js.
/**
* This is a utility script for dropping the questions table, and then
* re-populating it with new questions.
*/
// connect to the database
var mongoose = require('mongoose');
var configDB = require('./config/database.js');
mongoose.connect(configDB.url);
// load the schema for entries in the 'questions' table
var Question = require('./app/models/questions');
// here are the questions we'll load into the database. Field names don't
// quite match with the schema, but we'll be OK.
var questionlist = [
/*some question*/
];
// drop all data, and if the drop succeeds, run the insertion callback
Question.remove({}, function(err) {
// count successful inserts, and exit the process once the last insertion
// finishes (or any insertion fails)
var received = 0;
for (var i = 0; i < questionlist.length; ++i) {
var q = new Question();
/*some detail about defining q neglected*/
q.save(function(err, q) {
if (err) {
console.error(err);
process.exit();
}
received++;
if (received == questionlist.length)
process.exit();
});
}
});

To add some additional detail, mongoose is all based on using schemas and working with those to manipulate your data. In a mongodb database, you have collections, and each collection holds different kinds of data. When you're using mongoose, what's happening behind the scenes is every different Schema you work with maps to a mongodb collection. So when you're working with Question Schema in mongoose land, there's really some Question collection behind the scenes in the actual db that your working with. You might also have a Users Schema, which would act as an abstraction for some Users collection in the db, or maybe you could have a Products Schema, which again would map to some collection of products behind the scenes in the actual db.
As mentioned previously, when calling remove({}, callback) on the Questions Schema, you're telling mongoose to go find the Questions collection in the db and remove all entries, or documents as they're called in mongodb, that match a certain criteria. You specify that criteria in the object literal that is passed in as the first argument. So if the Questions Schema has some boolean field called correct and you wanted to delete all of the incorrect questions, you could say Question.remove({ correct: false }, callback). Also as mentioned previously, when passing an empty object to remove, your telling mongoose to remove ALL documents in the Schema, or collection rather. If you're not familiar with callback functions, pretty much the callback function says, "hey after you finish this async operation, go ahead and do this."
The save() function that is used here is a little different than how save() is used in the official mongodb driver, which is one reason why I don't prefer mongoose. But to explain, pretty much all save is doing here is you're creating this new question, referred to by the q variable, and when you call save() on that question object, you're telling mongoose to take that object and insert it as a new document into your Questions collection behind the scenes. So save here just means insert into the db. If you were using the official mongo driver, it would be db.getCollection('collectionName').insert({/* Object representing new document to insert */}).
And yes your TA is correct. This code will need to run before your server.js file. Whatever your server code does, I assume it's going to connect to your database.
I would encourage you to look at the mongoose API documentation. Long term though, the official mongodb driver might be your best bet.

Mongoose basically maps your MongoDB queries to JavaScript objects using schema.
remove() receives a selector, and callback function. Empty selector means, that all Questions will be affected.
After that a new Question object is created. I guess that you omitted some data being set on it. After that it's being saved back into MongoDB.
You can read more about that in the official documentation:
http://mongoosejs.com/docs/api.html#types-subdocument-js

remove query is use for removing all documents from collection and save is use for creating new document.
As per your code it seems like every time the script run it removes all the record from Question collection and then save new records for question from question list.

How to perform sql "LIKE" operation on firebase?

I am using firebase for data storage. The data structure is like this:
products:{
product1:{
name:"chocolate",
}
product2:{
name:"chochocho",
}
}
I want to perform an auto complete operation for this data, and normally i write the query like this:
"select name from PRODUCTS where productname LIKE '%" + keyword + "%'";
So, for my situation, for example, if user types "cho", i need to bring both "chocolate" and "chochocho" as result. I thought about bringing all data under "products" block, and then do the query at the client, but this may need a lot of memory for a big database. So, how can i perform sql LIKE operation?
Thanks

Update: With the release of Cloud Functions for Firebase, there's another elegant way to do this as well by linking Firebase to Algolia via Functions. The tradeoff here is that the Functions/Algolia is pretty much zero maintenance, but probably at increased cost over roll-your-own in Node.
There are no content searches in Firebase at present. Many of the more common search scenarios, such as searching by attribute will be baked into Firebase as the API continues to expand.
In the meantime, it's certainly possible to grow your own. However, searching is a vast topic (think creating a real-time data store vast), greatly underestimated, and a critical feature of your application--not one you want to ad hoc or even depend on someone like Firebase to provide on your behalf. So it's typically simpler to employ a scalable third party tool to handle indexing, searching, tag/pattern matching, fuzzy logic, weighted rankings, et al.
The Firebase blog features a blog post on indexing with ElasticSearch which outlines a straightforward approach to integrating a quick, but extremely powerful, search engine into your Firebase backend.
Essentially, it's done in two steps. Monitor the data and index it:
var Firebase = require('firebase');
var ElasticClient = require('elasticsearchclient')
// initialize our ElasticSearch API
var client = new ElasticClient({ host: 'localhost', port: 9200 });
// listen for changes to Firebase data
var fb = new Firebase('<INSTANCE>.firebaseio.com/widgets');
fb.on('child_added', createOrUpdateIndex);
fb.on('child_changed', createOrUpdateIndex);
fb.on('child_removed', removeIndex);
function createOrUpdateIndex(snap) {
client.index(this.index, this.type, snap.val(), snap.name())
.on('data', function(data) { console.log('indexed ', snap.name()); })
.on('error', function(err) { /* handle errors */ });
}
function removeIndex(snap) {
client.deleteDocument(this.index, this.type, snap.name(), function(error, data) {
if( error ) console.error('failed to delete', snap.name(), error);
else console.log('deleted', snap.name());
});
}
Query the index when you want to do a search:
<script src="elastic.min.js"></script>
<script src="elastic-jquery-client.min.js"></script>
<script>
ejs.client = ejs.jQueryClient('http://localhost:9200');
client.search({
index: 'firebase',
type: 'widget',
body: ejs.Request().query(ejs.MatchQuery('title', 'foo'))
}, function (error, response) {
// handle response
});
</script>
There's an example, and a third party lib to simplify integration, here.

I believe you can do :
admin
.database()
.ref('/vals')
.orderByChild('name')
.startAt('cho')
.endAt("cho\uf8ff")
.once('value')
.then(c => res.send(c.val()));
this will find vals whose name are starting with cho.
source

The elastic search solution basically binds to add set del and offers a get by wich you can accomplish text searches.
It then saves the contents in mongodb.
While I love and reccomand elastic search for the maturity of the project, the same can be done without another server, using only the firebase database.
That's what I mean:
(https://github.com/metaschema/oxyzen)
for the indexing part basically the function:
JSON stringifies a document.
removes all the property names and JSON to leave only the data
(regex).
removes all xml tags (therefore also html) and attributes (remember
old guidance, "data should not be in xml attributes") to leave only
the pure text if xml or html was present.
removes all special chars and substitute with space (regex)
substitutes all instances of multiple spaces with one space (regex)
splits to spaces and cycles:
for each word adds refs to the document in some index structure in
your db tha basically contains childs named with words with childs
named with an escaped version of "ref/inthedatabase/dockey"
then inserts the document as a normal firebase application would do
in the oxyzen implementation, subsequent updates of the document ACTUALLY reads the index and updates it, removing the words that don't match anymore, and adding the new ones.
subsequent searches of words can directly find documents in the words child. multiple words searches are implemented using hits

SQL"LIKE" operation on firebase is possible
let node = await db.ref('yourPath').orderByChild('yourKey').startAt('!').endAt('SUBSTRING\uf8ff').once('value');

This query work for me, it look like the below statement in MySQL
select * from StoreAds where University Like %ps%;
query = database.getReference().child("StoreAds").orderByChild("University").startAt("ps").endAt("\uf8ff");