database = firebase.database();
var ref = database.ref('Users');
ref.orderByChild('state').equalTo("0").once('value', function(snapshot) {
var Users = snapshot.val();
i=0;
if (Object.keys(Users).length > 0){
getUser(Users);
} else {
console.log("No Users")
}
});
What I am doing is having a node js bot run through my database and search for users with state= 0. If state equals to zero, I run another script that goes and gets some information about them, updates their entry, and then changes the state to 1.
I have quite a large database, so it would be great if I could run a few instances of my bot. It won't work, though, because when the bots initially run, they all look at the same entries and remember which ones have a state = 0 and then they all repeat each other's work.
I tried changing the ref.orderByChild from using "once" to "on" child changed. That didn't seem to work though because it seemed as though the script was always waiting/listening for changes.. and never actually finished one loop. It does not move on to the next entry.
What's the best way to tackle something like this: having multiple bots being able to edit a firebase database without repeating each other's work?
Query and save all the data with a "master" script, then have it divvy up the entire thing and offload the split data to other scripts that receive their portion of data as input.
Related
I have an app where I spawn several BrowserWindows, with html forms, and I'd like to collect all the data (in order to save it, to be able to spawn them in the same state at a restart) at a press of a button.
At the moment, the only solution I found to do so, is to have each BrowserWindow do ipcRenderer.send every single time any variable changes (not too hard to do with Vuejs 'watchers'), but this seems demanding and inefficient.
I also thought of doing 'executeJavascript' to each window but that does not allow to capture the return value afaik.
I'd just like to be able to send a message from main when a request for saving is made, and wait for the windows to respond before saving all.
EDIT
I found a slightly better way, it looks like this
app.js
// wait for update reponses
ipc.on('update-response', (evt,args) => {
updates[evt.sender.id] = args;
if(Object.keys(updates).length == BrowserWindow.getAllWindows().length) {
// here I do what I need to save my settings, using what is stored in 'updates'
// ...
// and now reset updates for next time
updates = {}
}
});
// now send the requests for updates
BrowserWindow.getAllWindows().map(w => w.send('update'));
renderer.js
ipcRenderer.on('update', () => {
// collect the data
// var data = ...
ipcRenderer.send('update-response', data);
})
and obviously on the renderer side I am listening to these 'update' messages and sending data with 'udpate-response'.
But it seems a bit complicated and so I am sure there is a simpler way to achieve this using the framework.
EDIT 2
I realized that the above does not always work, because for some reason, the evt.sender.id do not match the ids obtained from BrowserWindows.getAllWindows(). I worked around that by sending ids in the request, and having the responder include it. But this is all so much fine for so very little...
I am new in database systems and what I am trying to do is to check whether the e-mail entered by the user during login exists in the database or not. I use Firebase Databse. So, the code I have written is this:
function login(){
var e_mail = document.getElementById("e-mail").value;
rootRef = firebase.database().ref();
rootRef.orderByChild("E_mail").on("child_added", function(snapshot){
lst.push(snapshot.val().E_mail);
//console.log(lst);
})
console.log(lst);
}
let lst = [];
login_btn.onclick = function() {login()};
I want to fetch all e-mails from the database, add them in the list and then loop through that list. Maybe this is not the best way, but that's what I'm working on. I could also just say if (snapshot.val().E_mail == e_mail){alert("there is such a user");}but the problem I have encountered and want to deal with is not that, it's the "callback" function inside login function. When I console the list in the outer function it shows an empty list as it does not run the inner function until it is done with the outer one. I understand this. But how can I avoid or fix this. I want to get the full list of e-mails to be able to loop through it then. Also, I don't know how to end the "loop" in Firebase, because it is sort of looping when it gets the e-mails. So I would like to stop at the moment when it finds a matching e-mail.
You're downloading all users to see if one name exists already. That is a waste of bandwidth.
Instead you should use a query to match the email you're looking for, and only read that node:
rootRef.orderByChild("E_mail").equalTo(e_mail).once("value", function(snapshot){
if (snapshot.exists()) {
// A node with the requested email already exists
}
})
In general, if you need to process all nodes, you'll want to use a value event, which executes for all matching nodes at once. So to get all users from the database, add them to a list, and then do something with that list:
rootRef.orderByChild("E_mail").once("value", function(snapshot){
var list = [];
snapshot.forEach(function(childSnapshot) {
list.push(childSnapshot.val());
});
console.log(list); // this will display the populated array
})
Note that you won't be able to access the list outside of the callback. Even if you declare the variable outside of the callback, it will only be properly populated inside the callback. See Xufox' comment for a link explaining why that is.
DATABASE:
SITUATION:
My website sells keys for a game.
A key is a randomly generated string of 20 characters whose uniqueness is guaranteed (not created by me).
When someone buys a key, NTWKeysLeft is read to find it's first element. That element is then copied, deleted from NTWKeysLeft and pasted to NTWUsedKeys.
Said key is then displayed on the buyer's screen.
PROBLEM:
How can I prevent the following problem :
1) 2 users buy the game at the exact same time.
2) They both get the same key read from NTWKeysLeft (first element in list)
3) And thus both get the same key
I know about Firebase Transactions already. I am looking for a pseudo-code/code answer that will point me in the right direction.
CURRENT CODE:
Would something like this work ? Can I put a transaction inside another transaction ?
var keyRef = admin.database().ref("NTWKeysLeft");
keyRef.limitToFirst(1).transaction(function (keySnapshot) {
keySnapshot.forEach(function(childKeySnapshot) {
// Key is read here:
var key = childKeySnapshot.val();
// How can I prevent two concurrent read requests from reading the same key ? Using a transaction to change a boolean could only happen after the read happens since I first need to read in order to know which key boolean to change.
var selectedKeyRef = admin.database().ref("NTWKeysLeft/"+key);
var usedKeyRef = admin.database().ref("NTWUsedKeys/"+key);
var keysLeftRef = admin.database().ref("keysLeft");
selectedKeyRef.remove();
usedKeyRef.set(true);
keysLeftRef.transaction(function (keysLeft) {
if (!keysLeft) {
keysLeft = 0;
}
keysLeft = keysLeft - 1;
return keysLeft;
});
res.render("bought", {key:key});
});
});
Just to be clear: keyRef.limitToFirst(1).transaction(function (keySnapshot) { does not work, but I would like to accomplish something to that effect.
Most depends on how you generate the keys, since that determines how likely collisions are. I recommend reading about Firebase's push IDs to get an idea how unique those are and compare that to your keys. If you can't statistically guarantee uniqueness of your keys or if statistical uniqueness isn't good enough, you'll have to use transactions to prevent conflicting updates.
The OP has changed the question a bit so, i will update the answer as follows: I will leave the bottom part about transactions as it was and will put the new update on top.
I can see two ways to proceed:
1) handle the lock system on your own and use JavaScript callbacks or other mechanisms for preventing simultaneous access to a portion of the code.
or
2) Use transactions/fireBase. On this case, i don't have the setup ready to share code other than sample/pseudo code provided at the bottom of this page.
With respect to option 1 above:
I have coded a use-case and put in on plunker. It uses JavaScript callbacks to queue users as they try to access the part of the code under lock.
I. user comes in and he is placed in queue
II. It then calls the callback function which pops users as
first come first out bases. I have the keys on top of the page to
be shared by the functions.
I have a button click event to this and when you click the button twice quickly, you will see keys assigned and they're different keys.
To read this code, click on the script.js file on the left and read starting from the bottom of the page where it calls the functions.
Here is the sample code in plunker. After clicking it, click on Run on top of the page and then click on the button on right hand side. Alert will pop up to show which key is given (note, there are two calls back to back to show two users coming in at same time)
https://plnkr.co/edit/GVFfvqQrlLeMaKlo5FCj?p=info
The fireBase transactions:
Use fireBase transactions to prevent concurrent read/write issues - below is the transaction() method signiture
transaction(dataToBeWritten, onComplete, applyLocally) returns fireBase.promise containing {
committed: boolean, nullable fireBase.database.snapshot }
Note, transaction needs writeOperation as first parameter and in your case looks like you’re removing a key upon success! hence the following function to be called in place of write
Try this pseudo code :
//first, get reference to your db
var selectedKeyRef = admin.database().ref("NTWKeysLeft/"+key);
// needed by transaction as first parameter
function writeOperation() {
selectedKeyRef.remove();
}
selectedKeyRef.transaction(function(writeOperation) , function(error,
committed, snapshot) {
if (error) {
console.log('Transaction failed abnormally!', error);
} else if (!committed) {
console.log('We aborted the transaction (because xyz).’);
} else {
console.log(‘keyRemoved!’);
}
console.log(“showKey: ", snapshot.val());
}); // end of the transaction() method call
Docs + to see parameters/return objects of the transaction() method see:
https://firebase.google.com/docs/reference/js/firebase.database.Reference#transaction
In the Docs.... If another client writes to the location before your new value is successfully written, your update function is called again with the new current value, and the write is retried.
https://firebase.google.com/docs/database/web/read-and-write#save_data_as_transactions
I don't think the problem you're worried about can happen. JavaScript, including Node, is single-threaded and can only do one thing at a time. If you had a big server infrastructure with more than one server running this code, then it would be possible, but for a single Node program, there's no problem.
Since none of the previous answers discussing the scope of Transactions worked out, I would suggest a different workaround.
Is it possible to trigger the unique code generation when someone buys a code? If yes, you could generate the unique string if the "buy" button is clicked, display the ID and save the ID to your database.
Later the user enters the key in your game, which checks if the ID is written in your database. This might probably also save a bit of data, since you do not need to keep track of the unique IDs before they get bought and you will also not run out of IDs, since they will always get generated when necessary.
I have a users table on Firebase and each user has an email prop.
Structure looks like:
Users -> User UID (looks like n8haBbjgablobA2ranfuabu3aaaga2af) -> User Obj which includes email prop.
I'd like to get an array of all the users' emails (~1m).
How can I most efficiently do this?
Ps.:
I tried:
usersRef.startAt(0).endAt(20).once("value", function(snapshot) {
console.log('FIRST 20');
console.log(snapshot.val()); // null
});
But that fails.
Probably the most efficient approach in terms of data reads would be to denormalize your data. You could store the email addresses both in the individual user nodes and in an emailAddresses node. Then you could just query the emailAddresses node directly for your list of emails.
Still ~1m email address nodes would probably be too much all at once. I'd probably grab it in chunks... I'm guessing.
Update
"Grabbing in chunks" is essentially pagination. I would try to use something off the shelf before trying to roll my own pagination solution.
Pagination libraries to check out:
Firebase Utils Pagination: This is developed by Firebase, but they say it is experimental and not ready for production use. But, it's probably still worth messing around with.
firebase-paginator: this is developed by a community member and it seems pretty solid.
If you want to roll your own pagination, check out:
#kato's response in this StackOverflow answer He makes an interesting point about the potential problem with paginating a real time data set, but then provides some good starter code
Here's a good blog entry that talks about the code that I think is a part of the firebase-paginator library I linked to above
Everybody in their answers said that it was an easy thing, yet had no working solutions. Here's what I came up with:
usersRef.orderByChild('uid').limitToFirst(100).once('value', function (snapshot) {
var users = snapshot.val()
var uids = Object.keys(users);
var lastUid = uids[uids.length - 1];
// could be another property than uid, for example email, or username. Ps.: Consider that the last ID from the previous chunk will be duplicated.
usersRef.orderByChild('uid').startAt(lastUid).limitToFirst(100).once('value', function (snapshot) {
var users = snapshot.val()
var uids = Object.keys(users);
console.log(uids);
var lastUid = uids[uids.length - 1];
// re-run function until done
})
})
Since this is a one-time deal, an option is to simply iterate over each node in the parent 'data' node capturing the child data, stripping out the email address and dumping that to a file.
the event you want is
child_added: retrieve lists of items or listen for additions to a list
of items. This event is triggered once for each existing child and
then again every time a new child is added to the specified path. The
listener is passed a snapshot containing the new child's data.
and the code to iterate all of the child nodes in the data node is
var dataRef = firebase.database().ref('myRootRef/data');
datRef.on('child_added', function(data) {
//data.val() will contain the child data, such as the email address
//append it to a text file here (for example), save to disk etc.
});
The key here is that this event is triggered once for each child, similar to iterating over all of the indexes in an array.
This will retrieve each child and present it to your app, one child at a time, iterating over all the children within the node.
It's going to take a while with that many nodes to chew through.
This is driving me crazy.
I have a message queue that I'm using a local storage web DB to store these messages when the device is offline and when there is an internet connection it sends out these messages. After it sends the messages, I want them deleted from the table.
When I send out the messages I keep an array called MessageIDs that way I can reference what rows need deleted.
I loop through the length of the MessageIDs and grab each ID and have a DELETE transaction. My alert() gets the right value but when the transaction executes, the value is undefined. I tried hard coding a known "ID" into the transaction and it worked. Any thoughts?
var MessageIDs = new Array();
//In the block of code not shown I populate MessageIDs and send out messages
for(var j=0; j < MessageIDs.length; j++)
{
alert(MessageIDs[j]); //Pulls the right value
site.data.database.transaction(
function (transaction) {
//[MessageIDs[j]] has a value of undefined and thus doesn't get deleted but the transaction doesn't technically fail either
transaction.executeSql("DELETE FROM Messages WHERE id = ?;", [MessageIDs[j]],
site.contact.removeQueuedMessagesSuccess, site.contact.removeQueuedMessagesError);
}
);
}
Sorry I can't put this as a comment, so here as an answer, extending DCoder's answer: You can also put the loop inside the transaction, then it will work, too. His solution is cleaner, though.
Edit: Maybe I should give a reason for why this addition is not unimportant: Obviously, you can't always combine the queries in such a way. So before you start nesting transactions, just put the loop of queries inside one transaction.