Develop Reference

Cant read data from collection in MongoDB Atlas Trigger

New to MongoDB, very new to Atlas. I'm trying to set up a trigger such that it reads all the data from a collection named Config. This is my attempt:
exports = function(changeEvent) {
const mongodb = context.services.get("Cluster0");
const db = mongodb.db("TestDB");
var collection = db.collection("Config");
config_docs = collection.find().toArray();
console.log(JSON.stringify(config_docs));
}
the function is part of an automatically created realm application called Triggers_RealmApp, which has Cluster0 as a named linked data source. When I go into Collections in Cluster0, TestDB.Config is one of the collections.
Some notes:
it's not throwing an error, but simply returning {}.
When I change context.services.get("Cluster0"); to something else, it throws an error
When I change "TestDB" to a db that doesnt exist, or "Config" to a collection which doesn't exist, I get the same output; {}
I've tried creating new Realm apps, manually creating services, creating new databases and new collections, etc. I keep bumping into the same issue.
The mongo docs reference promises and awaits, which I haven't seen in any examples (link). I tried experimenting with that a bit and got nowhere. From what I can tell, what I've already done is the typical way of doing it.
Images:
Collection:
Linked Data Source:

I ended up taking it up with MongoDB directly, .find() is asynchronous and I was handling it incorrectly. Here is the reply straight from the horses mouth:
As I understand it, you are not getting your expected results from the query you posted above. I know it can be confusing when you are just starting out with a new technology and can't get something to work!
The issue is that the collection.find() function is an asynchronous function. That means it sends out the request but does not wait for the reply before continuing. Instead, it returns a Promise, which is an object that describes the current status of the operation. Since a Promise really isn't an array, your statment collection.find().toArray() is returning an empty object. You write this empty object to the console.log and end your function, probably before the asynchronous call even returns with your data.
There are a couple of ways to deal with this. The first is to make your function an async function and use the await operator to tell your function to wait for the collection.find() function to return before continuing.
exports = async function(changeEvent) {
const mongodb = context.services.get("Cluster0");
const db = mongodb.db("TestDB");
var collection = db.collection("Config");
config_docs = await collection.find().toArray();
console.log(JSON.stringify(config_docs));
};
Notice the async keyword on the first line, and the await keyword on the second to last line.
The second method is to use the .then function to process the results when they return:
exports = function(changeEvent) {
const mongodb = context.services.get("Cluster0");
const db = mongodb.db("TestDB");
var collection = db.collection("Config");
collection.find().toArray().then(config_docs => {
console.log(JSON.stringify(config_docs));
});
};

The connection has to be a connection to the primary replica set and the user log in credentials are of a admin level user (needs to have a permission of cluster admin)

Get data out of callbacks

I am using "pdf-text" module for Node.js to convert a pdf into a string array and then get specific elements out of it. But the problem is, I can only access the data, "chunks", only when I am inside the callback. I want to store it in some global variable so that I can use it in different files. I have tried storing the elements of the array inside variables while inside the function, but no luck. Here's the code:
var pdfText = require('pdf-text');
var pathToPdf = "PDF FILE NAME";
var fs = require('fs');
var buffer = fs.readFileSync(pathToPdf);
var output;
pdfText(buffer, function(err, chunks){
if (err){
console.dir(err);
return;
}
console.dir(chunks);
output = chunks;
}
console.dir(output);
P.S. I am fairly new to Node.js and JavaScript and help would be appreciated greatly.

The output variable will only be set with "chunks" contents when the callback is called.
Btw, you need to add ");" after the callback function declaration on the pdfText function call.
var pdfText = require('pdf-text');
var pathToPdf = "PDF FILE NAME";
var fs = require('fs');
var buffer = fs.readFileSync(pathToPdf);
var output;
pdfText(buffer, function(err, chunks){
if (err){
console.log(err);
return;
}
otherFunction(); // undefined
output = chunks;
otherFunction(); // chunks content
});
function otherFunction() {
console.log(output);
}
console.log(output); // undefined
About js callbacks: https://www.tutorialspoint.com/nodejs/nodejs_callbacks_concept.htm

But the problem is, I can only access the data, "chunks", only when I am inside the callback.
Yes, that is correct. You can't access the data before it is available, and when it becomes available, your callback gets called with the data.
I want to store it in some global variable so that I can use it in different files.
Suppose you did this. Now you have a problem. Your code in those different files: how will it know when the data is ready? It won't.
You need some way to tell that code the data is ready. The way you tell that code is by calling a function. And at that point you don't need global variables: when you call the function in that other file, you pass the data to it as a function parameter.
In other words, don't just have global code in some file that expects to be able to use your chunks data by referencing a global variable. Instead, write a function that you can call from your callback, and pass chunks into that function.

If you are using node 8, I believe you can use the async-await feature. So you can refactor your code so that it looks like the following:
var pdfText = require('pdf-text');
var pathToPdf = "PDF FILE NAME";
var fs = require('fs');
var buffer = fs.readFileSync(pathToPdf);
var output;
async function getPDF(buffer) {
pdfText(buffer, function(err, chunks){
if (err){
console.dir(err);
return;
}
return await chunks;
}
}
// you can get the chunks given the buffer here!
console.dir(getPDF(buffer));
I want to store it in some global variable so that I can use it in different files. I have tried storing the elements of the array inside variables while inside the function, but no luck.
I don't think you can store the chunks as a global variable though as you would have to export the chunk (e.g module.exports = getPDF(buffer);), which is synchronous, while the function getPDF is asynchronous. So you have to use it within the same file. What I would do, is import the function instead and then pass it a different buffer in different js files where different pdf is required. Hope this helps.

PouchDB gives 409 "Document update conflict" error with .put(), even with correct ._rev

I have a function that may either create or update documents in PouchDB, as follows. The function works perfectly when run the first time. However, every subsequent run yields a 409 error, even though the ._rev property appears to be correct.
function saveEventMatchesToDatabase(event, db) {
/* event: An event object from The Blue Alliance API. */
/* db: a reference ot the PouchDB database. */
/* Purpose: Given an event, extract the list of matches and teams, and save them to the database. */
TBA.event.matches(event.key, function(matches_list) {
var i = 0;
for (i = 0; i < matches_list.length; i++) {
var match = new Object();
var docrec = new Object();
match._id = 'matches/' + matches_list[i].key;
match.redTeam = matches_list[i].alliances.red.teams;
match.blueTeam = matches_list[i].alliances.blue.teams;
/* If the doc already exists, we need to add the _rev to update the existing doc. */
db.get(match._id).then(function(doc) {
match._rev = doc._rev;
docrec = doc;
}).catch(function(err) {
if ( err.status != 404 ) {
/* Ignore 404 errors: we expect them, if the doc is new. */
console.log(err);
}
});
db.put(match).then(function() {
// Success!
}).catch(function(err) {
console.log('\ndoc._rev: ' + docrec._rev);
console.log('match._rev: ' + match._rev);
console.log(err);
});
}
});
}
Sample console output from running this function the second time is below. The same error occurs for EVERY item in match_list, not just intermittently.
doc._rev: 1-7cfa2c6245dd939d8489159d8ca674d9
match._rev: 1-7cfa2c6245dd939d8489159d8ca674d9
r {status: 409, name: "conflict", message: "Document update conflict", error: true}
I'm not sure what I'm missing, that's causing this problem. Any suggestions for where to look next would be greatly appreciated.

The first problem seems to be that you're using a function within a loop, meaning that any variables used inside of the inner functions are randomly changing under your feet depending on when the function gets invoked. Instead of a for loop, you can use forEach().
However, the second problems is that you are not using promises correctly; you need to wait for the result of get() before you do your put(). So probably forEach() is not even what you want in the first place; you probably want to use Promise.all() and then compose your promises.
I wrote a piece on this awhile back; many people have told me that it's worth reading even though it's long: "We have a problem with promises." Read that, and you should hopefully understand promises by the end of it. :)

Issue with output list for learnyounode #6 MAKE IT MODULAR

Just started coding last thursday, bear with me here:
my code for this question of the tutorial is returning a list of just the extension names from the directory and not a list of the files with the said extension, e.g. if i used a directory with 3 .js files and used js as my extension argument in the command line, then i would get
1. js
2. js
3. js
as the output, here is the question from the tutorial and my code. THANK YOU!
the question from learnyounode tutorial number 6:
LEARN YOU THE NODE.JS FOR MUCH WIN!
─────────────────────────────────────
MAKE IT MODULAR
Exercise 6 of 13
This problem is the same as the previous but introduces the concept of modules. You will need to create two files to solve this.
Create a program that prints a list of files in a given directory, filtered by the extension of the files. The first argument is the directory name and the second argument is the extension filter. Pr
int the list of files (one file per line) to the console. You must use asynchronous I/O.
You must write a module file to do most of the work. The module must export a single function that takes three arguments: the directory name, the filename extension string and a callback function, in
that order. The filename extension argument must be the same as was passed to your program. i.e. don't turn it into a RegExp or prefix with "." or do anything else but pass it to your module where y
ou can do what you need to make your filter work.
The callback function must be called using the idiomatic node(err, data) convention. This convention stipulates that unless there's an error, the first argument passed to the callback will be null, a
nd the second will be your data. In this case, the data will be your filtered list of files, as an Array. If you receive an error, e.g. from your call to fs.readdir(), the callback must be called wi
th the error, and only the error, as the first argument.
You must not print directly to the console from your module file, only from your original program.
In the case of an error bubbling up to your original program file, simply check for it and print an informative message to the console.
These four things are the contract that your module must follow.
Export a single function that takes exactly the arguments described.
Call the callback exactly once with an error or some data as described.
Don't change anything else, like global variables or stdout.
Handle all the errors that may occur and pass them to the callback.
The benefit of having a contract is that your module can be used by anyone who expects this contract. So your module could be used by anyone else who does learnyounode, or the verifier, and just work. *
and my code is:
module (p6m.js):
var fs=require('fs'), ph=require('path'), exports =module.exports={}
exports.f=function(path,ext,callbk){
fs.readdir(path,function(err,files){
if(err){
return callbk(err,null)
}
files=files.filter(
function(file){
return ph.extname(file)==="."+ext
}
)
return callbk(null,files)}
)}
and my program (p6.js):
var p6m=require('./p6m'), path=process.argv[2], ext=process.argv[3]
p6m.f(path, ext, function(err,files){
if(err){return console.log.error('Error occured:', err)};
files.forEach(function(file){
console.log(file)})})

I got the same problem with my code as of need to use a single function export . So instead of exporting a module function like this :
exports =module.exports={}
exports.f=function(path,ext,callbk){...};
try it doing this way :
module.exports = function (path, ext, callbk) {...};
because its a single function so you don't need to specify that function with a name " f " as if you are doing it in this statement :
exports.f = function(path,ext,callbk){...};
whenever you will import the module,it will automatically call this function only, since the module contains this single function.
You can try this piece of code, it works well for me.
module code: mymodule.js
var fs = require('fs');
var ph= require('path');
module.exports = function (path, ext, callbk) {
var pathio = "." + ext;
fs.readdir(path, function (err, files) {
if (err)
return callbk(err);
else {
var listf = []; //listf is the resultant list
for (var i = 0; i < files.length; i++) {
if (ph.extname(files[i]) === pathio) {
listf.push(files[i]);
}
}
callbk(null, listf);
}
});
}
program code : moduletest.js
var mod = require('./mymodule');
mod(process.argv[2], process.argv[3], function (err, listf) {
if (err) {
console.log('Error!')
} else {
for (var i = 0; i < listf.length; i++) {
console.log(listf[i]);
}
}
});
and do remember, learnyounode series is very specific about its way of coding and syntax, so even if you are doing the logic right way still you won't get pass,you need your code to be the best and optimized. I'll suggest you to refer to discussions on nodeschool itself for various issues you might get in learnyounode series.

That will work and output the right results, but what they are looking for is something like this:
module.exports = function() {};
Because they only want one function total in the exports.
You could also do something like this:
module.exports = FindFilesByExtension;
function FindFilesByExtension(path, ext, callback) {
//your code
}

Here is my solution,
Thsi is my module file filteredls.js
var fs = require('fs');
var path = require('path');
module.exports = function filterFiles(folder, extension, callback) {
fs.readdir(folder, function(err, files) {
if(err) return callback(err);
var filesArray = [];
files.forEach(function(file) {
if(path.extname(file) === "."+extension) {
filesArray.push(file);
}
});
return callback(null, filesArray);
});
}
And here is my test file for reading module modular.js
var ff = require('./filteredls.js');
ff(process.argv[2], process.argv[3], function(err, data) {
if(err)
return console.error(err);
data.forEach(function(file) {
console.log(file);
});
});
And this is my result screenshot,

How can I work around deeply nested callbacks within NodeJS and sqlite-3?

I've been getting to grips with node and node-sqlite3 and need to build a report up based on a number of queries:
var db = require('./db');
module.exports = {
getActivity : function (user_id, done) {
var report = {};
db.get('SELECT * FROM warehouse WHERE user_id = ?', user_id, function (err, warehouse) {
report.warehouse = warehouse;
db.all('SELECT * FROM shops WHERE warehouse_id = ?', report.warehouse.id, function (err, shops) {
report.shops = shops;
return done(report);
});
});
}
};
My goal was to be able to generate a report from a route and serialize it as a JSON response. Here's how my route looks:
app.get('/api/hello',
auth.check,
function(req, res) {
hello.getActivity(1, function (data) {
res.send(data);
});
});
I will most likely have more queries to include in this report and thus more nested callbacks. What options do I have to avoid this? I'm familiar with promises etc but node-sqlite doesn't have anything built in for cleaning this stuff up. Maybe I am using it incorrectly?
Last of all, I am passing in a 'done' callback from the route. Maybe this is the node way of doing things but it would be great if I could just simply return the report once it's generated, without the callback. Is there a better pattern for this?
Any help appreciated!

I have a report engine built on node that has the same issues of multiple queries. To keep thinks clean, I use async, which is an awesome control flow library:
https://github.com/caolan/async#series
You will want to look at the async.series. It keeps your code a little cleaner than tons of embedded functions.
NOTE: You will need to create a reference to variables you need to access from one step to the next outside of the async.series context. For exaple I use the var one in function for two:
//keep context to shared values outside of the async function
var one,
two;
async.series([
function(callback){
// do some stuff ...
one = 'one';
callback(null, one);
},
function(callback){
//!access value from previous step
two = one + one;
callback(null, two);
}
],
// optional callback
function(err, results){
// results is now equal to ['one', 'oneone']
});