Question - the exercise from NODESCHOOL.IO;LEARNYOUNODE;MAKE_IT_MODULAR reads directory and filters files by a given file extension - uses a module to do most of the work. Question on the added code referenced below. First time see this - but i take it from this example you are able to add code to an already defined function when you call it. But want to clarify if I understand the execution of it. Is it the callback function - noted below that allows this "added" code to be executed? Thanks
solution.js:
var filterFn = require('./solution_filter.js')
var dir = process.argv[2]
var filterStr = process.argv[3]
filterFn(dir, filterStr, function (err, list) {
if (err)
return console.error('There was an error:', err)
// QUESTION ON THIS PART OF THE CODE - see below in the module part of the program
list.forEach(function (file) {
console.log(file)
})
})
// THIS IS THE MODULE FOR THE PROGRAM
solution_filter.js:
// require file system module
var fs = require('fs')
module.exports = function (dir, filterStr, callback) {
var regex = new RegExp('\\.' + filterStr + '$')
fs.readdir(dir, function (err, list) {
// callback err if the readdir method fails
if (err)
return callback(err)
list = list.filter(function (file) {
return regex.test(file)
})
// IS THIS CALLBACK SO IT LOOP AND IS ABLE TO EXECUTE ADDED CODE in the solution.js filterFn()
callback(null, list)
})
}
Thinking about it as "adding code to an already defined function" isn't really accurate. You're passing in a completely different function as the callback parameter. The code in the module is written to expect that, and it invokes the function you pass in.
Your function doesn't have access to the code in the module, and the module doesn't have access to the code in your function.
Creating functions on-the-fly and passing them as parameters to other functions is a very common pattern in JavaScript.
Related
I'm working on a pre-existing Twilio function and am trying to understand how the following code works. I've tested it a few times, and as long as I send the 'from' phone number any message, I get the response described in the 'body'. However, I'm trying to understand how the 'msg' and 'err' parameters can exist, given they aren't declared elsewhere in the function. Can someone help me understand how these parameters are valid? I'm new to JavaScript and only understand the basics of callbacks and promises.
EDIT: I'm starting to understand the idea more. So the 'then' and 'catch' statements are callbacks that occur after the create() function finishes. Thank you for the answers that helped me understand.
exports.handler = function(context, event, callback) {
context.getTwilioClient().messages.create({
to: '+14806484732 ',
from: '+12133151685',
body: 'Remember the special today is Shrimp Scampi'
}).then(msg => {
callback(null, msg.sid);
}).catch(err => callback(err));
}
err => callback(err) - this is a declaration of function. single parameter err, executing function callback
similar situation with
.then(msg => { // `msg` is a single parameter of declared function
callback(null, msg.sid);
}
Example
const callback = argument => console.log(argument)
// this line is similar to
// const callback = (argument) => { console.log(argument) }
// also similar to
// function callback(argument) { console.log(argument) }
const func = err => callback(err)
// this line is similar to
// const func = (err) => { callback(err) }
// also similar to
// function func(err) { callback(err) }
func('test')
The functions given in then and catch are callbacks that both take a single parameter. Each of these parameters have a specific meaning based on the function documentation, which you can look up. You are, however, free to call them whatever you like if you were to write the callbacks yourself. The names are simply chosen to convey the semantics as clearly as possible.
Thus, they are essentially just parameters of an anonymous function.
I strongly recommend looking into callbacks in Javascript, as they are used everywhere.
I am completely new to JavaScript, and I am having a trouble with using fs.readFile() function.
I am required to read a file, and determine whether I have to call successFn (when err boolean is false) or errorFn (if err boolean is true).
When I err is false, however, I am trying to call an "extractor" function as a parameter -- extractor separates the data read into each word.
I am required to call fs.readFile, so I am required to use parameters and one of the parameters is fileName.
The grader is going to test my code using his own files, so I am not supposed to define fileName, yet I am being asked to define it (I understand why I am being asked to do it, but I feel that there should be a way to get around this issue.)
Also, JavaScript syntax is very not intuitive, so I might be making some mistakes with the syntax, but here is my code, and please let me know how I can fix my issue:
function readAndExtractWith(extractor) {
fs.readFile(fileName, 'utf-8', (err, data) => {
if (err) {
errorFn(err);
}
else {
data = extractor();
successFn(data);
}
return extractor;
});
}
fileName is not defined because you never define it. Since you are using that in a function you need to either pass the variable as an argument, specify it inside the function, or define it globally.
var fileName = "stackoverflow.jpg";
// when executing the function pass the fileName parameter
readAndExtractWith(extractor, fileName);
function readAndExtractWith(extractor, fileName) {
fs.readFile(fileName, 'utf-8', (err, data) => {
if (err) {
errorFn(err);
}
else {
data = extractor();
successFn(data);
}
return extractor;
});
}
I'm trying to write some code which will take a string containing partial URLs like google.com and will transform them into a full URL like https://google.com.
I'm very inexperienced with Node.js and I'm still trying to get my head around the idea of asynchronicity.
I'm attempting to use callbacks instead of stuff like promises and async/await, and in my head the following code should work:
exports.rectifyDocumentURLs = function(baseUrl, document, callback) {
callback(null,
document.replace(url_patterns.any, (match) => {
return exports.fixUrl(match, baseUrl, (err, res) => {
if (err) {
callback(err, null)
}
return res
})
})
)
}
url_patterns.any is some regex code which will match any type of code, and the function exports.fixUrl is a function which will take a partial URL and return it in its full form.
When I run the code like so
exports.rectifyDocumentURLs("https://google.com", "google.com", (rectifyErr, rectifyRes) => {
console.log(rectifyRes)
})
The current code just returns undefined, but the res of the fixUrl function returns the correct result, http://google.com.
I understand that this a lot like many of the questions on here, but after extensive research and many tries and rewrites, I believe this may be the only way to fix the code.
Any help would be greatly appreciated.
You can use the url module who provide utilities for URL resolution and parsing.
const url = require('url');
const myURL = new URL('https://google.com');
console.log(myURL.host); // google.com
Your function rectifyDocumentURLs() is all messed up.
The idea of the callback is that when something will happen asynchronously you don't know when it will be finished. So you pass it a function, when it's done it calls the function you passed it with the result you want. Typically functions that take callbacks never return a meaningful value -- the value is passed to the callback. In rectifyDocumentURLs() you are calling the callback immediately, then calling it again with the callback of another function and also returning a (maybe undefined) value. That's just not going to work.
Here's a more standard approach:
exports.rectifyDocumentURLs = function(baseUrl, document, callback) {
// it looks like document.replace is async and takes a callback
// when it's done it will pass its result to the callback as match
document.replace(url_patterns.any, (match) => {
// you don't need return the result of a function that takes a callback
// it will pass its result to the callback
exports.fixUrl(match, baseUrl, (err, res) => {
if (err) {
callback(err, null)
} else {
// now that you have res and no error, call the callback
// you passed to rectifyDocumentURLs()
// this value will be the rectifyRes in the function you passed
callback(null, res)
}
})
})
}
Of course, as others pointed out, there is already a lot of existing code for handling URLS, but this is a good exercise.
For a project, I am using the net module to create a 'mini web framework'
I'm having a lot of trouble dealing with this one callback
var sendFile(path) {
fs.readFile(path, config, this.handleRead.bind(this));
}
where readFile is defined as:
var handleRead = function(contentType, data, err) {
if (err) {
console.log(err); //returns properly
} else {
console.log(data); //returns properly
console.log(contentType) //returning undefined
}
So far this code works in the sense that I can catch errors and also write the data properly.
My question is : how do I send the contentType through the call back?
I've tried -
var sendFile(path) {
var contentType = ContentType['the path type']
fs.readFile(path, config, this.handleRead(contentType).bind(this));
}
But then this causes data and err to be undefined.
I'm quite new to js and am still confused about how to work with callbacks. Any input is appreciated!
.bind() lets you do more than just set the "context" (this value of a function). You can also "bind" arguments in the function.
Try:
function sendFile(path) {
var contentType = ContentType['the path type']
fs.readFile(path, config, this.handleRead.bind(this, contentType));
}
This will pass a callback with its context set to whatever this is and its 1st parameter set to contentType. As long as this callback is called with data (and possibly err), then everything will work.
Is it possible to load a Node.js module asynchronously?
This is the standard code:
var foo = require("./foo.js"); // waiting for I/O
foo.bar();
But I would like to write something like this:
require("./foo.js", function(foo) {
foo.bar();
});
// doing something else while the hard drive is crunching...
Is there a way how to do this? Or is there a good reason why callbacks in require aren't supported?
While require is synchronous, and Node.js does not provide an asynchronous variant out of the box, you can easily build one for yourself.
First of all, you need to create a module. In my example I am going to write a module that loads data asynchronously from the file system, but of course YMMV. So, first of all the old-fashioned, not wanted, synchronous approach:
var fs = require('fs');
var passwords = fs.readFileSync('/etc/passwd');
module.exports = passwords;
You can use this module as usual:
var passwords = require('./passwords');
Now, what you want to do is turn this into an asynchronous module. As you can not delay module.exports, what you do instead is instantly export a function that does the work asynchronously and calls you back once it is done. So you transform your module into:
var fs = require('fs');
module.exports = function (callback) {
fs.readFile('/etc/passwd', function (err, data) {
callback(err, data);
});
};
Of course you can shorten this by directly providing the callback variable to the readFile call, but I wanted to make it explicit here for demonstration purposes.
Now when you require this module, at first, nothing happens, as you only get a reference to the asynchronous (anonymous) function. What you need to do is call it right away and provide another function as callback:
require('./passwords')(function (err, passwords) {
// This code runs once the passwords have been loaded.
});
Using this approach you can, of course, turn any arbitrary synchronous module initialization to an asynchronous one. But the trick is always the same: Export a function, call it right from the require call and provide a callback that continues execution once the asynchronous code has been run.
Please note that for some people
require('...')(function () { ... });
may look confusing. Hence it may be better (although this depends on your actual scenario) to export an object with an asynchronous initialize function or something like that:
var fs = require('fs');
module.exports = {
initialize: function (callback) {
fs.readFile('/etc/passwd', function (err, data) {
callback(err, data);
});
}
};
You can then use this module by using
require('./passwords').initialize(function (err, passwords) {
// ...
});
which may be slightly better readable.
Of course you can also use promises or any other asynchronous mechanism which makes your syntax look nicer, but in the end, it (internally) always comes down to the pattern I just described here. Basically, promises & co. are nothing but syntactic sugar over callbacks.
Once you build your modules like this, you can even build a requireAsync function that works like you initially suggested in your question. All you have to do is stick with a name for the initialization function, such as initialize. Then you can do:
var requireAsync = function (module, callback) {
require(module).initialize(callback);
};
requireAsync('./passwords', function (err, passwords) {
// ...
});
Please note, that, of course, loading the module will still be synchronous due to the limitations of the require function, but all the rest will be asynchronous as you wish.
One final note: If you want to actually make loading modules asynchronous, you could implement a function that uses fs.readFile to asynchronously load a file, and then run it through an eval call to actually execute the module, but I'd highly recommend against this: One the one hand, you lose all the convenience features of request such as caching & co., on the other hand you'll have to deal with eval - and as we all know, eval is evil. So don't do it.
Nevertheless, if you still want to do it, basically it works like this:
var requireAsync = function (module, callback) {
fs.readFile(module, { encoding: 'utf8' }, function (err, data) {
var module = {
exports: {}
};
var code = '(function (module) {' + data + '})(module)';
eval(code);
callback(null, module);
});
};
Please note that this code is not "nice", and that it lacks any error handling, and any other capabilities of the original require function, but basically, it fulfills your demand of being able to asynchronously load synchronously designed modules.
Anyway, you can use this function with a module like
module.exports = 'foo';
and load it using:
requireAsync('./foo.js', function (err, module) {
console.log(module.exports); // => 'foo'
});
Of course you can export anything else as well. Maybe, to be compatible with the original require function, it may be better to run
callback(null, module.exports);
as last line of your requireAsync function, as then you have direct access to the exports object (which is the string foo in this case). Due to the fact that you wrap the loaded code inside of an immediately executed function, everything in this module stays private, and the only interface to the outer world is the module object you pass in.
Of course one can argue that this usage of evil is not the best idea in the world, as it opens up security holes and so on - but if you require a module, you basically do nothing else, anyway, than eval-uating it. The point is: If you don't trust the code, eval is the same bad idea as require. Hence in this special case, it might be fine.
If you are using strict mode, eval is no good for you, and you need to go with the vm module and use its runInNewContext function. Then, the solution looks like:
var requireAsync = function (module, callback) {
fs.readFile(module, { encoding: 'utf8' }, function (err, data) {
var sandbox = {
module: {
exports: {}
}
};
var code = '(function (module) {' + data + '})(module)';
vm.runInNewContext(code, sandbox);
callback(null, sandbox.module.exports); // or sandbox.module…
});
};
The npm module async-require can help you to do this.
Install
npm install --save async-require
Usage
var asyncRequire = require('async-require');
// Load script myModule.js
asyncRequire('myModule').then(function (module) {
// module has been exported and can be used here
// ...
});
The module uses vm.runInNewContext(), a technique discussed in the accepted answer. It has bluebird as a dependency.
(This solution appeared in an earlier answer but that was deleted by review.)
Yes - export function accepting callback or maybe even export full featured promise object.
// foo.js + callback:
module.exports = function(cb) {
setTimeout(function() {
console.log('module loaded!');
var fooAsyncImpl = {};
// add methods, for example from db lookup results
fooAsyncImpl.bar = console.log.bind(console);
cb(null, fooAsyncImpl);
}, 1000);
}
// usage
require("./foo.js")(function(foo) {
foo.bar();
});
// foo.js + promise
var Promise = require('bluebird');
module.exports = new Promise(function(resolve, reject) {
// async code here;
});
// using foo + promises
require("./foo.js").then(function(foo) {
foo.bar();
});
Andrey's code below is the simplest answer which works, but his had a small mistake so I'm posting the correction here as answer. Also, I'm just using callbacks, not bluebird / promises like Andrey's code.
/* 1. Create a module that does the async operation - request etc */
// foo.js + callback:
module.exports = function(cb) {
setTimeout(function() {
console.log('module loaded!');
var foo = {};
// add methods, for example from db lookup results
foo.bar = function(test){
console.log('foo.bar() executed with ' + test);
};
cb(null, foo);
}, 1000);
}
/* 2. From another module you can require the first module and specify your callback function */
// usage
require("./foo.js")(function(err, foo) {
foo.bar('It Works!');
});
/* 3. You can also pass in arguments from the invoking function that can be utilised by the module - e.g the "It Works!" argument */
For anyone who uses ESM modules and top level await, this will just work out of the box without using callbacks to commonJS require or installing any packages like async-require.
// In foo.mjs
await doIOstuffHere();
export foo;
// in bar.mjs
import foo from "./foo.mjs";
foo.bar(); // this function would not run till the async work in foo.mjs is finished