Develop Reference

Accessing the data from a Node JS GET request without chaining callback functions

I'm using node.js simply so that I can run scheduled tasks and use GET requests. I'll paste some code that displays what I want to do, although it doesn't work for an obvious reason:
const http = require("http");
const request = require("request");
http.createServer(function (req, res) {
res.writeHead(200, {"Content-Type": "text/html"});
res.write("Hello, World!");
let a = getRequest();
console.log(a);
res.end();
}).listen(8080);
function getRequest() {
let b;
request("http://www.google.com", function(err, res, body) {
b = body;
})
return b;
}
So the b from the body does not work due to how the request is asynchronous and this leaves b as undefined when it is eventually printed. I know the way these callback functions are supposed to be used is to keep chaining of the callback function since that's the only place where the contents of body can be accessed. However, I don't want to keep chaining off functions because it completely destroys the structure of the program. I want to keep all my node server commands inside the http.createServer block. I don't want to place them in functions called from inside the callback function. In this example it doesn't really make sense for the process to be asynchronous since there's only 1 get request anyway and it can't be displayed in console.log until it's received anyway.
I just need a simple way to scrape data with get requests. What would be perfect is if I had some function that I could give a bunch of links, it gets the raw html from them, and then it waits for them to all be done so that I can process all the data at once.
How can something like this be implemented in Node.js?

You can do that using this module: sync-request.
With this module you will be able to make synchronous web requests from your NodeJS code.

Why modify request after request.post

Newbie question while trying to understand code created by others. Believe me I tried to understand this. Here goes..
For what reason would someone still call functions like .qs() and .json() in Request - module after we got what we need with .post() and sent the response already. They can't affect the request.post as they are called afterwards, can they?
With my skills I'm not able to understand from response module API docs (v2.22.0) what these actually do.
This is not the whole code but I tried to get the important parts here:
// When request comes to /getthisapge, make external query and return data in JSON format.
var request = require('request');
module.exports = function(app) {
app.get('/getthispage', function(req, res, next) {
var filter = {};
var query = {};
filter.category = req.query.category;
query.onBehalf = req.query.onBehalf;
request.post(URIandoptions, function(error, response, body) {
res.json(body.members)
}).qs(query).json(filter);
}
}

Without knowing exactly what the post function does (unnecessary to your question), you need to look at the order of execution.
request.post(URIandoptions, function (error, response, body){
res.json(body.members)
})
.qs(query) // ?
.json(filter); // ?
The function passed into post() does not get called at that specific moment. It is given to the post() function to do with as it pleases. This means technically that the function may never be called (depends on the api).
qs() and json() both get called upon the returning of the prior function. Usually this type of api means the following:
call post(), passing in a function to be run on completion
call qs() to setup the query details
call json() to tell the post function how to act, which in turn executes the actual post, running the completion function after data has been retrieved.

Unexpected "write after end" error in express

I am trying to proxy an api call from client side through my server for some third party service, reasons for this being CORS issues and adding in a secret key on the server side. I usually get it done in the following way:
app.use('/someService', (req, res) => {
let url = `https://example.com/${config.SECRET_KEY}/endpoint${req.url}`
req.pipe(request(url).pipe(res))
})
this way I can use any ajax library on the client side and perform get request for example: get: '/someService/params' and it would normally go through and perform that request and then pipe it back. However now I started getting:
Error: write after end
in express and I am not entirely sure about what could be causing it.

Your piping is wrong. As it is now, you're piping to res twice (.pipe() returns the argument passed to it for chainability).
Instead try this:
req.pipe(request(url)).pipe(res)
I should point out however that properly proxying the HTTP response is not quite that simple since currently this line will always respond with HTTP status code 200, no matter what the remote server for the middle request responds with. Also, any headers from that response will not be sent to res. With that in mind, you could naively try something like:
var proxiedRes = req.pipe(request(url));
proxiedRes.on('response', function(pres) {
res.writeHead(pres.statusCode, pres.headers);
// You will want to add a `pres` 'error' event handler too in case
// something goes wrong while reading the proxied response ...
pres.pipe(res);
});

In Node JS how to pass a variable to http.request callback?

I call http.request().end() and I need to pass a variable to the callback function.
How do I do this?
For my situation, a user accesses the server, the server sends an HTTP request to our API, and then the server needs to respond to that user with information returned from the HTTP request. I can't figure out a way to do that without using a global variable, which would be an issue, since if 2 people were to do this at once, both messages would be sent to the same user
Not sure how it would help but here's kind what I'm trying to do..
callback = function(response){
//...
respondToUser(userId);
}
function doStuff(userId){
http.request({host:'www.google.com'}, callback).end();
}

It is called a "closure" where you are keeping the context with the function you are writing, like this
function doStuff(userId){
var callback = function(response){
//...
respondToUser(userId);
}
http.request({host:'www.google.com'}, callback).end();
}

What is the difference between dnode and nowjs?

How do the two compare to each other?

TL;DR
DNode
provides RMI;
remote functions can accept callbacks as arguments;
which is nice, since it is fully asynchronous;
runs stand-alone or through an existing http server;
can have browser and Node clients;
supports middleware, just like connect;
has been around longer than NowJS.
NowJS
goes beyond just RMI and implements a "shared scope" API. It's like
Dropbox, only with variables and functions instead of files;
remote functions also accept callbacks (thanks to Sridatta and Eric from NowJS
for the clarification);
depends on a listening http server to work;
can only have browser clients;
became public very recently;
is somewhat buggy right now.
Conclusion
NowJS is more of a toy right now -- but keep a watch as it matures. For
serious stuff, maybe go with DNode. For a more detailed review of these
libraries, read along.
DNode
DNode provides a Remote Method Invocation framework. Both the client and server
can expose functions to each other.
// On the server
var server = DNode(function () {
this.echo = function (message) {
console.log(message)
}
}).listen(9999)
// On the client
dnode.connect(9999, function (server) {
server.echo('Hello, world!')
})
The function that is passed to DNode() is a handler not unlike the one passed to
http.createServer. It has two parameters: client can be used to access the
functions exported by the client and connection can be used to handle
connection-related events:
// On the server
var server = DNode(function (client, connection) {
this.echo = function (message) {
console.log(message)
connection.on('end', function () {
console.log('The connection %s ended.', conn.id)
})
}
}).listen(9999)
The exported methods can be passed anything, including functions. They are properly
wrapped as proxies by DNode and can be called back at the other endpoint. This is
fundamental: DNode is fully asynchronous; it does not block while waiting
for a remote method to return:
// A contrived example, of course.
// On the server
var server = DNode(function (client) {
this.echo = function (message) {
console.log(message)
return 'Hello you too.'
}
}).listen(9999)
// On the client
dnode.connect(9999, function (server) {
var ret = server.echo('Hello, world!')
console.log(ret) // This won't work
})
Callbacks must be passed around in order to receive responses from the other
endpoint. Complicated conversations can become unreadable quite fast. This
question discusses possible solutions for this problem.
// On the server
var server = DNode(function (client, callback) {
this.echo = function (message, callback) {
console.log(message)
callback('Hello you too.')
}
this.hello = function (callback) {
callback('Hello, world!')
}
}).listen(9999)
// On the client
dnode.connect(9999, function (server) {
server.echo("I can't have enough nesting with DNode!", function (response) {
console.log(response)
server.hello(function (greeting) {
console.log(greeting)
})
})
})
The DNode client can be a script running inside a Node instance or can be
embedded inside a webpage. In this case, it will only connect to the server that
served the webpage. Connect is of great assistance in this case. This scenario was tested with all modern browsers and with Internet Explorer 5.5 and 7.
DNode was started less than a year ago, on June 2010. It's as mature as a Node
library can be. In my tests, I found no obvious issues.
NowJS
NowJS provides a kind of magic API that borders on being cute. The server has an
everyone.now scope. Everything that is put inside everyone.now becomes
visible to every client through their now scope.
This code, on the server, will share an echo function with every client that
writes a message to the server console:
// Server-side:
everyone.now.echo = function (message) {
console.log(message)
}
// So, on the client, one can write:
now.echo('This will be printed on the server console.')
When a server-side "shared" function runs, this will have a now attribute
that is specific to the client that made that call.
// Client-side
now.receiveResponse = function (response) {
console.log('The server said: %s')
}
// We just touched "now" above and it must be synchronized
// with the server. Will things happen as we expect? Since
// the code is not multithreaded and NowJS talks through TCP,
// the synchronizing message will get to the server first.
// I still feel nervous about it, though.
now.echo('This will be printed on the server console.')
// Server-side:
everyone.now.echo = function (message) {
console.log(message)
this.now.receiveResponse('Thank you for using the "echo" service.')
}
Functions in NowJS can have return values. To get them, a callback must be
passed:
// On the client
now.twice(10, function (r) { console.log(r) }
// On the server
everyone.now.twice = function(n) {
return 2 * n
}
This has an implication if you want to pass a callback as an honest argument (not
to collect a return value) -- one must always pass the return value collector, or
NowJS may get confused. According to the developers, this way of retrieving the
return value with an implicit callback will probably change in the future:
// On the client
now.crunchSomeNumbers('compute-primes',
/* This will be called when our prime numbers are ready to be used. */
function (data) { /* process the data */ },
/* This will be called when the server function returns. Even if we
didn't care about our place in the queue, we'd have to add at least
an empty function. */
function (queueLength) { alert('You are number ' + queueLength + ' on the queue.') }
)
// On the server
everyone.now.crunchSomeNumbers = function(task, dataCallback) {
superComputer.enqueueTask(task, dataCallback)
return superComputer.queueLength
}
And this is it for the NowJS API. Well, actually there are 3 more functions that
can be used to detect client connection and disconnection. I don't know why they
didn't expose these features using EventEmitter, though.
Unlike DNode, NowJS requires that the client be a script running inside a web browser.
The page containing the script must be served by the same Node that is running
the server.
On the server side, NowJS also needs an http server listening. It must be passed
when initializing NowJS:
var server = http.createServer(function (req, response) {
fs.readFile(__dirname + '/now-client.html', function (err, data) {
response.writeHead(200, {'Content-Type':'text/html'})
response.write(data)
response.end()
})
})
server.listen(8080)
var everyone = now.initialize(server)
NowJS first commit is from a couple weeks ago (Mar 2011). As such, expect it to
be buggy. I found issues myself while writing this answer. Also expect its
API to change a lot.
On the positive side, the developers are very accessible -- Eric even guided me
to making callbacks work. The source code is not documented, but is fortunately
simple and short and the user guide and examples are enough to get one started.

NowJS team member here. Correction to andref's answer:
NowJS fully supports "Remote Method Invocation". You can pass functions as arguments in remote calls and you can have functions as return values as well.
These functions are wrapped by NowJS just as they are in DNode so that they are executed on the machine on which the function was defined. This makes it easy to expose new functions to the remote end, just like in DNode.
P.S. Additionally, I don't know if andref meant to imply that remote calls are only asynchronous on DNode. Remote calls are also async on NowJS. They do not block your code.

Haven't tried Dnode so my answer is not a comparison. But I would like to put forth few experiences using nowjs.
Nowjs is based on socket.io which is quite buggy. I frequently experience session time-outs, disconnects and now.ready event firing multiple times in a short duration. Check out this issue on nowjs github page.
Also I found using websockets unviable on certain platforms, however this can be circumvented by explicitly disabling websockets.
I had planned creating a production app using nowjs but it seems its not mature enough to be relied upon. I will try dnode if it serves my purpose, else I will switch to plain-old express.
Update:
Nowjs seems to be scrapped. No commits since 8 months.