I am very new to Node.js and how it's callbacks work exactly, I am trying to find some good documentation on it but it's just not clicking for me yet. I'm coming from python so I'll show an example of what I'm use to doing in python, I'm not sure if it's possible in node though
def getRequest(link):
url = urllib.request.urlopen(link).read().decode()
return url
class urlData:
def __init__(self, link):
self.results = getRequest(link)
I'm not sure if node can do this because it's async ways, or is it possible? I'm not sure how to go about this the correct way, how would I replicate this action in node? If not can the this code be toyed with to get similar results, a way to set the variable with the data that is going to come?
The way you might do this in node is the following:
Install Request. https://github.com/mikeal/request
var request = require('request');
Now we have a simple http client.
var data;
request.get({url:url, json:true}, function (e, r, body) {
// this will get called when the response returns. Your app will continue before this is called.
data = body; // i have no idea what you want to do with it
console.log(body); // should be json
})
Related
Brief Project Overview
I am working on a full page caching system using Node and Express. I have this working, however I am trying to add cache "hole-punching", where portions of a document can have different caching policies.
Attempted Approach
I am attempting to achieve this by injecting the rendered output of one Express route into the rendered output of another Express route. Unfortunately, the most popular methods of firing an http request within Node involve asynchronous promises. I have not been able to get promises working in a synchronous request because... well, they don't work that way.
From what I can tell, I am left with 2 options:
Make the entire route/request/response asynchronous (is this possible?)
Find a synchronous way of rendering an Express route
I don't know how to achieve either of these. I know many folks will say not to do #2 because performance - I'm open to other suggestions.
Code
In the code below, I override the res.render() method to analyze the output of the current response and determine if there is a secondary request that should be made, rendered, and injected.
async function getRemoteInclude(req, content) {
// remote include format: {{ includeRemote('path') }}
const handlebarRegex = '(?<={{)(.*?)(?=}})';
const includeRegex = '(?<=includeRemote[(])(.*?)(?=[)])';
const replaceHandlebarRegex = /\{\{[^\}]+\}\}/g;
let parsedContent = content;
let foundHandlebars = content.match(handlebarRegex); // does string with {{ * }} exist?
if (foundHandlebars) {
let foundInclude = foundHandlebars[0].match(includeRegex); // does string with includeRemote('*') exist?
if (foundInclude) {
const axios = require('axios');
let path = 'http://expresscache.com:3000/test'; // sample hardcoded route
let response = await axios.get(path); // fire async request to express route
return parsedContent.replace(replaceHandlebarRegex, response.body); // replace remote include string with route response
}
}
return content;
}
function interceptRender() {
return function (req, res, next) {
res.render = function (view, options, callback) {
var self = this;
options = options || res.locals;
self.app.render(view, options, function (err, content) {
if (!callback) {
content = getRemoteInclude(req, content); // check if content should be modified with rendered remote include
self.locals.body = content;
self.send(content);
} else {
callback();
}
});
}
next();
}
}
module.exports = {
interceptRender: interceptRender
};
Obviously this code performs an async request through use of the axios node library. I am expecting that this will not be part of the final solution, unless option 1 from above is possible.
Ask
Does anyone have an idea how to either get this async request working in a synchronous context or to modify this request so that it is synchronous itself? Or maybe a completely different approach?
I'm wondering if there's any way to listen for console messages and act on console messages when they're received. Mainly, is there any way to do this without an external module, and using the http module?
The goal is to trigger a NodeJS function or code snippet on an event like click in the HTML. If there's also a way to do this, then that's great. But once again, I'd like to do this without an external module, and just use those that are built-in to NodeJS.
Use onclick() function in JavaScript to trigger a function call when clicking on a element. Then use fetch to make a api call to the nodejs server.
I know #Haris Wilson already got the answer, but I'd just like to provide a code example.
Instead of trying to catch a console message and then execute a function if we find it, we can use fetch() to make a request to whatever URL we need, and this can allow us to make other requests.
In this case, we can use the url module and the http module to parse the url and serve the API and website, respectively.
const url = require('url')
const http = require('http')
const requestListener = async function (req, res) {
// Basic server setup
res.writeHead(200, {
'Content-Type': 'text/html'
});
res.end(/** Content here */)
// API
if (url.parse(req.url, true).pathname === '/APIcall') {
let arguments = url.parse(req.url, true).query
// Preform necassary actions here
}
}
We can now use onClick to call a function inside our webpage JavaScript, and use fetch([API URL]) to give our NodeJS data to preform an action. We can use URL params to do this, such as https://localhost:8080/APIcall?data=someData&moreParam=more-data, where ?data=someData&moreParam=more-data are the URL params.
I'm using node.js simply so that I can run scheduled tasks and use GET requests. I'll paste some code that displays what I want to do, although it doesn't work for an obvious reason:
const http = require("http");
const request = require("request");
http.createServer(function (req, res) {
res.writeHead(200, {"Content-Type": "text/html"});
res.write("Hello, World!");
let a = getRequest();
console.log(a);
res.end();
}).listen(8080);
function getRequest() {
let b;
request("http://www.google.com", function(err, res, body) {
b = body;
})
return b;
}
So the b from the body does not work due to how the request is asynchronous and this leaves b as undefined when it is eventually printed. I know the way these callback functions are supposed to be used is to keep chaining of the callback function since that's the only place where the contents of body can be accessed. However, I don't want to keep chaining off functions because it completely destroys the structure of the program. I want to keep all my node server commands inside the http.createServer block. I don't want to place them in functions called from inside the callback function. In this example it doesn't really make sense for the process to be asynchronous since there's only 1 get request anyway and it can't be displayed in console.log until it's received anyway.
I just need a simple way to scrape data with get requests. What would be perfect is if I had some function that I could give a bunch of links, it gets the raw html from them, and then it waits for them to all be done so that I can process all the data at once.
How can something like this be implemented in Node.js?
You can do that using this module: sync-request.
With this module you will be able to make synchronous web requests from your NodeJS code.
Newbie question while trying to understand code created by others. Believe me I tried to understand this. Here goes..
For what reason would someone still call functions like .qs() and .json() in Request - module after we got what we need with .post() and sent the response already. They can't affect the request.post as they are called afterwards, can they?
With my skills I'm not able to understand from response module API docs (v2.22.0) what these actually do.
This is not the whole code but I tried to get the important parts here:
// When request comes to /getthisapge, make external query and return data in JSON format.
var request = require('request');
module.exports = function(app) {
app.get('/getthispage', function(req, res, next) {
var filter = {};
var query = {};
filter.category = req.query.category;
query.onBehalf = req.query.onBehalf;
request.post(URIandoptions, function(error, response, body) {
res.json(body.members)
}).qs(query).json(filter);
}
}
Without knowing exactly what the post function does (unnecessary to your question), you need to look at the order of execution.
request.post(URIandoptions, function (error, response, body){
res.json(body.members)
})
.qs(query) // ?
.json(filter); // ?
The function passed into post() does not get called at that specific moment. It is given to the post() function to do with as it pleases. This means technically that the function may never be called (depends on the api).
qs() and json() both get called upon the returning of the prior function. Usually this type of api means the following:
call post(), passing in a function to be run on completion
call qs() to setup the query details
call json() to tell the post function how to act, which in turn executes the actual post, running the completion function after data has been retrieved.
I've node app in Meteor.js and short python script using Pafy.
import pafy
url = "https://www.youtube.com/watch?v=AVQpGI6Tq0o"
video = pafy.new(url)
allstreams = video.allstreams
for s in allstreams:
print(s.mediatype, s.extension, s.quality, s.get_filesize(), s.url)
What's the most effective way of connecting them so python script get url from node.js app and return back output to node.js? Would it be better to code it all in Python instead of Meteor.js?
Well, there are plenty of ways to do this, it depends on your requirements.
Some options could be:
Just use stdin/stdout and a child process. In this case, you just need to get your Python script to read the URL from stdin, and output the result to stdout, then execute the script from Node, maybe using child_process.spawn. This is I think the simplest way.
Run the Python part as a server, let's say HTTP, though it could be anything as long as you can send a request and get a response. When you need the data from Node, you just send an HTTP request to your Python server which will return you the data you need in the response.
In both cases, you should return the data in a format that can be parsed easily, otherwise you are going to have to write extra (and useless) logic just to get the data back. Using JSON for such things is quite common and very easy.
For example, to have your program reading stdin and writing JSON to stdout, you could change your script in the following way (input() is for Python 3, use raw_input() if you are using Python 2)
import pafy
import json
url = input()
video = pafy.new(url)
data = []
allstreams = video.allstreams
for s in allstreams:
data.append({
'mediatype': s.mediatype,
'extension': s.extension,
'quality': s.quality,
'filesize': s.get_filesize(),
'url': s.url
})
result = json.dumps(data)
print(result)
Here is a very short example in NodeJS using the Python script
var spawn = require('child_process').spawn;
var child = spawn('python', ['my_script.py']);
child.stdout.on('data', function (data) {
var parsedData = JSON.parse(data.toString());
console.log(parsedData);
});
child.on('close', function (code) {
if (code !== 0) {
console.log('an error has occurred');
}
});
child.stdin.write('https://www.youtube.com/watch?v=AVQpGI6Tq0o');
child.stdin.end();