Develop Reference

Cheerio not working. What am I doing wrong?

I am trying to scrape a classified ad search result page.
I have tried console logging everything I can to make sure I am getting a response, which I am, but when I actually use cheerio to query something I don't get anything back. For instance if I just query for how many children using $('.listing-group').children('section').length I get back 0 instead of 24 when I console log it.
Here is what I'm doing. Pardon the long URL.
const request = require("request");
const cheerio = require("cheerio");
const app = express();
app.get("/scrape", function(req, res) {
url =
"http://classifieds.ksl.com/search/?keyword=code&category%5B%5D=Books+and+Media&zip=&miles=25&priceFrom=&priceTo=&city=&state=&sort=";
request(url, function(error, response, html) {
if (!error) {
let $ = cheerio.load(html);
let test = $("#search-results").find("section").length
console.log(test);
} else {
console.log("there has been an error");
}
res.send("Check the console.");
});
});
app.listen("8081");
console.log("check localhost:8081/scrape");
exports = module.exports = app;
I'm new to cheerio so I'm assuming I'm probably making a simple error, but with all the tutorials I've checked, I can't seem to find an answer.
Ultimately I want to grab each result on the page (found in the 'section' tags) to display the data for a personal project I'm working on.

It looks like:
JSON.parse(html.match(/listings: (\[.*\])/)[1])

sending key/value data via POST method in a REST API

I'm working on a REST API that includes a POST method to create new widgets.
The data that I need to send looks like this:
key | value
widgetnum | 12345
02:00:00_03:00:00_mtwrfsn | johndoe#myemail.com
12:00:00_15:00:00_mt | janedoe#yahoo.com
Up until now, I have been POSTing data via two keys in the BODY:
widgetnum : 12345
tc : 02:00:00_03:00:00_mtwrfsn_johndoe#myemail.com,12:00:00_15:00:00_mt_janedoe#yahoo.com
As you can see in the example above, the key "tc" actually contains information for two records, just separated by a comma.
But yesterday as i was poking around in the Postman application... I realized that I can send multiple key / value pairs. I don't know why this never clicked before. (such a noob) In any case, I realize now, I can actually do this:
widgetnum : 12345
02:00:00_03:00:00_mtwrfsn : johndoe#myemail.com
12:00:00_15:00:00_mt : janedoe#yahoo.com
Questions
Is it more efficient to change way I POST data as far as my javascript code is concerned? So for example, today, my code looks like this:
router.post('/', function(req, res, next) {
var widgetnum = req.body.widgetnum;
var tc = req.body.tc;
var tc_array = tc.split(",");
aka - I'm able to look for my data using very specific key names.
However, if I change the way I POST my data, I believe I would have to loop to find all the keys.
So something like this:
router.post('/', function(req, res, next) {
//loop through req.body object.
for (var key in req.body) {
if (req.body.hasOwnProperty(key)) {
// add some logic here to test if the key is
// widgetnum or a tc type key.
console.log(key + " -> " + req.body[key]);
}
}
res.status(200).send('ok');
return;
});
I apologize if this seems like a remedial question - it probably is.
But if it's bad form to POST multiple tc records in one key/value pair, I'm happy to change it. Perhaps there's a more efficient way to write my javascript code to find the data.
Thanks.

To send and receive key/value pair, format already exist that (almost) does the work for you. One of them is JSON, which is widely supported, especially in JavaScript.
Since you are using express already, you can add body-parser and set it to automatically parse JSON:
Data sent (POSTMan: body type: application/json, copy paste in the body.
{
"widgetnum" : 12345,
"tc" : {
"02:00:00_03:00:00_mtwrfsn": johndoe#myemail.com,
"12:00:00_15:00:00_mt": janedoe#yahoo.com
}
}
Server code:
"use strict";
const express = require('express'),
router = express.Router(),
bodyParser = require('body-parser');
router.use(bodyParser.json());
router.post('/', function(req, res, next) {
var widgetnum = req.body.widgetnum;
var tc = req.body.tc;
//No more need for tc_array
//...
}
No more need for your custom format, and req.body will be a JS object of the same form as the JSON sent, so you can do things like:
console.log(req.body.tc['02:00:00_03:00:00_mtwrfsn']); //print johndoe#myemail.com
console.log(Object.keys(req.body.tc)); //Print ['02:00:00_03:00:00_mtwrfsn', '12:00:00_15:00:00_mt']

Node.js - JSON response to request cut off

I've distilled my issue to some really basic functionality here. Basically, we're sending a request to a server (you can go ahead and c/p the URL and see the json document we get in response).
We get the response, we pipe it into a write stream and save it as a .json file - but the problem is that the file keeps being cut off. Is the .json file too large? Or am I missing something? Node.js newbie - massively appreciate any help I can get.
var fs = require('fs');
var url = 'https://crest-tq.eveonline.com/market/10000002/history/?type=https://crest-tq.eveonline.com/inventory/types/34/'
var request = require('request');
request(url).pipe(fs.createWriteStream('34_sell.json'));

var fs = require('fs');
var request = require('request');
var url = 'https://crest-tq.eveonline.com/market/10000002/history/?type=https://crest-tq.eveonline.com/inventory/types/34/';
request(url).pipe(fs.createWriteStream('34_sell.json'));
Not an answer but this is the code I used. I'm using request version 2.74.0. And node version v5.4.1.

Try writing a get request to the url and send the json as response and write an error handling statement like if err then throw err..console log and see the result..hope it works

How to handle a get request with node.js (express)

I'm a node.js newbie and I'm creating my first big app with it (I'm using express). I need to have my webpage perform some javascript canvas-drawing when the user loads an id with a get request, e.g.
www.mywebsite.com/page?id=22
I know I can handle this with a simple
var express = require("express");
var app = express();
app.get('handle',function(request,response){
//request.id
});
but I don't know how to start my webpage with the asked drawing for that id. Every tutorial on the internet on express and get explains how to handle get requests... well this question is about "what happens next?"
Rephrased: I'm not sure how should I tell the html page "you need to draw what is associated with this id" from express and then send that page back to the user.

You can take the id from params and after this to return a response based on that id.
var express = require('express');
var app = express();
app.get("/page/:id",function(request, response){
var id = request.params.id;
// do something with id
// send a response to user based on id
var obj = { id : id, Content : "content " +id };
response.writeHead(200, {"Content-Type": "application/json"});
response.write(JSON.stringify(obj));
});
Notes:
You use /page/:id to make urls like www.mywebsite.com/page/22 or www.mywebsite.com/page?id=22 and you can have acces to id on server with request.params.id (output: 22).
With response you can write a response to the server. In this example i returned a json object.
In writeHead 200 come from status which means OK , and content-type means that I return a json object
You can return what you want, a page or something else, this is just an example (PoC).

If you want to pass multiple variables in the request, you can pass it in the following way:
var emailVar = "someEmail#gmail.com";
var nameVar = "someName";
var url = `/home?email=${emailVar}&name=${nameVar}`;
//Now make the request.
and in the backend logic, you can retrieve these values as:
app.get('/home', function(request, response)
{
console.log(request.query.email+" "+request.query.name);
var email = request.query.email;
var name = request.query.name;
response.setHeader('Content-Type', 'application/json');
if(request.query.email)
{
response.send(JSON.stringify({
message: 'Got the email'
}));
}
else
{
response.send(JSON.stringify({
message: 'No email sent'
}));
}
});
This approach is useful for performing query operations in the backend.

How to get readable string from mysql with node.js?

When user register, i save its name in mysql (russian name in database seems like - ÐрминÑÐ). When I take data from database with PHP and print out, it works good (show russian letters), but when user connect to node.js server (using socket.io), make mysql query (using node-mysql module) and receive data from query, then his name seems like - ÐрминÑÐ.
How to get readable letters?
Here are some lines from my server file:
var mysql = require('mysql');
var db = mysql.createConnection({host:'x', user:'x', password:'x', database:'x', charset:'UTF8_GENERAL_CI'});
db.connect();
var http = require('http');
var server = http.createServer(function(req,res){
req.setEncoding("utf8");
}).listen(8080);
var io = require('socket.io').listen(server);

I think this should help:
function onRequest(request, response) {
...
request.setEncoding("utf8");
...
}
http.createServer(onRequest).listen(8888);

Here is what I'm using in the front-end:
function encode_utf8(s) {
return unescape(encodeURIComponent(s));
}
function decode_utf8(s) {
return decodeURIComponent(escape(s));
}
So, try passing your string to
decodeURIComponent(escape(name));

I had the same problem.
Try to debug it. If console.log data from fronted have incorrect charset, just update your socket.io script.