Develop Reference

How can I run a js file from my nodejs server

I want to set up a server that scrapes data off an existing API, cleans it up and outputs it to my own server where I can manipulate the data further.
So far I've got all the pieces working.
My server runs.
I can scrape the data into the correct JSON format
I have a function that runs when a POST https request is called that writes JSON to my database.
Now I need to connect the two.
There are 3 main components:
the server.js file
The Post controller & schema file. This contains the function that handles the http request and constructs the object and writes to the database.
I have a separate file that contains a bunch of functions that scrape the data and prepare it in the proper JSON format.
My problem is getting scraping file to run. At the moment to test it I just run Node filename.js it worked fine when I was outputting the json to console.log. However now I've written in the http request. the problem is, when I run the server to open the http endpoint, I can't run the file.js that contains the scraper.
Essentially I want the scraper to run on a schedule, and send the data to the endpoint I've set up on the server.
Is there a way to replicate calling node filename.js in code? As per my understanding when you run node filename.js it just runs down the file, executing code as it finds it. Is there a way to do this? Or do I need to call each function 1 by 1 from my server file? Or do I just encapsulate the whole js file in a master function (my worry would then be that variables become siloed)?
I've posted the code below for reference:
server.js
require("dotenv").config(); // ALLOWS ENVIRONMENT VARIABLES TO BE SET ON PROCESS.ENV SHOULD BE AT TOP
const express = require("express");
const app = express();
const postRoutes = require("./routes/postRoutes.js");
const path = require('path');
// Middleware
app.use(express.json()); // parse json bodies in the request object
// Redirect requests to endpoint starting with /posts to postRoutes.js
app.use("/posts", postRoutes);
// Global Error Handler. IMPORTANT function params MUST start with err
app.use((err, req, res, next) => {
console.log(err.stack);
console.log(err.name);
console.log(err.code);
res.status(500).json({
message: "Something went rely wrong",
});
});
// Listen on pc port
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => console.log(`Server running on PORT ${PORT}`));
const battlesGetData = require(path.resolve(__dirname, "./battlesgetData.js")); // is this right?!?
battlesGetData.js
const fetch = require("node-fetch");
const postController = require("./controllers/postControllers");
//const fs = require('fs');
const distinct = (value, index, self) => {
return self.indexOf(value) === index;
}
async function getBattleHistory(player = '', data = {}) {
const battleHistory = await fetch('https://api2.splinterlands.com/battle/history?player=' + player)
.then((response) => {
if (!response.ok) {
throw new Error('Network response was not ok');
}
return response;
})
.then((battleHistory) => {
return battleHistory.json();
})
.catch((error) => {
console.error('There has been a problem with your fetch operation:', error);
});
return battleHistory.battles;
}
const extractGeneralInfo = (x) => {
return {
created_date: x.created_date ? x.created_date : '',
match_type: x.match_type ? x.match_type : '',
mana_cap: x.mana_cap ? x.mana_cap : '',
ruleset: x.ruleset ? x.ruleset : '',
inactive: x.inactive ? x.inactive : ''
}
}
const extractMonster = (team) => {
const monster1 = team.monsters[0];
const monster2 = team.monsters[1];
const monster3 = team.monsters[2];
const monster4 = team.monsters[3];
const monster5 = team.monsters[4];
const monster6 = team.monsters[5];
return {
summoner_id: team.summoner.card_detail_id,
summoner_level: team.summoner.level,
monster_1_id: monster1 ? monster1.card_detail_id : '',
monster_1_level: monster1 ? monster1.level : '',
monster_1_abilities: monster1 ? monster1.abilities : '',
monster_2_id: monster2 ? monster2.card_detail_id : '',
monster_2_level: monster2 ? monster2.level : '',
monster_2_abilities: monster2 ? monster2.abilities : '',
monster_3_id: monster3 ? monster3.card_detail_id : '',
monster_3_level: monster3 ? monster3.level : '',
monster_3_abilities: monster3 ? monster3.abilities : '',
monster_4_id: monster4 ? monster4.card_detail_id : '',
monster_4_level: monster4 ? monster4.level : '',
monster_4_abilities: monster4 ? monster4.abilities : '',
monster_5_id: monster5 ? monster5.card_detail_id : '',
monster_5_level: monster5 ? monster5.level : '',
monster_5_abilities: monster5 ? monster5.abilities : '',
monster_6_id: monster6 ? monster6.card_detail_id : '',
monster_6_level: monster6 ? monster6.level : '',
monster_6_abilities: monster6 ? monster6.abilities : ''
}
}
let battlesList = [];
usersToGrab = ["rus48-bot", "sbalani"]
const battles = usersToGrab.map(user =>
getBattleHistory(user)
.then(battles => battles.map(
battle => {
const details = JSON.parse(battle.details);
if (details.type != 'Surrender') {
if (battle.winner && battle.winner == battle.player_1) {
const monstersDetails = extractMonster(details.team1)
const info = extractGeneralInfo(battle)
return {
...monstersDetails,
...info,
battle_queue_id: battle.battle_queue_id_1,
player_rating_initial: battle.player_1_rating_initial,
player_rating_final: battle.player_1_rating_final,
winner: battle.player_1,
}
} else if (battle.winner && battle.winner == battle.player_2) {
const monstersDetails = extractMonster(details.team2)
const info = extractGeneralInfo(battle)
return {
...monstersDetails,
...info,
battle_queue_id: battle.battle_queue_id_2,
player_rating_initial: battle.player_2_rating_initial,
player_rating_final: battle.player_2_rating_final,
winner: battle.player_2,
}
}
}
})
).then(x => battlesList = [...battlesList, ...x])
)
Promise.all(battles).then(() => {
const cleanBattleList = battlesList.filter(x => x != undefined)
fetch("http://localhost:3000/posts/", {
method: "post",
body: cleanBattleList,
headers: {"Content-Type": "application/json"}
})
.then(json => console.log(json))
.catch(err => console.log(err))
/* fs.writeFile(`data/history.json`, JSON.stringify(cleanBattleList), function (err) {
if (err) {
console.log(err);
}
}); */
});
This is the POST function that gets called
exports.createNewPost = async (req, res, next) => {
/*
let { summoner_id, summoner_level,
monster_1_id, monster_1_level, monster_1_abilities,
monster_2_id, monster_2_level, monster_2_abilities,
monster_3_id, monster_3_level, monster_3_abilities,
monster_4_id, monster_4_level, monster_4_abilities,
monster_5_id, monster_5_level, monster_5_abilities,
monster_6_id, monster_6_level, monster_6_abilities,
created_date, match_type, mana_cap, ruleset, inactive,
battle_queue_id, player_rating_initial, player_rating_final, winner
} = req.body; // using postman this is what allows us to post JSON
let post = new PostBattle(summoner_id, summoner_level,
monster_1_id, monster_1_level, monster_1_abilities,
monster_2_id, monster_2_level, monster_2_abilities,
monster_3_id, monster_3_level, monster_3_abilities,
monster_4_id, monster_4_level, monster_4_abilities,
monster_5_id, monster_5_level, monster_5_abilities,
monster_6_id, monster_6_level, monster_6_abilities,
created_date, match_type, mana_cap, ruleset, inactive,
battle_queue_id, player_rating_initial, player_rating_final, winner); // the title & body defined in the previous line taken from the JSON are now deposited here.
*/
let json = req.body;
for (var obj in json) {
console.log(obj + ": " + json[obj]);
let post = new PostBattle(json[obj].summoner_id, json[obj].summoner_level,
json[obj].monster_1_id, json[obj].monster_1_level, json[obj].monster_1_abilities,
json[obj].monster_2_id, json[obj].monster_2_level, json[obj].monster_2_abilities,
json[obj].monster_3_id, json[obj].monster_3_level, json[obj].monster_3_abilities,
json[obj].monster_4_id, json[obj].monster_4_level, json[obj].monster_4_abilities,
json[obj].monster_5_id, json[obj].monster_5_level, json[obj].monster_5_abilities,
json[obj].monster_6_id, json[obj].monster_6_level, json[obj].monster_6_abilities,
json[obj].created_date, json[obj].match_type, json[obj].mana_cap, json[obj].ruleset, json[obj].inactive,
json[obj].battle_queue_id, json[obj].player_rating_initial, json[obj].player_rating_final, json[obj].winner);
console.log(post);
//let post = new PostBattle(json);
post = await post.save();
console.log("a post is happening");
}

How to send JavaScript object/array to front-end - node.js app

I am trying to send data to front end of my app to create a graph. I attach the array in the response object but this doesn't work, although I can use the user value that is also sent in the response object on the front end.
var scoreQuery = "SELECT q1, q2, q3, q4, q5, q6, q7, q8, q1, q2, q3, q4 FROM table WHERE id = user.id";
var scoreArray;
module.exports = function(app, passport){
app.get('/profile', isLoggedIn, function (req, res) {
connection.query(scoreQuery, function (err, result, fields) {
if (err) throw err;
getData(result);
console.log(scoreArray);
}, res.render('profile.ejs', {
user:req.user,
data: scoreArray
})
);
});
};
function getData(result){
Object.keys(result).forEach(function(key) {
const values = result[key];
scoreArray = Object.values(values);
});
});
Below is the public/javascripts/scripts.js file where I'm creating the graph.
/*Scripts*/
var quizCategory1 = data.scoreArray.slice(0,7);
var quizCategory2 = data.scoreArray.slice(8,11);
var cat1Total = totalScore(category1);
var cat2Total = totalScore(category2);
function totalScore(categoryScore){
return categoryScore = scoreArray.reduce((accum,scoreArray) =>
{
const splitValues = scoreArray.split('/');
return {
score:accum.score + parseInt(splitValues[0]),
maxScore:accum.maxScore + parseInt(splitValues[1]),
}
},{score:0,maxScore:0}
);
}
var ctx = document.getElementById("myChart").getContext('2d');
var barTotalCategoryScores = [cat1Total.score, cat2Total.score];
var labels = ["Java & Design", "Build & Versioning"];
var myChart = new Chart(ctx, {
type: 'bar',
data: {
labels: labels,
datasets: barTotalCategoryScores
}
}
});
the scoreArray prints out to the console whenever a user logsin so the sql query and getData() function are working. But when I try to use the data.scoreArray in my scripts.js file it doesn' work.

server :
// ... your routes
// + new route
app.get('/profile/:id', (req,res) => {
let id = req.params.id // e.g /profile/3 -> req.params.id = 3
let scoreArray;
// your query (it's very bad to set param in your query like this (injection SQL) )
let scoreQuery = `SELECT q1, q2, q3, q4, q5, q6, q7, q8, q1, q2, q3, q4 FROM
table WHERE id = ${id}`; // `... ${foo}` => https://developer.mozilla.org/fr/docs/Web/JavaScript/Reference/Litt%C3%A9raux_gabarits
connection.query(scoreQuery, function (err, result, fields) {
if (err) throw err;
getData(result);
console.log(scoreArray);
}, res.status(200).json({
user:req.user,
data: scoreArray
})
);
})
// ... your method
function getData(result){
Object.keys(result).forEach(function(key) {
const values = result[key];
scoreArray = Object.values(values);
});
});
client :
/* code to get data
let id_test = 1;
fetch(`http://localhost:<your_node_server_port>/profil/${id_test}`)
.then(response => response.json())
.then(json => console.log(json)) // return profil for id 1
*/
// e.g page is loaded
window.onload = () => {
let id_test = 1;
fetch(`http://localhost:<your_node_server_port>/profil/${id_test}`)
.then(response => response.json())
.then(json => console.log(json)) // return profil for id 1
}

Add new route:
app.get('/profile/:id', (req,res) => {
// your_array = query DB with param id (req.params.id) store in your array
res.status(200).json({data: <your_array>})
})
Then on your client make a GET query on /profile/:id (https://developer.mozilla.org/fr/docs/Web/API/XMLHttpRequest)
when the page loading (window.onload)
For MySQL, you should using :
http://docs.sequelizejs.com/
http://docs.sequelizejs.com/manual/getting-started.html#installing
http://docs.sequelizejs.com/manual/dialects.html#mysql
http://docs.sequelizejs.com/manual/querying.html

Thanks a million Sylvain, your explanation got me on the right track. I did the following...
//routes.js
app.get('/profile', function (req, res) {
let id = req.user.id;
getData(id, function (score) {
res.status(200).render('profile.ejs',{user: req.user, score})
});
});
I put my sql statement within the getData() function and used a '?' in place of id in my sql statement and send id parameter as value to insert into the query method. This has has worked and I can now get the score object on the client-side ejs file using <?= score.cat1Total.score ?>.
My problem now is still trying to use score.cat1Total.score in my public/javascripts/scripts.js file and it is undefined... and I'm not sure how to correctly implement your window.onload() method
// /public/javascripts/scripts.js
/* code to get data?
window.onload = () => {
fetch(`http://localhost:8080/profile)
.then(response => response.json())
.then(json => console.log(json))
}
*/
var ctx = document.getElementById("myChart").getContext('2d');
//This is where I am trying to get score objects values.
var barTotalCategoryScores = [score.cat1Total.score, cat2Total.score];
var labels = ["Java & Design", "Build & Versioning"];
var myChart = new Chart(ctx, {
type: 'bar',
data: {
labels: labels,
datasets: barTotalCategoryScores
}
}
});
FYI below is the getData(id, callback) method I used in routes
// routes.js
function getData(id, callback) {
let scoreQuery = "SELECT c1q1, c1q2, c1q3, c1q4, c1q5, c1q6, c1q7, c1q8, c2q1, c2q2, c2q3, c2q4 FROM nodejs_login.assessment_score AS a JOIN nodejs_login.users as u ON a.respondent_id = u.user_respondent_id WHERE a.respondent_id = ?";
connection.query(scoreQuery, [id],function (err, result) {
if (err) throw err;
Object.keys(result).forEach(function(key) {
let values = result[key];
let scoreArray = Object.values(values);
let category1 = scoreArray.slice(0,7);
let category2 = scoreArray.slice(8,11);
//parsing and accumlating each category score
var cat1Total = totalScore(category1);
var cat2Total = totalScore(category2);
//function to parse the strings into numbers and accumulate them
function totalScore(categoryScore){
return categoryScore.reduce((accum,scoreArray) =>
{
const splitValues = scoreArray.split('/');
return {
score:accum.score + parseInt(splitValues[0]),
maxScore:accum.maxScore + parseInt(splitValues[1]),
}
},{score:0,maxScore:0}
);
}
let categories = {cat1Total, cat2Total};
callback(categories);
});
})
}

Hyperledger query never return results

I`m trying to query my business network using buildQuery but it always returns an empty array.
My code is as follows.
This is the connection.js file:
module.exports = {
BusinessNetworkConnection : require('composer-client').BusinessNetworkConnection,
cardName : '',
connection: {},
connect : function() {
var cardType = { type: 'composer-wallet-filesystem' }
this.connection = new this.BusinessNetworkConnection(cardType);
return this.connection.connect(this.cardName);
},
disconnect : function(callback) {
this.connection.disconnect();
}
};
This is my query.js file which being invoked to get results:
const connection = require('./connection');
const getContacts = async (cardName,companyID) => {
connection.cardName = cardName;
try {
await connection.connect();
main();
} catch (error) {
main(error);
}
async function main(error) {
if (error) { return new Error("Ops Error: ",error) };
const statement = 'SELECT org.finance.einvoice.participant.Company WHERE (participantId == _$companyID)'
const query = await connection.connection.buildQuery(statement);
const company = await connection.connection.query(query, { companyID }).catch(err => {return new Error(err)});
await connection.connection.disconnect().catch(err => new Error(err));
console.log(company);
return company;
};
};
module.exports = {
getContacts
};
The expected behavior from getContacts() is to return an asset from business network but it actually returns an empty array.
Current versions: composer-cli 0.20 , composer-playground 0.20 , composer-client 0.20 , composer-common 0.20 and fabric-dev-server 1.2 .

i found the solution for this issue.
i was using card which was not allowed to perform queries. However, when i used the admin card it returned with results.
other way is to allow participants to issue queries in permission.acl file.

TypeError: Cannot read property 'json' of undefined google assistant

I'm working on a Bus Stop google assistant script in node.js
I based it on the weather API example by Google. Given the right API key, the weather function will work and return the weather for a place on a date.
The Bus Stop API will return the correct output in the console.log, but the output does not get passed on to the else if statement where the function is called.
I get 2 errors:
"Unhandled rejection" Which can be alleviated by commenting out the reject code in the callBusApi.
"TypeError: Cannot read property 'json' of undefined
at callBusApi.then.catch (/user_code/index.js:45:9)
at process._tickDomainCallback (internal/process/next_tick.js:135:7)" This is where it breaks. I think because it doesn't get the output from the function.
My script looks as follows:
'use strict';
const http = require('http');
const host = 'api.worldweatheronline.com';
const wwoApiKey = 'enter a working key';
exports.weatherWebhook = (req, res, re) => {
if(req.body.queryResult.intent['displayName'] == 'weather'){
// Get the city and date from the request
let city = req.body.queryResult.parameters['geo-city']; // city is a required param
// Get the date for the weather forecast (if present)
let date = '';
if (req.body.queryResult.parameters['date']) {
date = req.body.queryResult.parameters['date'];
console.log('Date: ' + date);
}
// Call the weather API
callWeatherApi(city, date).then((output) => {
res.json({ 'fulfillmentText': output }); // Return the results of the weather API to Dialogflow
}).catch(() => {
res.json({ 'fulfillmentText': `I don't know the weather but I hope it's good!` });
});
}
else if (req.body.queryResult.intent['displayName'] == 'mytestintent'){
callBusApi().then((output) => {
re.json({ 'fulfillmentText': output }); // Return the results of the bus stop API to Dialogflow
}).catch(() => {
re.json({ 'fulfillmentText': `I do not know when the bus goes.` });
});
}
};
function callBusApi () {
return new Promise((resolve, reject) => {
http.get({host: 'v0.ovapi.nl', path: '/stopareacode/beunav/departures/'}, (re) => {
let boy = '';
re.on('data', (d) => {boy+=d});
re.on('end',() => {
let response = JSON.parse(boy)
var firstKey = Object.keys(response['beunav']['61120250']['Passes'])[0];
var timeKey = Object.keys(response['beunav']['61120250']['Passes'][firstKey])[19];
var destKey = Object.keys(response['beunav']['61120250']['Passes'][firstKey])[1];
let destination = response['beunav']['61120250']['Passes'][firstKey][destKey];
let datetime = response['beunav']['61120250']['Passes'][firstKey][timeKey];
let fields = datetime.split('T');
let time = fields[1];
let output = `Next bus to ${destination} departs at ${time} .`;
console.log(output)
resolve(output);
});
re.on('error', (error) => {
console.log(`Error talking to the busstop: ${error}`)
reject();
});
});
});
};
function callWeatherApi (city, date) {
return new Promise((resolve, reject) => {
let path = '/premium/v1/weather.ashx?format=json&num_of_days=1' +
'&q=' + encodeURIComponent(city) + '&key=' + wwoApiKey + '&date=' + date;
console.log('API Request: ' + host + path);
http.get({host: host, path: path}, (res) => {
let body = '';
res.on('data', (d) => { body += d; });
res.on('end', () => {
let response = JSON.parse(body);
let forecast = response['data']['weather'][0];
let location = response['data']['request'][0];
let conditions = response['data']['current_condition'][0];
let currentConditions = conditions['weatherDesc'][0]['value'];
let output = `Current conditions in the ${location['type']}
${location['query']} are ${currentConditions} with a projected high of
${forecast['maxtempC']}°C or ${forecast['maxtempF']}°F and a low of
${forecast['mintempC']}°C or ${forecast['mintempF']}°F on
${forecast['date']}.`;
console.log(output);
resolve(output);
});
res.on('error', (error) => {
console.log(`Error calling the weather API: ${error}`)
reject();
});
});
});
}

It appears that your method has a parameter too much
exports.weatherWebhook = (req, res, re) => {
should be:
exports.weatherWebhook = (req, res) => {
And as well on the variable 're' used in the handling of the 'mytestintent' inside the webhook.
This explains the 'not defined' error when trying to set a json value on it.

Regarding your 2 question: It usually comes when the value of the variable is not defined.
First check wheather you have defined the JSON variable in your .js file.
Or its in some other format.

Angular 2 Synchronous File Upload

I am trying to upload a file to web api which takes the file as byte array using angular 2 application.
I am not able to pass the byte array from angular 2 page to web api. It looks like the File Reader read method is asynchronous. How do I make this as synchronous call or wait for the file content to be loaded before executing the next line of code?
Below is my code
//attachment on browse - when the browse button is clicked
//It only assign the file to a local variable (attachment)
fileChange = (event) => {
var files = event.target.files;
if (files.length > 0) {
this.attachment = files[0];
}
}
//when the submit button is clicked
onSubmit = () => {
//Read the content of the file and store it in local variable (fileData)
let fr = new FileReader();
let data = new Blob([this.attachment]);
fr.readAsArrayBuffer(data);
fr.onloadend = () => {
this.fileData = fr.result; //Note : This always "undefined"
};
//build the attachment object which will be sent to Web API
let attachment: Attachment = {
AttachmentId: '0',
FileName: this.form.controls["attachmentName"].value,
FileData: this.fileData
}
//build the purchase order object
let order: UpdatePurchaseOrder = {
SendEmail: true,
PurchaseOrderNumber: this.form.controls["purchaseOrderNumber"].value,
Attachment: attachment
}
//call the web api and pass the purchaseorder object
this.updatePoService
.updatePurchaseOrder(this.form.controls["purchaseOrderRequestId"].value, order)
.subscribe(data => {
if (data) {
this.saveSuccess = true;
}
else {
this.saveSuccess = false;
}
},
error => this.errors = error,
() => this.res = 'Completed'
);
}
Any hint would be useful.
regards,
-Alan-

You cannot make this async call synchronous. But you can take advantage of the observables to wait for the files to be read:
//when the submit button is clicked
onSubmit = () => {
let file = Observable.create((observer) => {
let fr = new FileReader();
let data = new Blob([this.attachment]);
fr.readAsArrayBuffer(data);
fr.onloadend = () => {
observer.next(fr.result);
observer.complete()
};
fr.onerror = (err) => {
observer.error(err)
}
fr.onabort = () => {
observer.error("aborted")
}
});
file.map((fileData) => {
//build the attachment object which will be sent to Web API
let attachment: Attachment = {
AttachmentId: '0',
FileName: this.form.controls["attachmentName"].value,
FileData: fileData
}
//build the purchase order object
let order: UpdatePurchaseOrder = {
SendEmail: true,
PurchaseOrderNumber: this.form.controls["purchaseOrderNumber"].value,
Attachment: attachment
}
return order;
})
.switchMap(order => this.updatePoService.updatePurchaseOrder(this.form.controls["purchaseOrderRequestId"].value, order))
.subscribe(data => {
if (data) {
this.saveSuccess = true;
} else {
this.saveSuccess = false;
}
},
error => this.errors = error,
() => this.res = 'Completed'
);
}

I arrived here looking for a solution for a similar issue. I'm performing requests to an endpoint which can response a binary blob if anything goes well or a JSON file in event of error.
this.httpClient.post(urlService, bodyRequest,
{responseType: 'blob', headers: headers})
.pipe(map((response: Response) => response),
catchError((err: Error | HttpErrorResponse) => {
if (err instanceof HttpErrorResponse) {
// here, err.error is a BLOB containing a JSON String with the error message
} else {
return throwError(ErrorDataService.overLoadError(err, message));
}
}));
As FileReaderSync apparently doesn't work in Angular6 I took n00dl3's solution (above) to throw the error after parsing the Blob content:
return this.httpClient.post(urlService, bodyRequest,
{responseType: 'blob', headers: headers})
.pipe(map((response: Response) => response),
catchError((err: Error | HttpErrorResponse) => {
const message = `In TtsService.getTts(${locale},${outputFormat}). ${err.message}`;
if (err instanceof HttpErrorResponse) {
const $errBlobReader: Observable<HttpErrorResponse> = Observable.create((observer) => {
const fr = new FileReader();
const errorBlob = err.error;
fr.readAsText(errorBlob, 'utf8');
fr.onloadend = () => {
const errMsg = JSON.parse(fr.result).message;
const msg = `In TtsService.getTts(${locale},${outputFormat}). ${errMsg}`;
observer.error(ErrorDataService.overLoadError(err, msg));
};
fr.onerror = (blobReadError) => {
observer.error(blobReadError);
};
fr.onabort = () => {
observer.error('aborted');
};
});
return $errBlobReader;
} else {
return throwError(ErrorDataService.overLoadError(err, message));
}
}));
Thanks! You really saved my day!