Develop Reference

My Query Quagmire: My node.js program works just fine, except when I try to execute queries, or filtered HTTP requests. Why?

I have been working on the backend of my app. At this point, it can access all data in a data base, and output it. I'm trying to implement some queries, so that the user can filter out the content that is returned. My DAL/DAO, looks like this
let mflix //Creates a variable used to store a ref to our DB
class MflixDAO {
static async injectDB(conn){
if(mflix){
return
}
try{
mflix = await conn.db(process.env.JD_NS).collection("movies")
}catch(e){
console.error('Unable to establish a collection handle in mflixDAO: ' + e)
}
}
// Creates a query to fetch data from the collection/table in the DB
static async getMovies({
mflix.controller
filters = null,
page = 0,
moviesPerPage = 20,
} = {}) {
let query
if (filters){
// Code
if("year" in filters){
query = {"year": {$eq: filters["year"]}}
}
// Code
}
// Cursor represents the returned data
let cursor
try{
cursor = await mflix.find(query)
}catch(e){
console.error('Unable to issue find command ' + e)
return {moviesList: [], totalNumMovies: 0}
}
const displayCursor = cursor.limit(moviesPerPage).skip(moviesPerPage * page)
try{
const moviesList = await displayCursor.toArray() // Puts data in an array
const totalNumMovies = await mflix.countDocuments(query) // Gets total number of documents
return { moviesList, totalNumMovies}
} catch(e){
console.error('Unable to convert cursor to array or problem counting documents ' + e)
return{moviesList: [], totalNumMovies: 0}
}
}
}
export default MflixDAO
Just so you know, I am using a sample database from MongoDB Atlas. I am using Postman to test HTTP requests. All the data follows JSON format
Anyway, when I execute a basic GET request. The program runs without any problems. All the data outputs as expected. However, if I execute something along the lines of
GET http://localhost:5000/api/v1/mflix?year=1903
Then moviesList returns an empty array [], but no error message.
After debugging, I suspect the problem lies either at cursor = await mflix.find(query) or displayCursor = cursor.limit(moviesPerPage).skip(moviesPerPage * page), but the callstacks for those methods is so complex for me, I don't know what to even look for.
Any suggestions?
Edit: Here is an example of the document I am trying to access:
{
"_id": "573a1390f29313caabcd42e8",
"plot": "A group of bandits stage a brazen train hold-up, only to find a determined posse hot on their heels.",
"genres": [
"Short",
"Western"
],
"runtime": 11,
"cast": [
"A.C. Abadie",
"Gilbert M. 'Broncho Billy' Anderson",
"George Barnes",
"Justus D. Barnes"
],
"poster": "https://m.media-amazon.com/images/M/MV5BMTU3NjE5NzYtYTYyNS00MDVmLWIwYjgtMmYwYWIxZDYyNzU2XkEyXkFqcGdeQXVyNzQzNzQxNzI#._V1_SY1000_SX677_AL_.jpg",
"title": "The Great Train Robbery",
"fullplot": "Among the earliest existing films in American cinema - notable as the first film that presented a narrative story to tell - it depicts a group of cowboy outlaws who hold up a train and rob the passengers. They are then pursued by a Sheriff's posse. Several scenes have color included - all hand tinted.",
"languages": [
"English"
],
"released": "1903-12-01T00:00:00.000Z",
"directors": [
"Edwin S. Porter"
],
"rated": "TV-G",
"awards": {
"wins": 1,
"nominations": 0,
"text": "1 win."
},
"lastupdated": "2015-08-13 00:27:59.177000000",
"year": 1903,
"imdb": {
"rating": 7.4,
"votes": 9847,
"id": 439
},
"countries": [
"USA"
],
"type": "movie",
"tomatoes": {
"viewer": {
"rating": 3.7,
"numReviews": 2559,
"meter": 75
},
"fresh": 6,
"critic": {
"rating": 7.6,
"numReviews": 6,
"meter": 100
},
"rotten": 0,
"lastUpdated": "2015-08-08T19:16:10.000Z"
},
"num_mflix_comments": 0
}
EDIT: It seems to be a datatype problem. When I request a data with a string/varchar type, the program returns values that contain that value. Example:
Input:
GET localhost:5000/api/v1/mflix?rated=TV-G
Output:
{
"_id": "XXXXXXXXXX"
// Data
"rated" = "TV-G"
// Data
}
EDIT: The problem has nothing to do with anything I've posted up to this point it seems. The problem is in this piece of code:
let filters = {}
if(req.query.year){
filters.year = req.query.year // This line needs to be changed
}
const {moviesList, totalNumMovies} = await MflixDAO.getMovies({
filters,
page,
moviesPerPage,
})
I will explain in the answer below

Ok so the problem, as it turns out, is that when I make an HTTP request, the requested value is passed as a string. So in
GET http://localhost:5000/api/v1/mflix?year=1903
the value of year is registered by the program as a string. In other words, the DAO ends up looking for "1903" instead of 1903. Naturally, year = "1903" does not exist. To fix this, the line filters.year = req.query.year must be changed to filters.year = parseInt(req.query.year).

Dialogflow Fulfilment webhook call failed

I am new to dialogflow fulfillment and I am trying to retrieve news from news API based on user questions. I followed documentation provided by news API, but I am not able to catch any responses from the search results, when I run the function in console it is not errors. I changed the code and it looks like now it is reaching to the newsapi endpoint but it is not fetching any results. I am utilizing https://newsapi.org/docs/client-libraries/node-js to make a request to search everything about the topic. when I diagnoise the function it says " Webhook call failed. Error: UNAVAILABLE. "
'use strict';
const functions = require('firebase-functions');
const {WebhookClient} = require('dialogflow-fulfillment');
const {Card, Suggestion} = require('dialogflow-fulfillment');
const http = require('http');
const host = 'newsapi.org';
const NewsAPI = require('newsapi');
const newsapi = new NewsAPI('63756dc5caca424fb3d0343406295021');
process.env.DEBUG = 'dialogflow:debug';
exports.dialogflowFirebaseFulfillment = functions.https.onRequest((req, res) =>
{
// Get the city
let search = req.body.queryResult.parameters['search'];// search is a required param
// Call the weather API
callNewsApi(search).then((response) => {
res.json({ 'fulfillmentText': response }); // Return the results of the news API to Dialogflow
}).catch((xx) => {
console.error(xx);
res.json({ 'fulfillmentText': `I don't know the news but I hope it's good!` });
});
});
function callNewsApi(search)
{
console.log(search);
newsapi.v2.everything
(
{
q: 'search',
langauge: 'en',
sortBy: 'relevancy',
source: 'cbc-news',
domains: 'cbc.ca',
from: '2019-12-31',
to: '2020-12-12',
page: 2
}
).then (response => {console.log(response);
{
let articles = response['data']['articles'][0];
// Create response
let responce = `Current news in the $search with following title is ${articles['titile']} which says that
${articles['description']}`;
// Resolve the promise with the output text
console.log(output);
}
});
}
Also here is RAW API response
{
"responseId": "a871b8d2-16f2-4873-a5d1-b907a07adb9a-b4ef8d5f",
"queryResult": {
"queryText": "what is the latest news about toronto",
"parameters": {
"search": [
"toronto"
]
},
"allRequiredParamsPresent": true,
"fulfillmentMessages": [
{
"text": {
"text": [
""
]
}
}
],
"intent": {
"name": "projects/misty-ktsarh/agent/intents/b52c5774-e5b7-494a-8f4c-f783ebae558b",
"displayName": "misty.news"
},
"intentDetectionConfidence": 1,
"diagnosticInfo": {
"webhook_latency_ms": 543
},
"languageCode": "en"
},
"webhookStatus": {
"code": 14,
"message": "Webhook call failed. Error: UNAVAILABLE."
},
"outputAudio": "UklGRlQqAABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YTAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA... (The content is truncated. Click `COPY` for the original JSON.)",
"outputAudioConfig": {
"audioEncoding": "OUTPUT_AUDIO_ENCODING_LINEAR_16",
"synthesizeSpeechConfig": {
"speakingRate": 1,
"voice": {}
}
}
}
And Here is fulfillment request:
{
"responseId": "a871b8d2-16f2-4873-a5d1-b907a07adb9a-b4ef8d5f",
"queryResult": {
"queryText": "what is the latest news about toronto",
"parameters": {
"search": [
"toronto"
]
},
"allRequiredParamsPresent": true,
"fulfillmentMessages": [
{
"text": {
"text": [
""
]
}
}
],
"intent": {
"name": "projects/misty-ktsarh/agent/intents/b52c5774-e5b7-494a-8f4c-f783ebae558b",
"displayName": "misty.news"
},
"intentDetectionConfidence": 1,
"diagnosticInfo": {
"webhook_latency_ms": 543
},
"languageCode": "en"
},
"webhookStatus": {
"code": 14,
"message": "Webhook call failed. Error: UNAVAILABLE."
},
"outputAudio": "UklGRlQqAABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YTAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA... (The content is truncated. Click `COPY` for the original JSON.)",
"outputAudioConfig": {
"audioEncoding": "OUTPUT_AUDIO_ENCODING_LINEAR_16",
"synthesizeSpeechConfig": {
"speakingRate": 1,
"voice": {}
}
}
}
Also here is the screenshot from the firebase console.
Can anyone guide me what is that I am missing in here?

The key is the first three lines in the error message:
Function failed on loading user code. Error message: Code in file index.js can't be loaded.
Did you list all required modules in the package.json dependencies?
Detailed stack trace: Error: Cannot find module 'newsapi'
It is saying that the newsapi module couldn't be loaded and that the most likely cause of this is that you didn't list this as a dependency in your package.json file.
If you are using the Dialogflow Inline Editor, you need to select the package.json tab and add a line in the dependencies section.
Update
It isn't clear exactly when/where you're getting the "UNAVAILABLE" error, but one likely cause if you're using Dialogflow's Inline Editor is that it is using the Firebase "Spark" pricing plan, which has limitations on network calls outside Google's network.
You can upgrade to the Blaze plan, which does require a credit card on file, but does include the Spark plan's free tier, so you shouldn't incur any costs during light usage. This will allow for network calls.
Update based on TypeError: Cannot read property '0' of undefined
This indicates that either a property (or possibly an index of a property) is trying to reference against something that is undefined.
It isn't clear which line, exactly, this may be, but these lines all are suspicious:
let response = JSON.parse(body);
let source = response['data']['source'][0];
let id = response['data']['id'][0];
let name = response['data']['name'][0];
let author = response['author'][0];
let title = response['title'][0];
let description = response['description'][0];
since they are all referencing a property. I would check to see exactly what comes back and gets stored in response. For example, could it be that there is no "data" or "author" field in what is sent back?
Looking at https://newsapi.org/docs/endpoints/everything, it looks like none of these are fields, but that there is an articles property sent back which contains an array of articles. You may wish to index off that and get the attributes you want.
Update
It looks like that, although you are loading the parameter into a variable with this line
// Get the city and date from the request
let search = req.body.queryResult.parameters['search'];// city is a required param
You don't actually use the search variable anywhere. Instead, you seem to be passing a literal string "search" to your function with this line
callNewsApi('search').then((output) => {
which does a search for the word "search", I guess.
You indicated that "it goes to the catch portion", which indicates that something went wrong in the call. You don't show any logging in the catch portion, and it may be useful to log the exception that is thrown, so you know why it is going to the catch portion. Something like
}).catch((xx) => {
console.error(xx);
res.json({ 'fulfillmentText': `I don't know the news but I hope it's good!` });
});
is normal, but since it looks like you're logging it in the .on('error') portion, showing that error might be useful.

The name of the intent and the variable I was using to make the call had a difference in Casing, I guess calls are case sensitive just be aware of that

How do I get extended REtweets in Node.js with Twitter API? (Only getting a MAX of 140 characters)

I am trying to get extended retweets, but I cant seem to get that extra field called retweeted_status as a parameter in the result. I don't get any errors when including a parameter called "retweeted_status" and I don't know if it even exists as a parameter in Node.js since Twitter API docs doesn't include any type of examples or clear information for retweets and I tried to implement methods written in other programming lenguages that I searched in Google but didn't get any good results at the end. (Note: tweet_mode only works for Tweets not RETweets!)
Things I tried:
retweeted_status: 'extended' //doesn't work but node.js doesn't give an error
retweeted_status: true //doesn't work but node.js doesn't give an error
retweeted_status: 1 //doesn't work but node.js doesn't give an error
My code is the following (It never enters the if-condition and if it does it gives the error I mention in the code):
var params = {q: '#Avengers', count: 100, include_rts: 0, tweet_mode: 'extended', retweeted_status: 'extended'};
client.get('search/tweets', params, function(error, tweets, response) {
if (!error) {
console.log(tweets);
for(var i=0; i < params.count; i++){
if (tweets.retweeted_status){ //It never enters this condition
//even if it does I get this error
//TypeError: Cannot read property 'statuses' of undefined tweets.statuses[i]
tweets = tweets.retweeted_status;
var tweet = (JSON.stringify(tweets.statuses[i].full_text)+"\n");
var string = tweet.replace(/[&\/\\#,+()$~%.'":*?<>{}]/g, '');
console.log('hi');
}
else{
var tweet = (JSON.stringify(tweets.statuses[i].full_text)+"\n");
var string = tweet.replace(/[&\/\\#,+()$~%.'":*?<>{}]/g, '');
console.log('hi2');
}
fs.appendFile("tweet.txt", string, function (err) {
if (err) throw err;
console.log('Saved!');
});
}
}
});
I looked for a few hours into this website https://dev.to/kehers/formatting-tweets-a-look-at-extended-tweets-retweets-and-quotes-n5j and tried to make some code (that I already provided above) which didn't work since the IF-statement got ignored. When getting the tweet it should give an extra parameter "retweeted_status" like below:
{
"created_at": "Sun Mar 11 12:00:27 +0000 2018",
"id": 972804442667003900,
"full_text": "RT #jasongorman: As a physics grad, I understand how snooker works at a level I imagine a lot of pro snooker players don't. But I suck at s…",
"truncated": false,
"display_text_range": [...],
"entities": {...},
"retweeted_status": { // <======================= I am not getting this extra parameter!
"created_at": "Sun Mar 11 08:10:46 +0000 2018",
"id": 972746641957642200,
"full_text": "As a physics grad, I understand how snooker works at a level I imagine a lot of pro snooker players don't. But I suck at snooker. Understanding != ability.",
"truncated": false,
"display_text_range": [0, 155],
"entities": {
"hashtags": [],
"symbols": [],
"user_mentions": [],
"urls": []
},
}
}

Failing to add a key and value to a javascript object

I'm new to JavaScript and Node etc but enjoying the experience of developing - last serious development i did was in the 90s - I used to be a 370 assembler programmer back 30 years ago!
I've been stuck on this all day.
My issue is that I am trying to add a new element to the entries in an array of objects. I have simplified my code here in the hope that I am doing something blindingly stupid and obviously wrong though I have tried this in javascript in the browser and I know I am barking up the right tree.
What I am trying to do is add a new key/value pair to the collectibles returned. I have simplified the code here but in essence the value is coming for a different collection. the issue is that the lines that set the new keys are having no effect on my objects in the array collectibles. I have tried several methods all of which I believe should work.
I can read the key/values in the returned array of objects and I can also set existing keys to new values, but setting a new key is not working.
Here is my code:
var sendJsonResponse = function (res, status, content) {
res.status(status);
res.json(content);
};
var mongoose = require("mongoose");
var Coll = mongoose.model("collectible");
var Img = mongoose.model("image");
module.exports.collectiblesList = function (req, res) {
var firstImage_ids = [];
var imageDir = "https://s3.amazonaws.com/xxxxxxxxxxx/images/";
var defaultImage = imageDir + "default.jpg";
// get the collectibles
Coll.find(req.body).exec(function (err, collectibles) {
if (!collectibles) {
// not found
sendJsonResponse(res, 404, {"message": "no collectibles found"});
return;
}
if (err) {
console.log(collectible);
sendJsonResponse(res, 400, err);
return;
}
//found the collectibles
// now find the list of images we need to pull (each collectible can have multiple images
collectibles.forEach(function (collectible, index) {
if (collectible.image_ids) {
firstImage_ids.push(collectible.image_ids[0]);
collectibles[index].imageUrl = defaultImage; // this is the line that has no effect.
collectible.imageUrl = defaultImage; // tried this too.
collectible["imageUrl"] = defaultImage; // and this. :-(
console.log(collectible);
}
});
sendJsonResponse(res, 200, collectibles);
});
};
Using Postman to test this API. This is what is being returned:
[
{
"_id": "5a43c61134aaea2025158cab",
"name": "A-0_5",
"title": "Occoneechee Lodge 104: A-0_5",
"issueDate": null,
"quantity": null,
"event_id": null,
"type": "A - Arrowhead patches",
"tag_ids": [
"5a401a17b720186ab61c649e"
],
"description": "Elangomat, no name",
"organisation_id": "5a413340b720186ab61c661e",
"supercedes_id": null,
"sortName": "A-000.5",
"image_ids": [
"5a41685ab720186ab61c663d"
],
"deleted": false,
"schemaVersion": "1.0"
},
{
"_id": "5a43c63334aaea2025158cac",
"name": "A-1",
"title": "Occoneechee Lodge 104: A-1",
"issueDate": null,
"quantity": null,
"event_id": null,
"type": "A - Arrowhead patches",
"tag_ids": [
"5a401a17b720186ab61c649e"
],
"description": "Service",
"organisation_id": "5a413340b720186ab61c661e",
"supercedes_id": null,
"sortName": "A-001",
"image_ids": [
"5a41685ab720186ab61c663e"
],
"deleted": false,
"schemaVersion": "1.0"
}]
i.e. it contains no imageUrl keys.
thanks for the assist.
WWW
GT

I might have misunderstood. But if your issue is that collectibles[i].imageUrl is later on returning undefined/null, then i believe it's simply because you need to add
collectibles.save();
after setting the collectibles[i].imageUrl value. you can add that line after the foreach loop, before the json response.
collectibles.forEach(function (collectible, index) {
if (collectible.image_ids) {
firstImage_ids.push(collectible.image_ids[0]);
collectible.imageUrl = defaultImage;
}
});
collectibles.save(); //here, this saves the changes done to the mongoose object.
sendJsonResponse(res, 200, collectibles);

I solved it in the end after a lot of effort. This article explains (but i don't fully yet follow the explanation)
I found that the object has more metadata in it and the actual object is in _doc, so the first (and i suspect frowned upon solution) is to update the code to:
collectible._doc.imageUrl = defaultImage;
The better way was to simply add imageUrl to the mongoose model and leave _doc out of it. The issue there is that this is not a field i intend to populate in the mongoDB.

AngularJS / Restangular routing "Cannot set property 'route' of undefined"

I have a AngularJS-based frontend using restangular to fetch records from a Django backend I've built.
I'm making a call for a client list with the following:
var app;
app = angular.module("myApp", ["restangular"]).config(function(RestangularProvider) {
RestangularProvider.setBaseUrl("http://172.16.91.149:8000/client/v1");
RestangularProvider.setResponseExtractor(function(response, operation) {
return response.objects;
});
return RestangularProvider.setRequestSuffix("/?callback=abc123");
});
angular.module("myApp").controller("MainCtrl", function($scope, Restangular) {
return $scope.client = Restangular.all("client").getList();
});
Chrome is showing the backend returning data with an HTTP 200:
abc123({
"meta": {
"limit": 20,
"next": "/client/v1/client/?callback=abc123&limit=20&offset=20",
"offset": 0,
"previous": null,
"total_count": 2
},
"objects": [{
"id": 1,
"name": "Test",
"resource_uri": "/client/v1/client/1/"
}, {
"id": 2,
"name": "Test 2",
"resource_uri": "/client/v1/client/2/"
}]
})
But once that happens I'm seeing the following stack trace appear in Chrome's console:
TypeError: Cannot set property 'route' of undefined
at restangularizeBase (http://172.16.91.149:9000/components/restangular/src/restangular.js:395:56)
at restangularizeCollection (http://172.16.91.149:9000/components/restangular/src/restangular.js:499:35)
at http://172.16.91.149:9000/components/restangular/src/restangular.js:556:44
at wrappedCallback (http://ajax.googleapis.com/ajax/libs/angularjs/1.0.7/angular.js:6846:59)
at http://ajax.googleapis.com/ajax/libs/angularjs/1.0.7/angular.js:6883:26
at Object.Scope.$eval (http://ajax.googleapis.com/ajax/libs/angularjs/1.0.7/angular.js:8057:28)
at Object.Scope.$digest (http://ajax.googleapis.com/ajax/libs/angularjs/1.0.7/angular.js:7922:25)
at Object.Scope.$apply (http://ajax.googleapis.com/ajax/libs/angularjs/1.0.7/angular.js:8143:24)
at done (http://ajax.googleapis.com/ajax/libs/angularjs/1.0.7/angular.js:9170:20)
at completeRequest (http://ajax.googleapis.com/ajax/libs/angularjs/1.0.7/angular.js:9333:7) angular.js:5754
I did a breakpoint on line 395 in in restangular.js:
L394 function restangularizeBase(parent, elem, route) {
L395 elem[config.restangularFields.route] = route;
The first time it hits the breakpoint elem is just an object and route has the value of client.
The second time the breakpoint is hit elem is undefined and route has the value of client.
Any ideas why elem would be undefined the second time around?

When requesting lists, Restangular expects the data from the server to be a simple array. However, if the resulting data is wrapped with result metadata, such as pagination info, it falls apart.
If you are using Django REST Framework, it will return results wrapped like this:
{
"count": 2,
"next": null,
"previous": null,
"results": [
{
"id": 1,
"name": "Foo"
},
{
"id": 2,
"name": "Bar"
}
]
}
To translate this, you need to create a response extractor function. It's easiest to specify in the module config:
angular.module('myApp', ['myApp.controllers', 'restangular']).
config(function(RestangularProvider) {
RestangularProvider.setBaseUrl("/api");
// This function is used to map the JSON data to something Restangular
// expects
RestangularProvider.setResponseExtractor(function(response, operation, what, url) {
if (operation === "getList") {
// Use results as the return type, and save the result metadata
// in _resultmeta
var newResponse = response.results;
newResponse._resultmeta = {
"count": response.count,
"next": response.next,
"previous": response.previous
};
return newResponse;
}
return response;
});
});
This rearranges the results to be a simple array, with an additional property of _resultmeta, containing the metadata. Restangular will do it's thing with the array, and it's objects, and you can access the _resultmeta property when handling the array as you would expect.

I'm the creator of Restangular.
The restangularizeBase function is called first for your collection and then for each of your elements.
From the StackTrace, the element is OK, but once the collection is sent to restangularizeBase, it's actually undefined. Could you please console.log response.objects? Also, please update to the latest version.
Also, for the default request parameter, you should be using defaultRequestParams instead of the requestSuffix. requestSuffix should only be used for the ending "/"
Let me know if I can help you some more!