Develop Reference

MongoDB: How do I find coordinates nearby?

In this function, I want to check if there are any posts in the database made nearby to a set of input lat/long coords so that I can display them based on where the user currently is, and I also want to write a function to check if they are in a "too close" range (less than 2m or so).
I'm not sure how to do it properly and every answer I have found online after extensive research has given me nothing. I am using NodeJS 18 LTS with the latest version of mongodb.
This is the code I tried to get the posts
const getPostsInRange = (lng, lat) => {
// get all posts in range
const postCollection = db.collection('posts')
return postCollection.find({
"location": {
"$near": [parseFloat(lng), parseFloat(lat)],
"$maxDistance": 100000
}})
}
and here is the code that calls it, from index.js
app.get('/getposts/:lat/:long', requiresAuth(), (req, res) => {
const { lat, long } = req.params
getPostsInRange(long, lat)
.then((posts) => {
res.send(posts)
})
.catch((err) => {
console.log(err)
res.status(500).end()
})
what my data looks like in mongodb:
{
"textBody": "new message 3",
"image": "xxxx",
"location": {
"type": "Point",
"coordinates": [xxxx, xxxx]
},
"timestamp": 1674503437
}
I get errors such as
TypeError: getPostsInRange(...).then is not a function, and similar but slightly different ones that Google seems to have never have heard of.
i would love some help building this project, full code snippets would be greatly appreciate due to my lack of experience working with mongodb.

GET request to webpage returns string, but cannot use it

I am creating a google chrome extension, and when I make a get request to this web page https://www.rightmove.co.uk/property-for-sale/find.html?locationIdentifier=REGION%5E27675&maxBedrooms=2&minBedrooms=2&sortType=6&propertyTypes=&mustHave=&dontShow=&furnishTypes=&keywords=, I get the webpage HTML as a response which is what I want (the website I am requesting info from does not have an API and I cannot web scrape for reasons too long to explain here). This response comes in the form of a string. When I attempt to split this string at a certain point, bis_skin_checked, I am returned an array of length 1, meaning that there was no match and nothing has been split. But when I look at the string returned it has it included.
I have tried things like removing spaces and carriage returns but nothing seems to be working. This is my GET request code:
function getNewPage(url) {
let returnedValue = fetch(url, {
method: 'GET',
headers: {
'Content-Type': 'text/html',
},
})
.then(response => response.text())
.then(text => {
return text
})
return returnedValue
}
I then go on to resolve the promise which is returnedValue:
let newURL = getHousePrices(currentUrl) // Get Promise of new page as a string
newURL.then(function(value) { // Resolve promise and do stuff
console.log(value.split('bis_skin_checked').length)
})
And then work with the string which looks like this: (I have attached an image as I cannot copy the text from the popup)
Image Link To API Request return

I'm assuming you want to get the home values, given certain search parameters.
Scraping raw text is usually not the way to go. I took a look at the site and saw it uses uniform network requests that you can modify to capture the data you need directly instead of scraping the raw HTML.
I built a solution that allows you to dynamically pass whatever parameters you want to the getHomes() function. Passing nothing will use the default params that you can use as a baseline while you try to adjust the request for any additional modifications/use cases.
Install the below solution and run getHomes() from the service worker.
Here's a brief video I made explaining the solution: https://vimeo.com/772275354/07fd1025ed
--- manifest.JSON ---
{
"name": "UK Housing - Stackoverflow",
"description": "Example for how to make network requests and mimic them in background.js to avoid web scraping raw text",
"version": "1.0.0",
"manifest_version": 3,
"background": {
"service_worker": "background.js"
},
"host_permissions": [
"*://*.rightmove.co.uk/*"
]
}
--- background.js ---
async function getHomes(passedParams) {
const newParams = passedParams ? passedParams : {}; // set to an empty object if no new params passed - avoid error in object.entries loop.
// starts with default params for a working request, you can then pass params to override the defaults to test new requests.
var params = {
"locationIdentifier": "REGION%5E27675",
"maxBedrooms": "2",
"minBedrooms": "1",
"numberOfPropertiesPerPage": "25",
"radius": "0.0",
"sortType": "6",
"index": "0",
"viewType": "LIST",
"channel": "BUY",
"areaSizeUnit": "sqft",
"currencyCode": "GBP",
"isFetching": "false"
}
Object.entries(params).forEach(([key, value]) => {
// we iterate through each key in our default params to check if we passed a new value for that key.
// we then set the params object to the new value if we did.
if (newParams[key]) params[key] = newParams[key];
});
const rightMoveAPISearch = `https://www.rightmove.co.uk/api/_search?
locationIdentifier=${params['locationIdentifier']}
&maxBedrooms=${params['maxBedrooms']}
&minBedrooms=${params['minBedrooms']}
&numberOfPropertiesPerPage=${params['numberOfPropertiesPerPage']}
&radius=${params['radius']}
&sortType=${params['sortType']}
&index=${params['index']}
&viewType=${params['viewType']}
&channel=${params['channel']}
&areaSizeUnit=${params['areaSizeUnit']}
&currencyCode=${params['currencyCode']}
&isFetching=${params['isFetching']}
`.replace(/\s/g, ''); // removes the whitespaces we added to make it look nicer / easier to adjust.
const data = await
fetch(rightMoveAPISearch, {
"method": "GET",
})
.then(data => data.json())
.then(res => { return res })
if (data.resultCount) {
console.log('\x1b[32m%s\x1b[0m', 'Request successful! Result count: ', parseInt(data.resultCount));
console.log('All data: ', data);
console.log('Properties: ', data.properties);
}
else console.log('\x1b[31m%s\x1b[0m', `Issue with the request:`, data)
return data
}
Hope this is helpful. Let me know if you need anything else.

Dialogflow Fulfilment webhook call failed

I am new to dialogflow fulfillment and I am trying to retrieve news from news API based on user questions. I followed documentation provided by news API, but I am not able to catch any responses from the search results, when I run the function in console it is not errors. I changed the code and it looks like now it is reaching to the newsapi endpoint but it is not fetching any results. I am utilizing https://newsapi.org/docs/client-libraries/node-js to make a request to search everything about the topic. when I diagnoise the function it says " Webhook call failed. Error: UNAVAILABLE. "
'use strict';
const functions = require('firebase-functions');
const {WebhookClient} = require('dialogflow-fulfillment');
const {Card, Suggestion} = require('dialogflow-fulfillment');
const http = require('http');
const host = 'newsapi.org';
const NewsAPI = require('newsapi');
const newsapi = new NewsAPI('63756dc5caca424fb3d0343406295021');
process.env.DEBUG = 'dialogflow:debug';
exports.dialogflowFirebaseFulfillment = functions.https.onRequest((req, res) =>
{
// Get the city
let search = req.body.queryResult.parameters['search'];// search is a required param
// Call the weather API
callNewsApi(search).then((response) => {
res.json({ 'fulfillmentText': response }); // Return the results of the news API to Dialogflow
}).catch((xx) => {
console.error(xx);
res.json({ 'fulfillmentText': `I don't know the news but I hope it's good!` });
});
});
function callNewsApi(search)
{
console.log(search);
newsapi.v2.everything
(
{
q: 'search',
langauge: 'en',
sortBy: 'relevancy',
source: 'cbc-news',
domains: 'cbc.ca',
from: '2019-12-31',
to: '2020-12-12',
page: 2
}
).then (response => {console.log(response);
{
let articles = response['data']['articles'][0];
// Create response
let responce = `Current news in the $search with following title is ${articles['titile']} which says that
${articles['description']}`;
// Resolve the promise with the output text
console.log(output);
}
});
}
Also here is RAW API response
{
"responseId": "a871b8d2-16f2-4873-a5d1-b907a07adb9a-b4ef8d5f",
"queryResult": {
"queryText": "what is the latest news about toronto",
"parameters": {
"search": [
"toronto"
]
},
"allRequiredParamsPresent": true,
"fulfillmentMessages": [
{
"text": {
"text": [
""
]
}
}
],
"intent": {
"name": "projects/misty-ktsarh/agent/intents/b52c5774-e5b7-494a-8f4c-f783ebae558b",
"displayName": "misty.news"
},
"intentDetectionConfidence": 1,
"diagnosticInfo": {
"webhook_latency_ms": 543
},
"languageCode": "en"
},
"webhookStatus": {
"code": 14,
"message": "Webhook call failed. Error: UNAVAILABLE."
},
"outputAudio": "UklGRlQqAABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YTAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA... (The content is truncated. Click `COPY` for the original JSON.)",
"outputAudioConfig": {
"audioEncoding": "OUTPUT_AUDIO_ENCODING_LINEAR_16",
"synthesizeSpeechConfig": {
"speakingRate": 1,
"voice": {}
}
}
}
And Here is fulfillment request:
{
"responseId": "a871b8d2-16f2-4873-a5d1-b907a07adb9a-b4ef8d5f",
"queryResult": {
"queryText": "what is the latest news about toronto",
"parameters": {
"search": [
"toronto"
]
},
"allRequiredParamsPresent": true,
"fulfillmentMessages": [
{
"text": {
"text": [
""
]
}
}
],
"intent": {
"name": "projects/misty-ktsarh/agent/intents/b52c5774-e5b7-494a-8f4c-f783ebae558b",
"displayName": "misty.news"
},
"intentDetectionConfidence": 1,
"diagnosticInfo": {
"webhook_latency_ms": 543
},
"languageCode": "en"
},
"webhookStatus": {
"code": 14,
"message": "Webhook call failed. Error: UNAVAILABLE."
},
"outputAudio": "UklGRlQqAABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YTAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA... (The content is truncated. Click `COPY` for the original JSON.)",
"outputAudioConfig": {
"audioEncoding": "OUTPUT_AUDIO_ENCODING_LINEAR_16",
"synthesizeSpeechConfig": {
"speakingRate": 1,
"voice": {}
}
}
}
Also here is the screenshot from the firebase console.
Can anyone guide me what is that I am missing in here?

The key is the first three lines in the error message:
Function failed on loading user code. Error message: Code in file index.js can't be loaded.
Did you list all required modules in the package.json dependencies?
Detailed stack trace: Error: Cannot find module 'newsapi'
It is saying that the newsapi module couldn't be loaded and that the most likely cause of this is that you didn't list this as a dependency in your package.json file.
If you are using the Dialogflow Inline Editor, you need to select the package.json tab and add a line in the dependencies section.
Update
It isn't clear exactly when/where you're getting the "UNAVAILABLE" error, but one likely cause if you're using Dialogflow's Inline Editor is that it is using the Firebase "Spark" pricing plan, which has limitations on network calls outside Google's network.
You can upgrade to the Blaze plan, which does require a credit card on file, but does include the Spark plan's free tier, so you shouldn't incur any costs during light usage. This will allow for network calls.
Update based on TypeError: Cannot read property '0' of undefined
This indicates that either a property (or possibly an index of a property) is trying to reference against something that is undefined.
It isn't clear which line, exactly, this may be, but these lines all are suspicious:
let response = JSON.parse(body);
let source = response['data']['source'][0];
let id = response['data']['id'][0];
let name = response['data']['name'][0];
let author = response['author'][0];
let title = response['title'][0];
let description = response['description'][0];
since they are all referencing a property. I would check to see exactly what comes back and gets stored in response. For example, could it be that there is no "data" or "author" field in what is sent back?
Looking at https://newsapi.org/docs/endpoints/everything, it looks like none of these are fields, but that there is an articles property sent back which contains an array of articles. You may wish to index off that and get the attributes you want.
Update
It looks like that, although you are loading the parameter into a variable with this line
// Get the city and date from the request
let search = req.body.queryResult.parameters['search'];// city is a required param
You don't actually use the search variable anywhere. Instead, you seem to be passing a literal string "search" to your function with this line
callNewsApi('search').then((output) => {
which does a search for the word "search", I guess.
You indicated that "it goes to the catch portion", which indicates that something went wrong in the call. You don't show any logging in the catch portion, and it may be useful to log the exception that is thrown, so you know why it is going to the catch portion. Something like
}).catch((xx) => {
console.error(xx);
res.json({ 'fulfillmentText': `I don't know the news but I hope it's good!` });
});
is normal, but since it looks like you're logging it in the .on('error') portion, showing that error might be useful.

The name of the intent and the variable I was using to make the call had a difference in Casing, I guess calls are case sensitive just be aware of that

Only download Firebase snapshots that have child key

My data structure looks like this (removed unnecessary parts):
{
"threads" : {
"PUSHID" : {
"info" : {
"members" : {
"uid" : true,
"uid2" : true
}
}
}
}
}
I'm trying to write some javascript to pull snapshots of threads a user is in, but I can't figure out a way for it to work without pulling snapshots of each thread. This is my code now that pulls each thread snapshot.
firebase.database().ref('threads').on('child_added', function(snapshot) {
if (snapshot.hasChild('info/members/' + userUid)) {
// Display thread info
}
});
I tried to make a query with .orderByChild('info/members/' + userUid) and removing null snapshots, but I would have to add a .indexOn for each userUid which is obviously not practical.

Your current structure makes it easy/efficient to look up the users for a thread. But your use case is to look up the threads for a user. You'll need to augment your data model to allow the use-case:
{
"user_threads" : {
"uid": {
"PUSHID": true,
"PUSHID2": true
},
"uid2": {
"PUSHID": true,
"PUSHID3": true
}
}
}
And then read it with:
firebase.database().ref('user_threads/'+userUid).on('child_added', function(snapshot) {
...
});
Modifying/expanding your data model to match the use-cases of your app is quite common when using NoSQL databases.

Ember ActiveModelAdapter URL without underscore?

i have this json response to GET-request => /checklists
{"check_lists": [
{
"id": 1,
"name": "Example-list-1",
"description": ""
},
{
"id": 2,
"name": "Example-list-1",
"description": ""
}
}
To handle the underscored naming-convention of this server, i use the ActiveModelAdapter and ActiveModelSerializer.
The problem i got stucked on is the request-url.
My model is named App.CheckList = DS.Model.extend({... and thats the point where it starts to get complicated.
if i call
return this.store.find('checkList');
in my route, Ember start a GET request to /checkLists route instead of /check_lists =>
GET http://localhost:3000/checkLists 404 (Not Found)
Error while processing route: checklists
surprisingly for this error the
buildURL: function(type, id) {
return this._super(type, id);
},
is not used so i have no chance to modify the url.
Does anyone know how to change the request to /check_lists ??

The mapping of camelized case and underscored case is done in ActiveModelAdapter.pathForType function. You can override it and make your changes there. For instance, to change from camelized case to underscored case:
App.ApplicationAdapter = DS.ActiveModelAdapter.extend({
pathForType: function(type) {
var decamelized = Ember.String.decamelize(type);
var underscored = Ember.String.underscore(decamelized);
//Alternatively, you can change urls to dasherized case using this line
//var dasherized = Ember.String.dasherize(decamelized);
return Ember.String.pluralize(underscored);
}
});
The strange part is that the latest Ember Data already has this code in ActiveModelAdapter. You may want to check the version you are running and upgrade to that instead of the change I suggest above.