Develop Reference

GET request to webpage returns string, but cannot use it

I am creating a google chrome extension, and when I make a get request to this web page https://www.rightmove.co.uk/property-for-sale/find.html?locationIdentifier=REGION%5E27675&maxBedrooms=2&minBedrooms=2&sortType=6&propertyTypes=&mustHave=&dontShow=&furnishTypes=&keywords=, I get the webpage HTML as a response which is what I want (the website I am requesting info from does not have an API and I cannot web scrape for reasons too long to explain here). This response comes in the form of a string. When I attempt to split this string at a certain point, bis_skin_checked, I am returned an array of length 1, meaning that there was no match and nothing has been split. But when I look at the string returned it has it included.
I have tried things like removing spaces and carriage returns but nothing seems to be working. This is my GET request code:
function getNewPage(url) {
let returnedValue = fetch(url, {
method: 'GET',
headers: {
'Content-Type': 'text/html',
},
})
.then(response => response.text())
.then(text => {
return text
})
return returnedValue
}
I then go on to resolve the promise which is returnedValue:
let newURL = getHousePrices(currentUrl) // Get Promise of new page as a string
newURL.then(function(value) { // Resolve promise and do stuff
console.log(value.split('bis_skin_checked').length)
})
And then work with the string which looks like this: (I have attached an image as I cannot copy the text from the popup)
Image Link To API Request return

I'm assuming you want to get the home values, given certain search parameters.
Scraping raw text is usually not the way to go. I took a look at the site and saw it uses uniform network requests that you can modify to capture the data you need directly instead of scraping the raw HTML.
I built a solution that allows you to dynamically pass whatever parameters you want to the getHomes() function. Passing nothing will use the default params that you can use as a baseline while you try to adjust the request for any additional modifications/use cases.
Install the below solution and run getHomes() from the service worker.
Here's a brief video I made explaining the solution: https://vimeo.com/772275354/07fd1025ed
--- manifest.JSON ---
{
"name": "UK Housing - Stackoverflow",
"description": "Example for how to make network requests and mimic them in background.js to avoid web scraping raw text",
"version": "1.0.0",
"manifest_version": 3,
"background": {
"service_worker": "background.js"
},
"host_permissions": [
"*://*.rightmove.co.uk/*"
]
}
--- background.js ---
async function getHomes(passedParams) {
const newParams = passedParams ? passedParams : {}; // set to an empty object if no new params passed - avoid error in object.entries loop.
// starts with default params for a working request, you can then pass params to override the defaults to test new requests.
var params = {
"locationIdentifier": "REGION%5E27675",
"maxBedrooms": "2",
"minBedrooms": "1",
"numberOfPropertiesPerPage": "25",
"radius": "0.0",
"sortType": "6",
"index": "0",
"viewType": "LIST",
"channel": "BUY",
"areaSizeUnit": "sqft",
"currencyCode": "GBP",
"isFetching": "false"
}
Object.entries(params).forEach(([key, value]) => {
// we iterate through each key in our default params to check if we passed a new value for that key.
// we then set the params object to the new value if we did.
if (newParams[key]) params[key] = newParams[key];
});
const rightMoveAPISearch = `https://www.rightmove.co.uk/api/_search?
locationIdentifier=${params['locationIdentifier']}
&maxBedrooms=${params['maxBedrooms']}
&minBedrooms=${params['minBedrooms']}
&numberOfPropertiesPerPage=${params['numberOfPropertiesPerPage']}
&radius=${params['radius']}
&sortType=${params['sortType']}
&index=${params['index']}
&viewType=${params['viewType']}
&channel=${params['channel']}
&areaSizeUnit=${params['areaSizeUnit']}
&currencyCode=${params['currencyCode']}
&isFetching=${params['isFetching']}
`.replace(/\s/g, ''); // removes the whitespaces we added to make it look nicer / easier to adjust.
const data = await
fetch(rightMoveAPISearch, {
"method": "GET",
})
.then(data => data.json())
.then(res => { return res })
if (data.resultCount) {
console.log('\x1b[32m%s\x1b[0m', 'Request successful! Result count: ', parseInt(data.resultCount));
console.log('All data: ', data);
console.log('Properties: ', data.properties);
}
else console.log('\x1b[31m%s\x1b[0m', `Issue with the request:`, data)
return data
}
Hope this is helpful. Let me know if you need anything else.

Http PUT deletes data that hasn't been changed

I am trying to edit a JSON data base of shifts. I am writing in Javascript using react.
This is my understanding of the PUT syntax:
const editShift = async (changed, id) => {
const res = await fetch(`http://localhost:5000/shifts/${id}`, {
method: 'PUT',
headers: {
'Content-type': 'application/json'
},
body: JSON.stringify(changed)
})
const data = await res.json()
setShifts([...shifts, data])
}
data.json:
{
"shifts": [
{
"title": "test",
"startDate": "2018-06-25T07:30:00.000Z",
"endDate": "2018-06-25T08:00:00.000Z",
"allDay": false,
"id": 1
},
{
"title": "test2",
"startDate": "2018-06-28T07:30:00.000Z",
"endDate": "2018-06-28T08:00:00.000Z",
"allDay": false,
"id": 2
}
]
}
The result is that the new shift will hold only the fields that have been changed and delete the rest. Any ideas why?

To get previous shifts you should use a callback inside setShifts like:
setShifts(prevShifts => ([...prevShifts, data]))

The PUT request should work like that.
A PUT request should PUT the data into place of the old record.
You would want to use a PATCH request for only PATCHing data upon the old record.
That set aside, it depends on your /shifts endpoint on how this is actually handled.

I had to send the whole updated shift into the changed field, with the fields that were not changed. I don't know why but it works so okay.

For a question like this, it'd be helpful if you posted before and after examples of data
database empty
request #1, payload equals X
request #2, payload equals Y
Also, knowing what your Server handler is doing would help determine what's going on. I have a feeling what's happening is something like this
// API handler
(req) => {
const dbData = getDBData();
dbData.shifts = req.shifts; // overwrite old with new
}
When what you need is something like
(req) => {
const dbData = getDBData();
req.shifts.forEach((newShiftData) => {
const shiftId = newShiftData.id;
if (dbData[shiftId]) {
dbData[shiftId] = { ...dbData[shiftId], ...newShiftData }; // merge old with new
}
});
// - check if data changed
// - update DB
}

PUT requests replace all the data at the target URI. They don't automatically merge changes.
It's not clear if you are the author of this API or just a consumer, but if you are consuming the API you should either look for a different API that lets you only send edits (HTTP method might be POST or PATCH), or you need to send the entire list of shifts.
Alternatively it might also be possible that there's a specific endpoint for each shift, so you can send exactly 1 PUT request to a shift endpoint just to edit that specific endpoint.

Dialogflow Fulfilment webhook call failed

I am new to dialogflow fulfillment and I am trying to retrieve news from news API based on user questions. I followed documentation provided by news API, but I am not able to catch any responses from the search results, when I run the function in console it is not errors. I changed the code and it looks like now it is reaching to the newsapi endpoint but it is not fetching any results. I am utilizing https://newsapi.org/docs/client-libraries/node-js to make a request to search everything about the topic. when I diagnoise the function it says " Webhook call failed. Error: UNAVAILABLE. "
'use strict';
const functions = require('firebase-functions');
const {WebhookClient} = require('dialogflow-fulfillment');
const {Card, Suggestion} = require('dialogflow-fulfillment');
const http = require('http');
const host = 'newsapi.org';
const NewsAPI = require('newsapi');
const newsapi = new NewsAPI('63756dc5caca424fb3d0343406295021');
process.env.DEBUG = 'dialogflow:debug';
exports.dialogflowFirebaseFulfillment = functions.https.onRequest((req, res) =>
{
// Get the city
let search = req.body.queryResult.parameters['search'];// search is a required param
// Call the weather API
callNewsApi(search).then((response) => {
res.json({ 'fulfillmentText': response }); // Return the results of the news API to Dialogflow
}).catch((xx) => {
console.error(xx);
res.json({ 'fulfillmentText': `I don't know the news but I hope it's good!` });
});
});
function callNewsApi(search)
{
console.log(search);
newsapi.v2.everything
(
{
q: 'search',
langauge: 'en',
sortBy: 'relevancy',
source: 'cbc-news',
domains: 'cbc.ca',
from: '2019-12-31',
to: '2020-12-12',
page: 2
}
).then (response => {console.log(response);
{
let articles = response['data']['articles'][0];
// Create response
let responce = `Current news in the $search with following title is ${articles['titile']} which says that
${articles['description']}`;
// Resolve the promise with the output text
console.log(output);
}
});
}
Also here is RAW API response
{
"responseId": "a871b8d2-16f2-4873-a5d1-b907a07adb9a-b4ef8d5f",
"queryResult": {
"queryText": "what is the latest news about toronto",
"parameters": {
"search": [
"toronto"
]
},
"allRequiredParamsPresent": true,
"fulfillmentMessages": [
{
"text": {
"text": [
""
]
}
}
],
"intent": {
"name": "projects/misty-ktsarh/agent/intents/b52c5774-e5b7-494a-8f4c-f783ebae558b",
"displayName": "misty.news"
},
"intentDetectionConfidence": 1,
"diagnosticInfo": {
"webhook_latency_ms": 543
},
"languageCode": "en"
},
"webhookStatus": {
"code": 14,
"message": "Webhook call failed. Error: UNAVAILABLE."
},
"outputAudio": "UklGRlQqAABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YTAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA... (The content is truncated. Click `COPY` for the original JSON.)",
"outputAudioConfig": {
"audioEncoding": "OUTPUT_AUDIO_ENCODING_LINEAR_16",
"synthesizeSpeechConfig": {
"speakingRate": 1,
"voice": {}
}
}
}
And Here is fulfillment request:
{
"responseId": "a871b8d2-16f2-4873-a5d1-b907a07adb9a-b4ef8d5f",
"queryResult": {
"queryText": "what is the latest news about toronto",
"parameters": {
"search": [
"toronto"
]
},
"allRequiredParamsPresent": true,
"fulfillmentMessages": [
{
"text": {
"text": [
""
]
}
}
],
"intent": {
"name": "projects/misty-ktsarh/agent/intents/b52c5774-e5b7-494a-8f4c-f783ebae558b",
"displayName": "misty.news"
},
"intentDetectionConfidence": 1,
"diagnosticInfo": {
"webhook_latency_ms": 543
},
"languageCode": "en"
},
"webhookStatus": {
"code": 14,
"message": "Webhook call failed. Error: UNAVAILABLE."
},
"outputAudio": "UklGRlQqAABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YTAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA... (The content is truncated. Click `COPY` for the original JSON.)",
"outputAudioConfig": {
"audioEncoding": "OUTPUT_AUDIO_ENCODING_LINEAR_16",
"synthesizeSpeechConfig": {
"speakingRate": 1,
"voice": {}
}
}
}
Also here is the screenshot from the firebase console.
Can anyone guide me what is that I am missing in here?

The key is the first three lines in the error message:
Function failed on loading user code. Error message: Code in file index.js can't be loaded.
Did you list all required modules in the package.json dependencies?
Detailed stack trace: Error: Cannot find module 'newsapi'
It is saying that the newsapi module couldn't be loaded and that the most likely cause of this is that you didn't list this as a dependency in your package.json file.
If you are using the Dialogflow Inline Editor, you need to select the package.json tab and add a line in the dependencies section.
Update
It isn't clear exactly when/where you're getting the "UNAVAILABLE" error, but one likely cause if you're using Dialogflow's Inline Editor is that it is using the Firebase "Spark" pricing plan, which has limitations on network calls outside Google's network.
You can upgrade to the Blaze plan, which does require a credit card on file, but does include the Spark plan's free tier, so you shouldn't incur any costs during light usage. This will allow for network calls.
Update based on TypeError: Cannot read property '0' of undefined
This indicates that either a property (or possibly an index of a property) is trying to reference against something that is undefined.
It isn't clear which line, exactly, this may be, but these lines all are suspicious:
let response = JSON.parse(body);
let source = response['data']['source'][0];
let id = response['data']['id'][0];
let name = response['data']['name'][0];
let author = response['author'][0];
let title = response['title'][0];
let description = response['description'][0];
since they are all referencing a property. I would check to see exactly what comes back and gets stored in response. For example, could it be that there is no "data" or "author" field in what is sent back?
Looking at https://newsapi.org/docs/endpoints/everything, it looks like none of these are fields, but that there is an articles property sent back which contains an array of articles. You may wish to index off that and get the attributes you want.
Update
It looks like that, although you are loading the parameter into a variable with this line
// Get the city and date from the request
let search = req.body.queryResult.parameters['search'];// city is a required param
You don't actually use the search variable anywhere. Instead, you seem to be passing a literal string "search" to your function with this line
callNewsApi('search').then((output) => {
which does a search for the word "search", I guess.
You indicated that "it goes to the catch portion", which indicates that something went wrong in the call. You don't show any logging in the catch portion, and it may be useful to log the exception that is thrown, so you know why it is going to the catch portion. Something like
}).catch((xx) => {
console.error(xx);
res.json({ 'fulfillmentText': `I don't know the news but I hope it's good!` });
});
is normal, but since it looks like you're logging it in the .on('error') portion, showing that error might be useful.

The name of the intent and the variable I was using to make the call had a difference in Casing, I guess calls are case sensitive just be aware of that

pouchDB query() bug when updating documents

Let's say I have these three documents:
{ "_id": "11111", "type": "template", "name": "person" }
{ "_id": "22222", "type": "template", "name": "place" }
{ "_id": "33333", "type": "template", "name": "thing" }
I have a cloud database and then I have a device with pouchDB syncing from that database.
These are the steps that I do:
I sync both databases together. So now I have the most recent versions of this document on my device.
I run the below query and I get back all three templates like so:
Code
var template_obj = {};
return device_db.query('filters/templates')
.then((templates) => {
for (let t of templates.rows) templates_obj[t.id] = true;
return templates_obj;
});
filters/templates
function (doc) {
if(doc.type == "template")
emit(doc._id);
}
return
{ "11111": true, "22222": true, "33333": true }
I update template: person on cloud. And then I update it again. So 2 revisions have gone by without syncing to my device.
I sync with my device.
Now when I run the same query and I only get back the document I edited. Which is weird because I haven't touched any of the other documents. The same view returns the expected results on the cloud but not on the device.
return
{"11111": true}
If I do the following code however, all templates come back as normal and the same _rev from the cloud show up on the device. Meaning the sync was successful and view is getting confused.
new code
return device_db.allDocs({conflicts: true})
.then((data) => {
for (let d of data.rows) {
if(d.doc.type == "template") {
templates_obj[d.doc._id] = true;
}
}
return templates_obj;
}).catch((err) => {
console.log(JSON.stringify(err));
})
I'm starting to believe this is a bug because if I destroy my database and do these steps again, I can reproduce this issue.

After realizing you are using React Native, I think this actually has to do with PouchDB in React Native, and it's indeed a bug. There are several reports of that behavior:
https://github.com/pouchdb/pouchdb/issues/7219
https://github.com/pouchdb/pouchdb/issues/7188
https://github.com/pouchdb/pouchdb/issues/7293

[edit: Seems to be a bug in PouchDB with React Native. I leave this answer because it might be helpful in other ways.]
I suspect it's some side effect with the global variable template_obj you are using. Try to console.log(templates.rows) directly instead of storing it in a variable in the top scope, or use Array.reduce() to avoid side effects. Then you'd always get the correct view results.
This is step by step code:
return device_db.query('filters/templates')
.then(templates => templates.rows) // Take only the rows into account.
.then(rows => rows.map(row => row.id) // Extract the id. If you wanted the name instead this would be possible with a slightly different view.
// I think it would suffice to log the result right now,
// but if you really want to have a single object with boolean values,
// you can do the following:
.then(ids => ids.reduce((asObject, id) => { // Use Array.reduce() here to avoid any potential side effects.
asObject[id] = true;
return asObject;
}, {})
.then(asObject => { console.log(asObject); }; // Debug the result.
Or more concise with ES2015+:
return device_db.query('filters/templates')
.then(({rows}) => rows.reduce((acc, {id}) => ({...acc, [id]: true }), {}))
.then(result => console.log(result))
By the way: You could also use other strategies to "filter" your documents, as it's not necessary to emit the _id. Instead you can use the key and/or value for "secondary indexes":
{
"_id": "_design/docs",
"views": {
"byType": "function(doc) { emit(doc.type); }",
"templatesByName": "function(doc) { if (doc.type === 'template') emit(doc.name); }",
"byTypeAndName": "function(doc) { emit([doc.type, doc.name], doc.name); }
}
}
you can use docs/byType as an universal view for other doc types too. Just call it with db.query('docs/byType', { key: 'template' })
If you want the templates sorted by name, use db.query('docs/templatesByName') or db.query('docs/byTypeAndName', { startkey: ['template'], endkey: ['template', {}]}).
A word of caution: This is all untested and just from memory, so some brackets might be missing in the code, or some bugs might hide in there.

It's not a bug in PDB, it's about outdated unfortunately components in pouchdb-react-native.
Confirmed way is to combine pouchdb-react-native yourself like this - then queries work as expected:
import PouchDB from 'pouchdb-core';
PouchDB
.plugin(require('pouchdb-adapter-asyncstorage').default)
.plugin(require('pouchdb-adapter-http'))
.plugin(require('pouchdb-mapreduce'))
.plugin(require('pouchdb-replication'))
.plugin(require('pouchdb-authentication'));
const localDB = new PouchDB(localDBname, {adapter: 'asyncstorage', auto_compaction: true});
This way one can be sure that all components are the latest.

Looking for design pattern to create multiple models/collections out of single JSON responses

I have a Backbone application where the JSON I get from the server isn't exactly 1 on 1 with how I want my models to look. I use custom parse functions for my models, ex:
parse: function(response) {
var content = {};
content.id = response.mediaId;
content.image = response.image.url;
return content;
}
This works. But, in some cases I have an API call where I get lots of information at once, for instance, information about an image with its user and comments:
{
"mediaId": "1",
"image": {
"title": "myImage",
"url": "http://image.com/234.jpg"
},
"user": {
"username": "John"
},
"comments": [
{
"title": "Nice pic!"
},
{
"title": "Great stuff."
}
]
}
How would I go about creating a new User model and a Comments collection from here? This is an option:
parse: function(response) {
var content = {};
content.id = response.mediaId;
content.image = response.image.url;
content.user = new User(response.user);
content.comments = new Comments(response.comments);
return content;
}
The trouble here is, by creating a new User or new Comments with raw JSON as input, Backbone will just add the JSON properties as attributes. Instead, I'd like to have an intermediate parse-like method to gain control over the objects' structure. The following is an option:
parse: function(response) {
// ...
content.user = new User({
username: response.user.username
});
// ...
}
...but that's not very DRY-proof.
So, my question is: what would be a nice pattern to create several models/collections out of 1 JSON response, with control over the models/collections attributes?
Thanks!

It may not be the nicest way possible, but this is how I do it:
content.user = new User(User.prototype.parse(response.user));
The only problem is that the this context in User.parse will be wrong. If you don't have any specific code in the User constructor, you can also do:
content.user = new User();
content.user.set(user.parse(response.user));
I also noticed an interesting note in the Backbone version 0.9.9 change log:
The parse function is now always run if defined, for both collections and models — not only after an Ajax call.
And looking at the source code of Model and Collection constructor, they do it like so:
if (options && options.parse) attrs = this.parse(attrs);
Maybe upgrading to 0.9.9 will give you what you need? If upgrade is not an option, you can of course implement the same in your own constructor.