Develop Reference

GET request to webpage returns string, but cannot use it

I am creating a google chrome extension, and when I make a get request to this web page https://www.rightmove.co.uk/property-for-sale/find.html?locationIdentifier=REGION%5E27675&maxBedrooms=2&minBedrooms=2&sortType=6&propertyTypes=&mustHave=&dontShow=&furnishTypes=&keywords=, I get the webpage HTML as a response which is what I want (the website I am requesting info from does not have an API and I cannot web scrape for reasons too long to explain here). This response comes in the form of a string. When I attempt to split this string at a certain point, bis_skin_checked, I am returned an array of length 1, meaning that there was no match and nothing has been split. But when I look at the string returned it has it included.
I have tried things like removing spaces and carriage returns but nothing seems to be working. This is my GET request code:
function getNewPage(url) {
let returnedValue = fetch(url, {
method: 'GET',
headers: {
'Content-Type': 'text/html',
},
})
.then(response => response.text())
.then(text => {
return text
})
return returnedValue
}
I then go on to resolve the promise which is returnedValue:
let newURL = getHousePrices(currentUrl) // Get Promise of new page as a string
newURL.then(function(value) { // Resolve promise and do stuff
console.log(value.split('bis_skin_checked').length)
})
And then work with the string which looks like this: (I have attached an image as I cannot copy the text from the popup)
Image Link To API Request return

I'm assuming you want to get the home values, given certain search parameters.
Scraping raw text is usually not the way to go. I took a look at the site and saw it uses uniform network requests that you can modify to capture the data you need directly instead of scraping the raw HTML.
I built a solution that allows you to dynamically pass whatever parameters you want to the getHomes() function. Passing nothing will use the default params that you can use as a baseline while you try to adjust the request for any additional modifications/use cases.
Install the below solution and run getHomes() from the service worker.
Here's a brief video I made explaining the solution: https://vimeo.com/772275354/07fd1025ed
--- manifest.JSON ---
{
"name": "UK Housing - Stackoverflow",
"description": "Example for how to make network requests and mimic them in background.js to avoid web scraping raw text",
"version": "1.0.0",
"manifest_version": 3,
"background": {
"service_worker": "background.js"
},
"host_permissions": [
"*://*.rightmove.co.uk/*"
]
}
--- background.js ---
async function getHomes(passedParams) {
const newParams = passedParams ? passedParams : {}; // set to an empty object if no new params passed - avoid error in object.entries loop.
// starts with default params for a working request, you can then pass params to override the defaults to test new requests.
var params = {
"locationIdentifier": "REGION%5E27675",
"maxBedrooms": "2",
"minBedrooms": "1",
"numberOfPropertiesPerPage": "25",
"radius": "0.0",
"sortType": "6",
"index": "0",
"viewType": "LIST",
"channel": "BUY",
"areaSizeUnit": "sqft",
"currencyCode": "GBP",
"isFetching": "false"
}
Object.entries(params).forEach(([key, value]) => {
// we iterate through each key in our default params to check if we passed a new value for that key.
// we then set the params object to the new value if we did.
if (newParams[key]) params[key] = newParams[key];
});
const rightMoveAPISearch = `https://www.rightmove.co.uk/api/_search?
locationIdentifier=${params['locationIdentifier']}
&maxBedrooms=${params['maxBedrooms']}
&minBedrooms=${params['minBedrooms']}
&numberOfPropertiesPerPage=${params['numberOfPropertiesPerPage']}
&radius=${params['radius']}
&sortType=${params['sortType']}
&index=${params['index']}
&viewType=${params['viewType']}
&channel=${params['channel']}
&areaSizeUnit=${params['areaSizeUnit']}
&currencyCode=${params['currencyCode']}
&isFetching=${params['isFetching']}
`.replace(/\s/g, ''); // removes the whitespaces we added to make it look nicer / easier to adjust.
const data = await
fetch(rightMoveAPISearch, {
"method": "GET",
})
.then(data => data.json())
.then(res => { return res })
if (data.resultCount) {
console.log('\x1b[32m%s\x1b[0m', 'Request successful! Result count: ', parseInt(data.resultCount));
console.log('All data: ', data);
console.log('Properties: ', data.properties);
}
else console.log('\x1b[31m%s\x1b[0m', `Issue with the request:`, data)
return data
}
Hope this is helpful. Let me know if you need anything else.

Vue3 JS - How to access/print values stored in an object onto document <template>

New to Vue, JS and Firebase been taking tutorials for weeks and now trying to build a test project. Embarrassed to say I have been stuck on this particular problem for several days. Trying to write this question as best I can given my current level of understanding which is pretty basic.
Desired Outcome: What I am trying to do is query firebase database to get the last event created by current user so that, if it exists, I can then pre-fill some of the Vue form data when they create their next entry.
Sounds simple enough but my tutorials only cover single document retrieval/display when you already have the document id, OR, how to retrieve and display all documents from a collection in a list. My need is hybrid in that don't know the document ID until I query to get it and every attempt to combine the functionality from tutorials has failed...multiple times.
Currently, I am able to query the DB just fine and can verify it returns the results I want to an object which I can log to the console and 'see', but, can't for the life of me figure out how to get to the data within that object once back on the doc template.
JS Code (getLastEvent-SO.js)
import { ref } from 'vue'
import { db } from '#/firebase/config'
import { collection, getDocs, query, where, orderBy, limit } from 'firebase/firestore'
const getLastEvent = (u) => { // u = user id
const lastEvent = ref([])
let collectionRef = collection(db, 'events') // establish reference
collectionRef = query(collectionRef, where("userId", "==", u ), orderBy("created", "desc"), limit(1)) // should return last event if exists
getDocs(collectionRef)
.then(snapshot => {
let docs = []
let eId = null
snapshot.docs.forEach(doc => { // using a loop because I couldnt figure out how to get a single doc without first having the id of that doc which I don't know until I query to get users last event created
docs.push({ ...doc.data(), id: doc.id })
})
lastEvent.value = docs // assign value of docs to lastEvent
})
console.log('lastEvent1:', lastEvent) // console display proves the record stored is the one I want but how do I access/print to console the values?
return { lastEvent }
}
export default getLastEvent
Vue Code:
<template>
<form #submit.prevent="handleSubmit">
<h2>New event</h2>
<p v-if="lastEvent">Last Start Date: {{ lastEvent.startDate }} </p> <!-- PROBLEM: this will NOT display - why??? -->
...more form inputs...
<button>Add event</button>
</form>
</template>
<script>
import getUser from "#/composables/getUser";
import getLastEvent from "#/composables/getLastEvent-SO";
export default {
setup() {
// Query last user event created so default form values based on last entry can be set
const { user } = getUser();
const { lastEvent } = getLastEvent(user.value.uid)
return { lastEvent };
},
};
</script>
Copied from Chrome/console (lastEvent):
{
"__v_isShallow": false,
"dep": {
"w": 0,
"n": 0
},
"__v_isRef": true,
"_rawValue": [
{
"userId": "H0jZIKGTggeX4DsjucGR50Cb8663",
"reservation": "",
"startDate": "2022-07-07",
"created": 1656595528420,
"location": "mpmCHrVYDK9icr3yTnYG",
"startTime": "09:00a",
"type": "Open",
"duration": 1,
"id": "ZHnbrLWkYO9gHdjKiygk"
}
],
"_value": [
{
"userId": "H0jZIKGTggeX4DsjucGR50Cb8663",
"reservation": "",
"startDate": "2022-07-07",
"created": 1656595528420,
"location": "mpmCHrVYDK9icr3yTnYG",
"startTime": "09:00a",
"type": "Open",
"duration": 1,
"id": "ZHnbrLWkYO9gHdjKiygk"
}
]
}
Any hint as to why I am unable to access the data from within the document template as {{ lastEvent.startDate }} ??
[edit: added this screen-shot of the console where I copied the object from by request]
Screen shot of console

Aside from getLastEvent being asynchronous (make sure you await your database call before returning lastEvent), what you're returning from the function is an array, while in your Vue file you're using it as if it's an object. Either set lastEvent.value equal to desired element in your array (preferred)
const snapshot = await getDocs(collectionRef);
...
lastEvent.value = docs[0]; // assign value of docs to lastEvent
or in your Vue file you can call (though you still need to make your function asynchronous):
<p v-if="lastEvent">Last Start Date: {{ lastEvent[0].startDate }}</p>

Blank screen in vue.js if not using created() life cycle hook or spread operator (...)

The following example gives me a blank screen (jsfiddle here). Even the parts which have nothing to do with the loop are not being rendered.
HTML:
<div id="app">
<button #click="objectFromApi">
run objectFromApi function
</button>
<div
v-for="obj in myObject[0].results"
:key="obj.id"
>
<p>
{{ obj.message }}
</p>
</div>
</div>
JavaScript:
new Vue({
el: "#app",
data: {
myObject: []
},
methods: {
objectFromApi: function(){
this.myObject.push(
{
"count": 5,
"results": [
{
"id": 1,
"message": "object 1"
},
{
"id": 2,
"message": "object 2"
}
]
}
)
}
},
//created() {
// this.objectFromApi()
//}
})
Nevertheless it does work if:
1.) Either using objectFromApi function directly in the created life cycle hook (what I don't want!)
created() {
this.objectFromApi()
}
2.) Or (without the use of created life cycle hook) if I go directly into the nested results array and spread the objects out like this (what I also don't want!)
this.myObject.push(
...{
"count": 5,
"next": "http://127.0.0.1:8000/api/someurl/?page=2",
"previous": null,
"results": [
{
"id": 1,
"message": "object 1"
},
{
"id": 2,
"message": "object 2"
}
]
}.results
)
When using option 2.) of course the v-for loop has to look different:
v-for="obj in myObject" instead of v-for="obj in myObject[0].results"
What is wrong with my initial example?

When the component is first rendering the array myObject will be empty.
During rendering it attempts this:
<div
v-for="obj in myObject[0].results"
:key="obj.id"
>
The value of myObject[0] will be undefined. Attempting to access the results property of undefined will result in an error. This error will cause rendering to fail. Nothing will be shown, even the parts that didn't fail.
There are various ways to fix this problem. You could prepopulate the data with suitable empty properties:
data: {
myObject: [
{
results: []
}
]
}
Alternatively, as you've noted, you could change the loop to use v-for="obj in myObject", changing objectFromApi accordingly to only store the results array in myObject. Even if you don't want that exact change some similar change is probably a good idea because the [0] part strongly suggests you've got a problem with your data model. The key thing here is that it avoids trying to access nested objects that don't exist. The use of the spread operator in your second example is largely irrelevant.
Or you could skip the loop in the template:
<template v-if="myObject[0]">
<div
v-for="obj in myObject[0].results"
:key="obj.id"
>
...
</div>
</template>

Dialogflow Fulfilment webhook call failed

I am new to dialogflow fulfillment and I am trying to retrieve news from news API based on user questions. I followed documentation provided by news API, but I am not able to catch any responses from the search results, when I run the function in console it is not errors. I changed the code and it looks like now it is reaching to the newsapi endpoint but it is not fetching any results. I am utilizing https://newsapi.org/docs/client-libraries/node-js to make a request to search everything about the topic. when I diagnoise the function it says " Webhook call failed. Error: UNAVAILABLE. "
'use strict';
const functions = require('firebase-functions');
const {WebhookClient} = require('dialogflow-fulfillment');
const {Card, Suggestion} = require('dialogflow-fulfillment');
const http = require('http');
const host = 'newsapi.org';
const NewsAPI = require('newsapi');
const newsapi = new NewsAPI('63756dc5caca424fb3d0343406295021');
process.env.DEBUG = 'dialogflow:debug';
exports.dialogflowFirebaseFulfillment = functions.https.onRequest((req, res) =>
{
// Get the city
let search = req.body.queryResult.parameters['search'];// search is a required param
// Call the weather API
callNewsApi(search).then((response) => {
res.json({ 'fulfillmentText': response }); // Return the results of the news API to Dialogflow
}).catch((xx) => {
console.error(xx);
res.json({ 'fulfillmentText': `I don't know the news but I hope it's good!` });
});
});
function callNewsApi(search)
{
console.log(search);
newsapi.v2.everything
(
{
q: 'search',
langauge: 'en',
sortBy: 'relevancy',
source: 'cbc-news',
domains: 'cbc.ca',
from: '2019-12-31',
to: '2020-12-12',
page: 2
}
).then (response => {console.log(response);
{
let articles = response['data']['articles'][0];
// Create response
let responce = `Current news in the $search with following title is ${articles['titile']} which says that
${articles['description']}`;
// Resolve the promise with the output text
console.log(output);
}
});
}
Also here is RAW API response
{
"responseId": "a871b8d2-16f2-4873-a5d1-b907a07adb9a-b4ef8d5f",
"queryResult": {
"queryText": "what is the latest news about toronto",
"parameters": {
"search": [
"toronto"
]
},
"allRequiredParamsPresent": true,
"fulfillmentMessages": [
{
"text": {
"text": [
""
]
}
}
],
"intent": {
"name": "projects/misty-ktsarh/agent/intents/b52c5774-e5b7-494a-8f4c-f783ebae558b",
"displayName": "misty.news"
},
"intentDetectionConfidence": 1,
"diagnosticInfo": {
"webhook_latency_ms": 543
},
"languageCode": "en"
},
"webhookStatus": {
"code": 14,
"message": "Webhook call failed. Error: UNAVAILABLE."
},
"outputAudio": "UklGRlQqAABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YTAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA... (The content is truncated. Click `COPY` for the original JSON.)",
"outputAudioConfig": {
"audioEncoding": "OUTPUT_AUDIO_ENCODING_LINEAR_16",
"synthesizeSpeechConfig": {
"speakingRate": 1,
"voice": {}
}
}
}
And Here is fulfillment request:
{
"responseId": "a871b8d2-16f2-4873-a5d1-b907a07adb9a-b4ef8d5f",
"queryResult": {
"queryText": "what is the latest news about toronto",
"parameters": {
"search": [
"toronto"
]
},
"allRequiredParamsPresent": true,
"fulfillmentMessages": [
{
"text": {
"text": [
""
]
}
}
],
"intent": {
"name": "projects/misty-ktsarh/agent/intents/b52c5774-e5b7-494a-8f4c-f783ebae558b",
"displayName": "misty.news"
},
"intentDetectionConfidence": 1,
"diagnosticInfo": {
"webhook_latency_ms": 543
},
"languageCode": "en"
},
"webhookStatus": {
"code": 14,
"message": "Webhook call failed. Error: UNAVAILABLE."
},
"outputAudio": "UklGRlQqAABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YTAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA... (The content is truncated. Click `COPY` for the original JSON.)",
"outputAudioConfig": {
"audioEncoding": "OUTPUT_AUDIO_ENCODING_LINEAR_16",
"synthesizeSpeechConfig": {
"speakingRate": 1,
"voice": {}
}
}
}
Also here is the screenshot from the firebase console.
Can anyone guide me what is that I am missing in here?

The key is the first three lines in the error message:
Function failed on loading user code. Error message: Code in file index.js can't be loaded.
Did you list all required modules in the package.json dependencies?
Detailed stack trace: Error: Cannot find module 'newsapi'
It is saying that the newsapi module couldn't be loaded and that the most likely cause of this is that you didn't list this as a dependency in your package.json file.
If you are using the Dialogflow Inline Editor, you need to select the package.json tab and add a line in the dependencies section.
Update
It isn't clear exactly when/where you're getting the "UNAVAILABLE" error, but one likely cause if you're using Dialogflow's Inline Editor is that it is using the Firebase "Spark" pricing plan, which has limitations on network calls outside Google's network.
You can upgrade to the Blaze plan, which does require a credit card on file, but does include the Spark plan's free tier, so you shouldn't incur any costs during light usage. This will allow for network calls.
Update based on TypeError: Cannot read property '0' of undefined
This indicates that either a property (or possibly an index of a property) is trying to reference against something that is undefined.
It isn't clear which line, exactly, this may be, but these lines all are suspicious:
let response = JSON.parse(body);
let source = response['data']['source'][0];
let id = response['data']['id'][0];
let name = response['data']['name'][0];
let author = response['author'][0];
let title = response['title'][0];
let description = response['description'][0];
since they are all referencing a property. I would check to see exactly what comes back and gets stored in response. For example, could it be that there is no "data" or "author" field in what is sent back?
Looking at https://newsapi.org/docs/endpoints/everything, it looks like none of these are fields, but that there is an articles property sent back which contains an array of articles. You may wish to index off that and get the attributes you want.
Update
It looks like that, although you are loading the parameter into a variable with this line
// Get the city and date from the request
let search = req.body.queryResult.parameters['search'];// city is a required param
You don't actually use the search variable anywhere. Instead, you seem to be passing a literal string "search" to your function with this line
callNewsApi('search').then((output) => {
which does a search for the word "search", I guess.
You indicated that "it goes to the catch portion", which indicates that something went wrong in the call. You don't show any logging in the catch portion, and it may be useful to log the exception that is thrown, so you know why it is going to the catch portion. Something like
}).catch((xx) => {
console.error(xx);
res.json({ 'fulfillmentText': `I don't know the news but I hope it's good!` });
});
is normal, but since it looks like you're logging it in the .on('error') portion, showing that error might be useful.

The name of the intent and the variable I was using to make the call had a difference in Casing, I guess calls are case sensitive just be aware of that

Trying to access my TypeScript class properties so that I can display in HTML

I am trying to redevelop a project I am working on into Angular 2 from Angular 1. Currently I am using Ionic to build it as it is used on iOS. This redevelopment is my first interaction with TypeScript at all.
So far I have been able to get most of what I need done. But now my issue is that I want to access a property of my class and display it in the HTML code but only after it has been set, this way I wouldn't run into errors. But my code or at least Ionic says that the property is undefined even if I use the ngOnInit or ngAfterContentInit functions to try and do this.
I may be going about this code the wrong way but I would like to ask how to access these properties so that I can display them in the HTML as *ngFor= 'let e of EventInfo' then {{e.eventName}}. Please help me out. I will continue to search the previous questions to hopefully find some inspiration to answer my question.
I will add my class code so far below.
To add some more information, the e.eventName I am planning to use on a menu that toggles out and it will display the information I want from eventInfo.
For the code below, I also added what pullJson.getMondayJson looks like.
This is what eventInfo initially looked like when I tried it out.
this.eventInfo = [{rType: this.mondayJson.agendaEvents[0].rowType, eventName:this.mondayJson.agendaEvents[0].eventName, startTime:this.mondayJson.agendaEvents[0].startTime, endTime:this.mondayJson.agendaEvents[0].endTime}];
#Component({
selector: 'center-panel',
host: {'(window:scroll)': 'track($event)'},
templateUrl: 'center-panel.html'
})
export class CenterPanel {
mondayJson;
tuesdayJson;
wednesdayJson;
pages: Array<{title: string}>;
eventInfo: Array<{eventName: string, startTime: any, endTime: any}>;
public constructor(public pullJson:PullJson, public menu: MenuController, locationA2:Location) {
this.pages = [
{ title: 'CenterPanel'},
{ title: 'RightPanel'}
];
pullJson.getMondayJson
.subscribe(
getMondayJson => this.mondayJson = {
dayOfWeek: getMondayJson.dayOfWeek.toUpperCase(),
agendaEvents: getMondayJson.agendaEvents
},
console.error,
() => console.log('Completed HTTP for Monday!!')
);
}
this.getMondayJson = this.http.get('https://../remote-server-file.json')
.map(response => response.json());
Someone asked me to post the template code:
<div id="{{ 'MONDAY-events-' + event.startTime }}" menuToggle class="agenda-event-container" *ngFor="let event of mondayJson.agendaEvents" (click)="event.itemClicked = !event.itemClicked">
<div class="agenda-event-row {{event.rowType}}">
<div class="time-container">
<div class="event-left-border-{{event.iconType}}">
<div class="start-time">
{{event.startTime}}
</div>
<div id="MONDAY-events-endTime" class="end-time">
{{event.endTime}}
</div>
</div>
</div>
</div>
</div>
</div>
Here is an example of what the Json files look like:
{
"rowType": "event",
"eventName": "Facilitator Training",
"iconType": "workshop",
"startTime": "8:00AM",
"endTime": "10:00AM",
"headerLocation": "Go To Brooklyn",
"locationDetails": {
"jsonGroupFile": {
"subData": "",
},
"hardcodedList": ["<b>Test:<br>New York"]
},
"subEvents": [
{
"presentationName": "",
"durationInMinutes": "120",
"speakers": "Test Person"
}
],
"images": [
{
"imageType": "hotel"
},
{
"imageType": "map"
}
],
"files": []
}