Develop Reference

GET request to webpage returns string, but cannot use it

I am creating a google chrome extension, and when I make a get request to this web page https://www.rightmove.co.uk/property-for-sale/find.html?locationIdentifier=REGION%5E27675&maxBedrooms=2&minBedrooms=2&sortType=6&propertyTypes=&mustHave=&dontShow=&furnishTypes=&keywords=, I get the webpage HTML as a response which is what I want (the website I am requesting info from does not have an API and I cannot web scrape for reasons too long to explain here). This response comes in the form of a string. When I attempt to split this string at a certain point, bis_skin_checked, I am returned an array of length 1, meaning that there was no match and nothing has been split. But when I look at the string returned it has it included.
I have tried things like removing spaces and carriage returns but nothing seems to be working. This is my GET request code:
function getNewPage(url) {
let returnedValue = fetch(url, {
method: 'GET',
headers: {
'Content-Type': 'text/html',
},
})
.then(response => response.text())
.then(text => {
return text
})
return returnedValue
}
I then go on to resolve the promise which is returnedValue:
let newURL = getHousePrices(currentUrl) // Get Promise of new page as a string
newURL.then(function(value) { // Resolve promise and do stuff
console.log(value.split('bis_skin_checked').length)
})
And then work with the string which looks like this: (I have attached an image as I cannot copy the text from the popup)
Image Link To API Request return

I'm assuming you want to get the home values, given certain search parameters.
Scraping raw text is usually not the way to go. I took a look at the site and saw it uses uniform network requests that you can modify to capture the data you need directly instead of scraping the raw HTML.
I built a solution that allows you to dynamically pass whatever parameters you want to the getHomes() function. Passing nothing will use the default params that you can use as a baseline while you try to adjust the request for any additional modifications/use cases.
Install the below solution and run getHomes() from the service worker.
Here's a brief video I made explaining the solution: https://vimeo.com/772275354/07fd1025ed
--- manifest.JSON ---
{
"name": "UK Housing - Stackoverflow",
"description": "Example for how to make network requests and mimic them in background.js to avoid web scraping raw text",
"version": "1.0.0",
"manifest_version": 3,
"background": {
"service_worker": "background.js"
},
"host_permissions": [
"*://*.rightmove.co.uk/*"
]
}
--- background.js ---
async function getHomes(passedParams) {
const newParams = passedParams ? passedParams : {}; // set to an empty object if no new params passed - avoid error in object.entries loop.
// starts with default params for a working request, you can then pass params to override the defaults to test new requests.
var params = {
"locationIdentifier": "REGION%5E27675",
"maxBedrooms": "2",
"minBedrooms": "1",
"numberOfPropertiesPerPage": "25",
"radius": "0.0",
"sortType": "6",
"index": "0",
"viewType": "LIST",
"channel": "BUY",
"areaSizeUnit": "sqft",
"currencyCode": "GBP",
"isFetching": "false"
}
Object.entries(params).forEach(([key, value]) => {
// we iterate through each key in our default params to check if we passed a new value for that key.
// we then set the params object to the new value if we did.
if (newParams[key]) params[key] = newParams[key];
});
const rightMoveAPISearch = `https://www.rightmove.co.uk/api/_search?
locationIdentifier=${params['locationIdentifier']}
&maxBedrooms=${params['maxBedrooms']}
&minBedrooms=${params['minBedrooms']}
&numberOfPropertiesPerPage=${params['numberOfPropertiesPerPage']}
&radius=${params['radius']}
&sortType=${params['sortType']}
&index=${params['index']}
&viewType=${params['viewType']}
&channel=${params['channel']}
&areaSizeUnit=${params['areaSizeUnit']}
&currencyCode=${params['currencyCode']}
&isFetching=${params['isFetching']}
`.replace(/\s/g, ''); // removes the whitespaces we added to make it look nicer / easier to adjust.
const data = await
fetch(rightMoveAPISearch, {
"method": "GET",
})
.then(data => data.json())
.then(res => { return res })
if (data.resultCount) {
console.log('\x1b[32m%s\x1b[0m', 'Request successful! Result count: ', parseInt(data.resultCount));
console.log('All data: ', data);
console.log('Properties: ', data.properties);
}
else console.log('\x1b[31m%s\x1b[0m', `Issue with the request:`, data)
return data
}
Hope this is helpful. Let me know if you need anything else.

is there a reason why a notification script would suddenly start throwing out Slack API Code 400: Invalid Blocks?

I've had this script working for over 3 months with no issues, however, last week it started throwing out this:
Error
Exception: Request failed for https://hooks.slack.com returned code 400. Truncated server response: invalid_blocks (use muteHttpExceptions option to examine full response)
Here's the relevant part of the script i think.
for (let i = 0; i in vistasVencimiento; i++){
if (vistasVencimiento[i][0] instanceof Date == true){
let differenceInMs = dateToday - vistasVencimiento[i][0];
let differenceInDaysRaw = differenceInMs / (1000 * 60 * 60 * 24);
if(differenceInDaysRaw >= 0 && statusColumn[i] !== "Pendiente_de_despacho"){
//agregar a array de vencidos
result.push(sumariantesVistas[i], causasCaratulas[i], '\n');
}
}
};
console.log(result);
if (result === undefined || result.length === 0 ){
let payload ={
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": ":bell: *Cuales Vistas estan vencidas?* :bell:"
}
},
{
"type": "section",
"text": {
"type": "mrkdwn",
"text":"Aparentemente, ninguna"
}
},
]
}
let headers = {
'Content-type': 'application/json'
};
let options = {
headers : headers,
method: 'POST',
payload: JSON.stringify(payload)
};
UrlFetchApp.fetch(url, options);
}else{
let result_string = result.join('\n');
let payload ={
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": ":bell: *Cuales Vistas estan vencidas?* :bell:"
}
},
{
"type": "section",
"text": {
"type": "mrkdwn",
"text":"Estas: "
}
},
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": result_string
}
},
]
}
let headers = {
'Content-type': 'application/json'
};
let options = {
muteHttpExceptions: false,
headers : headers,
method: 'POST',
payload: JSON.stringify(payload)
};
UrlFetchApp.fetch(url, options);
};
Now, I know its related to "text": result_string because ive tried replacing that with something like "text": "test" and it works, but the whole point of the notifications is to send that formatted string to a slack workplace channel, and thats where ive basically no clue on why it would start failing out of a sudden like that
edit:
Theres something genuinely very weird going on here, first of all going causasCaratulas[i].replace(/"/g, "'") doesnt fix anything, second, causasCaratulas has two columns inside, if i load only one column (doesnt matter which) the script works, ive even tried splitting the array into two sorta-lists, in that case, it still throws out the error.
So the error happens somewhere between trying to send out those two columns to the slack api at the same time? im clueless as to whats happening here
edit 2:
the data is structured something like this. the problem seems to be that whenever i try sending both Case Number and Case Name at the same time i get an error, however, this seems to be dependant on some kind of mystery symbol idk how to identify, if it even is a symbol.

Perhaps your text data contains &<>? Try this:
let result_string = result
.join('\n')
.replace(/&/g, '&')
.replace(/</g, '<')
.replace(/>/g, '>');
See Escaping text at Slack API reference.
You should also search your data for markdown symbols that could throw a spanner in the works.

How to access a specific element in JSON in Google Apps Script?

I need to access the nav for a specific date from the below JSON. eg : data[date="20-04-2022"].nav
How do I do it in Google Apps Script?
The standard JSON notation is not working.
{
"meta":{
"fund_house":"Mutual Fund",
"scheme_type":"Open Ended Schemes",
},
"data":[
{
"date":"22-04-2022",
"nav":"21.64000"
},
{
"date":"21-04-2022",
"nav":"21.69000"
},
{
"date":"20-04-2022",
"nav":"21.53000"
}
],
"status":"SUCCESS"
}

In your situation, I thought that it is required to retrieve the element including "date": "20-04-2022" from the array of data. So, how about the following sample script?
Sample script:
const obj = {
"meta": {
"fund_house": "Mutual Fund",
"scheme_type": "Open Ended Schemes",
},
"data": [
{
"date": "22-04-2022",
"nav": "21.64000"
},
{
"date": "21-04-2022",
"nav": "21.69000"
},
{
"date": "20-04-2022",
"nav": "21.53000"
}
],
"status": "SUCCESS"
};
const search = "20-04-2022";
const res = obj.data.find(({ date }) => date == search);
const value = res && res.nav;
console.log(value) // 21.53000
For example, if the search value is always found, you can use the following script.
const res2 = obj.data.find(({ date }) => date == search).nav;
Reference:
find()
Added 1:
From your following reply,
This looks like standard java script. Does not work in google apps script(script.google.com/home). Getting syntax error for this line : const res = obj.data.find(({ date }) => date == search);
I'm worried that you are not enabling V8 runtime. Ref If you cannot use V8 runtime, how about the following sample script?
Sample script:
var obj = {
"meta": {
"fund_house": "Mutual Fund",
"scheme_type": "Open Ended Schemes",
},
"data": [
{
"date": "22-04-2022",
"nav": "21.64000"
},
{
"date": "21-04-2022",
"nav": "21.69000"
},
{
"date": "20-04-2022",
"nav": "21.53000"
}
],
"status": "SUCCESS"
};
var search = "20-04-2022";
var res = obj.data.filter(function (e) { return e.date == search })[0];
var value = res && res.nav;
console.log(value) // 21.53000
Added 2:
From your following reply,
This looks like standard java script. Does not work in google apps script(script.google.com/home). Getting syntax error for this line : const res = obj.data.find(({ date }) => date == search);
I am trying to write a google apps script to fetch data from a url. But google seems to have its own way of handling the JSON data which I am unable to figure out. developers.google.com/apps-script/guides/services/…
I understood that your actual goal was to retrieve the value using Web Apps. If my understanding of your actual goal, how about the following sample script?
1. Sample script:
Please copy and paste the following script to the script editor and save the script.
function doGet(e) {
var search = e.parameter.search;
var obj = {
"meta": {
"fund_house": "Mutual Fund",
"scheme_type": "Open Ended Schemes",
},
"data": [
{
"date": "22-04-2022",
"nav": "21.64000"
},
{
"date": "21-04-2022",
"nav": "21.69000"
},
{
"date": "20-04-2022",
"nav": "21.53000"
}
],
"status": "SUCCESS"
};
var res = obj.data.filter(function (e) { return e.date == search })[0];
var value = res && res.nav;
return ContentService.createTextOutput(value);
}
I think that this sample script can be used with and without V8 runtime.
2. Deploy Web Apps.
In this case, please check the official document. Please set it as follows.
Execute as: Me
Anyone with Google account: Anyone
In this case, it supposes that you are using new IDE. Please be careful this.
3. Testing.
Please access to the URL like https://script.google.com/macros/s/{deploymentId}/exec?search=20-04-2022 using your browser. By this, the result value is returned.
`

JavaScript parsing nested JSON data

I'm creating a discord bot (using discord.js)
I'm writing a help command using pages in embeds, I have the pages working fine - however, the problem seems to come when trying to get the data from the JSON.
I have the JSON file setup in a larger config file, so it's nested inside of it.
Each command has it's name attached to it
{
"other-json-data": "other data",
"commands": {
"rule": {
"info": "This command gives the rules",
"expectedArgs": "<number>"
},
"invite": {
"info": "Invite new users",
"expectedArgs": "<no-args>"
},
"flip": {
"info": "Flip a coin",
"expectedArgs": "<no-args>"
}
},
"other-json-data": "other data"
}
I need to get the data from the commands area for each page.
I only have a integer input (from the page number), but I haven't got a clue how I would get the data from whatever command needs to be shown.
For something else in my project, I am using this to get the expectedArgs from the JSON object config.commands[arguments].expectedArgs, where the config is just a reference to the JSON file, this works perfectly fine. The arguments is a string input (i.e. rule), which returns whatever the info from that command.
However, would there be a way to get say the second one down (invite). I've tried config.commands[pageNumber].expectedArgs}, however, this doesn't seem to work. pageNumber would be an integer, so would it would get whatever value and then I could grab the expectedArgs.

You can get all keys from an object and select one using their index.
const json = {
"other-json-data": "other data",
"commands": {
"rule": {
"info": "This command gives the rules",
"expectedArgs": "<number>"
},
"invite": {
"info": "Invite new users",
"expectedArgs": "<no-args>"
},
"flip": {
"info": "Flip a coin",
"expectedArgs": "<no-args>"
}
},
"other-json-data": "other data"
}
const pageNumber = 1
// key will be a command name, e.g. 'invite'
const key = Object.keys(json.commands)[pageNumber]
const { expectedArgs } = json.commands[key]
console.log(`${key} expects ${expectedArgs}`)
Remember that indexes range starts at zero.

Dialogflow Fulfilment webhook call failed

I am new to dialogflow fulfillment and I am trying to retrieve news from news API based on user questions. I followed documentation provided by news API, but I am not able to catch any responses from the search results, when I run the function in console it is not errors. I changed the code and it looks like now it is reaching to the newsapi endpoint but it is not fetching any results. I am utilizing https://newsapi.org/docs/client-libraries/node-js to make a request to search everything about the topic. when I diagnoise the function it says " Webhook call failed. Error: UNAVAILABLE. "
'use strict';
const functions = require('firebase-functions');
const {WebhookClient} = require('dialogflow-fulfillment');
const {Card, Suggestion} = require('dialogflow-fulfillment');
const http = require('http');
const host = 'newsapi.org';
const NewsAPI = require('newsapi');
const newsapi = new NewsAPI('63756dc5caca424fb3d0343406295021');
process.env.DEBUG = 'dialogflow:debug';
exports.dialogflowFirebaseFulfillment = functions.https.onRequest((req, res) =>
{
// Get the city
let search = req.body.queryResult.parameters['search'];// search is a required param
// Call the weather API
callNewsApi(search).then((response) => {
res.json({ 'fulfillmentText': response }); // Return the results of the news API to Dialogflow
}).catch((xx) => {
console.error(xx);
res.json({ 'fulfillmentText': `I don't know the news but I hope it's good!` });
});
});
function callNewsApi(search)
{
console.log(search);
newsapi.v2.everything
(
{
q: 'search',
langauge: 'en',
sortBy: 'relevancy',
source: 'cbc-news',
domains: 'cbc.ca',
from: '2019-12-31',
to: '2020-12-12',
page: 2
}
).then (response => {console.log(response);
{
let articles = response['data']['articles'][0];
// Create response
let responce = `Current news in the $search with following title is ${articles['titile']} which says that
${articles['description']}`;
// Resolve the promise with the output text
console.log(output);
}
});
}
Also here is RAW API response
{
"responseId": "a871b8d2-16f2-4873-a5d1-b907a07adb9a-b4ef8d5f",
"queryResult": {
"queryText": "what is the latest news about toronto",
"parameters": {
"search": [
"toronto"
]
},
"allRequiredParamsPresent": true,
"fulfillmentMessages": [
{
"text": {
"text": [
""
]
}
}
],
"intent": {
"name": "projects/misty-ktsarh/agent/intents/b52c5774-e5b7-494a-8f4c-f783ebae558b",
"displayName": "misty.news"
},
"intentDetectionConfidence": 1,
"diagnosticInfo": {
"webhook_latency_ms": 543
},
"languageCode": "en"
},
"webhookStatus": {
"code": 14,
"message": "Webhook call failed. Error: UNAVAILABLE."
},
"outputAudio": "UklGRlQqAABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YTAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA... (The content is truncated. Click `COPY` for the original JSON.)",
"outputAudioConfig": {
"audioEncoding": "OUTPUT_AUDIO_ENCODING_LINEAR_16",
"synthesizeSpeechConfig": {
"speakingRate": 1,
"voice": {}
}
}
}
And Here is fulfillment request:
{
"responseId": "a871b8d2-16f2-4873-a5d1-b907a07adb9a-b4ef8d5f",
"queryResult": {
"queryText": "what is the latest news about toronto",
"parameters": {
"search": [
"toronto"
]
},
"allRequiredParamsPresent": true,
"fulfillmentMessages": [
{
"text": {
"text": [
""
]
}
}
],
"intent": {
"name": "projects/misty-ktsarh/agent/intents/b52c5774-e5b7-494a-8f4c-f783ebae558b",
"displayName": "misty.news"
},
"intentDetectionConfidence": 1,
"diagnosticInfo": {
"webhook_latency_ms": 543
},
"languageCode": "en"
},
"webhookStatus": {
"code": 14,
"message": "Webhook call failed. Error: UNAVAILABLE."
},
"outputAudio": "UklGRlQqAABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YTAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA... (The content is truncated. Click `COPY` for the original JSON.)",
"outputAudioConfig": {
"audioEncoding": "OUTPUT_AUDIO_ENCODING_LINEAR_16",
"synthesizeSpeechConfig": {
"speakingRate": 1,
"voice": {}
}
}
}
Also here is the screenshot from the firebase console.
Can anyone guide me what is that I am missing in here?

The key is the first three lines in the error message:
Function failed on loading user code. Error message: Code in file index.js can't be loaded.
Did you list all required modules in the package.json dependencies?
Detailed stack trace: Error: Cannot find module 'newsapi'
It is saying that the newsapi module couldn't be loaded and that the most likely cause of this is that you didn't list this as a dependency in your package.json file.
If you are using the Dialogflow Inline Editor, you need to select the package.json tab and add a line in the dependencies section.
Update
It isn't clear exactly when/where you're getting the "UNAVAILABLE" error, but one likely cause if you're using Dialogflow's Inline Editor is that it is using the Firebase "Spark" pricing plan, which has limitations on network calls outside Google's network.
You can upgrade to the Blaze plan, which does require a credit card on file, but does include the Spark plan's free tier, so you shouldn't incur any costs during light usage. This will allow for network calls.
Update based on TypeError: Cannot read property '0' of undefined
This indicates that either a property (or possibly an index of a property) is trying to reference against something that is undefined.
It isn't clear which line, exactly, this may be, but these lines all are suspicious:
let response = JSON.parse(body);
let source = response['data']['source'][0];
let id = response['data']['id'][0];
let name = response['data']['name'][0];
let author = response['author'][0];
let title = response['title'][0];
let description = response['description'][0];
since they are all referencing a property. I would check to see exactly what comes back and gets stored in response. For example, could it be that there is no "data" or "author" field in what is sent back?
Looking at https://newsapi.org/docs/endpoints/everything, it looks like none of these are fields, but that there is an articles property sent back which contains an array of articles. You may wish to index off that and get the attributes you want.
Update
It looks like that, although you are loading the parameter into a variable with this line
// Get the city and date from the request
let search = req.body.queryResult.parameters['search'];// city is a required param
You don't actually use the search variable anywhere. Instead, you seem to be passing a literal string "search" to your function with this line
callNewsApi('search').then((output) => {
which does a search for the word "search", I guess.
You indicated that "it goes to the catch portion", which indicates that something went wrong in the call. You don't show any logging in the catch portion, and it may be useful to log the exception that is thrown, so you know why it is going to the catch portion. Something like
}).catch((xx) => {
console.error(xx);
res.json({ 'fulfillmentText': `I don't know the news but I hope it's good!` });
});
is normal, but since it looks like you're logging it in the .on('error') portion, showing that error might be useful.

The name of the intent and the variable I was using to make the call had a difference in Casing, I guess calls are case sensitive just be aware of that