Develop Reference

GET request to webpage returns string, but cannot use it

I am creating a google chrome extension, and when I make a get request to this web page https://www.rightmove.co.uk/property-for-sale/find.html?locationIdentifier=REGION%5E27675&maxBedrooms=2&minBedrooms=2&sortType=6&propertyTypes=&mustHave=&dontShow=&furnishTypes=&keywords=, I get the webpage HTML as a response which is what I want (the website I am requesting info from does not have an API and I cannot web scrape for reasons too long to explain here). This response comes in the form of a string. When I attempt to split this string at a certain point, bis_skin_checked, I am returned an array of length 1, meaning that there was no match and nothing has been split. But when I look at the string returned it has it included.
I have tried things like removing spaces and carriage returns but nothing seems to be working. This is my GET request code:
function getNewPage(url) {
let returnedValue = fetch(url, {
method: 'GET',
headers: {
'Content-Type': 'text/html',
},
})
.then(response => response.text())
.then(text => {
return text
})
return returnedValue
}
I then go on to resolve the promise which is returnedValue:
let newURL = getHousePrices(currentUrl) // Get Promise of new page as a string
newURL.then(function(value) { // Resolve promise and do stuff
console.log(value.split('bis_skin_checked').length)
})
And then work with the string which looks like this: (I have attached an image as I cannot copy the text from the popup)
Image Link To API Request return

I'm assuming you want to get the home values, given certain search parameters.
Scraping raw text is usually not the way to go. I took a look at the site and saw it uses uniform network requests that you can modify to capture the data you need directly instead of scraping the raw HTML.
I built a solution that allows you to dynamically pass whatever parameters you want to the getHomes() function. Passing nothing will use the default params that you can use as a baseline while you try to adjust the request for any additional modifications/use cases.
Install the below solution and run getHomes() from the service worker.
Here's a brief video I made explaining the solution: https://vimeo.com/772275354/07fd1025ed
--- manifest.JSON ---
{
"name": "UK Housing - Stackoverflow",
"description": "Example for how to make network requests and mimic them in background.js to avoid web scraping raw text",
"version": "1.0.0",
"manifest_version": 3,
"background": {
"service_worker": "background.js"
},
"host_permissions": [
"*://*.rightmove.co.uk/*"
]
}
--- background.js ---
async function getHomes(passedParams) {
const newParams = passedParams ? passedParams : {}; // set to an empty object if no new params passed - avoid error in object.entries loop.
// starts with default params for a working request, you can then pass params to override the defaults to test new requests.
var params = {
"locationIdentifier": "REGION%5E27675",
"maxBedrooms": "2",
"minBedrooms": "1",
"numberOfPropertiesPerPage": "25",
"radius": "0.0",
"sortType": "6",
"index": "0",
"viewType": "LIST",
"channel": "BUY",
"areaSizeUnit": "sqft",
"currencyCode": "GBP",
"isFetching": "false"
}
Object.entries(params).forEach(([key, value]) => {
// we iterate through each key in our default params to check if we passed a new value for that key.
// we then set the params object to the new value if we did.
if (newParams[key]) params[key] = newParams[key];
});
const rightMoveAPISearch = `https://www.rightmove.co.uk/api/_search?
locationIdentifier=${params['locationIdentifier']}
&maxBedrooms=${params['maxBedrooms']}
&minBedrooms=${params['minBedrooms']}
&numberOfPropertiesPerPage=${params['numberOfPropertiesPerPage']}
&radius=${params['radius']}
&sortType=${params['sortType']}
&index=${params['index']}
&viewType=${params['viewType']}
&channel=${params['channel']}
&areaSizeUnit=${params['areaSizeUnit']}
&currencyCode=${params['currencyCode']}
&isFetching=${params['isFetching']}
`.replace(/\s/g, ''); // removes the whitespaces we added to make it look nicer / easier to adjust.
const data = await
fetch(rightMoveAPISearch, {
"method": "GET",
})
.then(data => data.json())
.then(res => { return res })
if (data.resultCount) {
console.log('\x1b[32m%s\x1b[0m', 'Request successful! Result count: ', parseInt(data.resultCount));
console.log('All data: ', data);
console.log('Properties: ', data.properties);
}
else console.log('\x1b[31m%s\x1b[0m', `Issue with the request:`, data)
return data
}
Hope this is helpful. Let me know if you need anything else.

How to insert new child data in Firebase web app?

I am building a simple car database app. When a user adds a new car it creates the below JSON data with blank fields for the gallery:
{
"name": "Cars",
"gallery": [{
"img": "http://",
"title": "BMW"
}, {
"img": "http://",
"title": "Lexus"
}, {
"img": "http://",
"title": "Ferrari"
}],
"pageId": "23",
"storeURL": "/app/cars/gallery"
}
Now what I want to do in the app is if a user adds a new gallery image it should add new child after the last gallery picture. So after Ferrari it should insert a new object. I cannot seem to do this as every tutorial leads to a dead end and also the official Google docs specify that set() will replace all data so if you use update() it will add and in this case it does not add.
This code deletes all data:
function add(){
var data = [
{
title: 'Mercedes',
img: 'http://'
}
]
var postData = {
pageGallery: data
};
var updates = {};
updates['/app/cars/'] = postData;
firebase.database().ref().update(updates);
console.log('New car added');
}
And if I change and add the child:
firebase.database().ref().child('gallery').update(updates);
It does not do anything to do the data. Please can you help?

See this blog post on the many reasons not use arrays in our Firebase Database. Read it now, I'll wait here...
Welcome back. Now that you know why your current approach will lead to a painful time, let's have a look at how you should model this data. The blog post already told you about push IDs, which are Firebase's massively scalable, offline-ready approach to array-like collections.
As our documentation on reading and writing lists of data explains, you can add items to a list by calling push().
So if this creates a car:
function addStore(){
var rootRef = firebase.database().ref();
var storesRef = rootRef.child('app/cars');
var newStoreRef = storesRef.push();
newStoreRef.set({
name: "Cars",
"pageId": "23",
"storeURL": "/app/cars/gallery"
});
}
Then to add an image to the gallery of that store:
var newCarRef = newStoreRef.child('gallery').push();
newCarRef.set({
title: 'Mercedes',
img: 'http://'
})
This will add a new car/image to the end of the gallery (creating the gallery if it doesn't exist yet).

If somebody wants it With Error Handling :
var rootRef = firebase.database().ref();
var storesRef = rootRef.child('Lorelle/Visitors');
var newStoreRef = storesRef.push();
newStoreRef.set($scope.CarModal,
function(error) {
//NOTE: this completion has a bug, I need to fix.
if (error) {
console.log("Data could not be saved." + error);
Materialize.toast('Error: Failed to Submit' + error, 2000);
} else {
console.log("Data saved successfully.");
Materialize.toast('Data Submitted, Thank You.', 2000);
}
});
}

Mongoose data wont save new instance of model

I have data coming in from a form, posting to /submit. My route looks like this:
var submission = require('../models/submission');
router.post('/submit', function(req,res){
var ObjectId = require('mongoose').Types.ObjectId;
var formObjId = new ObjectId(req.body.formId);
var data = new submission({
formId: formObjId,
fields: req.body.fields,
});
data.save(function (err) {
if (!err) {
//NO ERROR
} else {
return console.log(err);
}
});
res.end("Successful submission!");
});
and my submissions model:
var mongoose = require('mongoose');
module.exports = mongoose.model('Submission',{
createdAt: {type: Date, default: Date.now},
formId: [],
sourceId: [],
fields: [],
});
The first part is where I'm trying to cast the string into an object id. At the save, nothing shows up as an error, although the two fields that I'm trying to save don't save with the object. The object is saved in the database with a createdAt and object id attribute. The others are blank.
Am I doing something wrong to keep the attributes from saving with the rest of the object?

if i understand your code in right way , i found following things
Your using mongoose driver this is fine.
In the schema the data type of formId is Array .But in the response method the datatype which is assigning to this field is ObjectId.
Please check once is the fields field is exists in request body .i.e Do you sending any field which is named as 'fields' in the post request.
These are things i found when i was first look in to the code .
I hope these may helps to you .

Expected response to contain an array but got an object

So I'm new in Angular, and I've looked over various other solutions, but no one seemed to work for me. My app needs to take some data from mongodb database and show it to the client. The thing is I get
Error: [$resource:badcfg] Error in resource configuration for action query. Expected response to contain an array but got an object
Here is my SchoolCtrl.js on the client
app.controller('SchoolsCtrl', function($scope, SchoolResource) {
$scope.schools = SchoolResource.query();
});
Here is my ngResource
app.factory('SchoolResource', function($resource) {
var SchoolResource = $resource('/api/schools/:id', {id: '#id'}, { update: {method: 'PUT', isArray: false}});
return SchoolResource;
});
This is my SchoolsController on the server
var School = require('mongoose').model('School');
module.exports.getAllSchools = function(req, res, next) {
School.find({}).exec(function(err, collection) {
if(err) {
console.log('Schools could not be loaded: ' + err);
}
res.send(collection);
})
};
I tried adding IsArray: true, tried adding [] after 'SchoolResource' in the resource, tried changing routes, nothing worked. I wanted to see what actually is returned, that the query complained is not array, so I turned it to string and this was the result:
function Resource(value) { shallowClearAndCopy(value || {}, this); }
I have no idea why it returns a function. Can anyone help me?

That error message usually means your server is returning JSON representing a single object, such as:
{"some": "object", "with": "properties"}
when Angular is expecting JSON representing an array, like:
[ {"some": "object", "with": "properties"}, {"another": "object", "with": "stuff"} ]
Even if there is only a single result, query expects JSON for an array:
[ {"a": "single", "result": "object"} ]
You can verify this by simply loading your API call into the browser and checking it out. If there aren't square brackets around the whole JSON response, it isn't an array.

It happened to me too, but then I printed the object in the console and found out that there was something like this:
{ options:{....},results:[ ...the Array I was looking for... ]}
so all you need to do there is
res.send(collection.results);
I hope this helps

Looking for design pattern to create multiple models/collections out of single JSON responses

I have a Backbone application where the JSON I get from the server isn't exactly 1 on 1 with how I want my models to look. I use custom parse functions for my models, ex:
parse: function(response) {
var content = {};
content.id = response.mediaId;
content.image = response.image.url;
return content;
}
This works. But, in some cases I have an API call where I get lots of information at once, for instance, information about an image with its user and comments:
{
"mediaId": "1",
"image": {
"title": "myImage",
"url": "http://image.com/234.jpg"
},
"user": {
"username": "John"
},
"comments": [
{
"title": "Nice pic!"
},
{
"title": "Great stuff."
}
]
}
How would I go about creating a new User model and a Comments collection from here? This is an option:
parse: function(response) {
var content = {};
content.id = response.mediaId;
content.image = response.image.url;
content.user = new User(response.user);
content.comments = new Comments(response.comments);
return content;
}
The trouble here is, by creating a new User or new Comments with raw JSON as input, Backbone will just add the JSON properties as attributes. Instead, I'd like to have an intermediate parse-like method to gain control over the objects' structure. The following is an option:
parse: function(response) {
// ...
content.user = new User({
username: response.user.username
});
// ...
}
...but that's not very DRY-proof.
So, my question is: what would be a nice pattern to create several models/collections out of 1 JSON response, with control over the models/collections attributes?
Thanks!

It may not be the nicest way possible, but this is how I do it:
content.user = new User(User.prototype.parse(response.user));
The only problem is that the this context in User.parse will be wrong. If you don't have any specific code in the User constructor, you can also do:
content.user = new User();
content.user.set(user.parse(response.user));
I also noticed an interesting note in the Backbone version 0.9.9 change log:
The parse function is now always run if defined, for both collections and models — not only after an Ajax call.
And looking at the source code of Model and Collection constructor, they do it like so:
if (options && options.parse) attrs = this.parse(attrs);
Maybe upgrading to 0.9.9 will give you what you need? If upgrade is not an option, you can of course implement the same in your own constructor.