Develop Reference

GET request to webpage returns string, but cannot use it

I am creating a google chrome extension, and when I make a get request to this web page https://www.rightmove.co.uk/property-for-sale/find.html?locationIdentifier=REGION%5E27675&maxBedrooms=2&minBedrooms=2&sortType=6&propertyTypes=&mustHave=&dontShow=&furnishTypes=&keywords=, I get the webpage HTML as a response which is what I want (the website I am requesting info from does not have an API and I cannot web scrape for reasons too long to explain here). This response comes in the form of a string. When I attempt to split this string at a certain point, bis_skin_checked, I am returned an array of length 1, meaning that there was no match and nothing has been split. But when I look at the string returned it has it included.
I have tried things like removing spaces and carriage returns but nothing seems to be working. This is my GET request code:
function getNewPage(url) {
let returnedValue = fetch(url, {
method: 'GET',
headers: {
'Content-Type': 'text/html',
},
})
.then(response => response.text())
.then(text => {
return text
})
return returnedValue
}
I then go on to resolve the promise which is returnedValue:
let newURL = getHousePrices(currentUrl) // Get Promise of new page as a string
newURL.then(function(value) { // Resolve promise and do stuff
console.log(value.split('bis_skin_checked').length)
})
And then work with the string which looks like this: (I have attached an image as I cannot copy the text from the popup)
Image Link To API Request return

I'm assuming you want to get the home values, given certain search parameters.
Scraping raw text is usually not the way to go. I took a look at the site and saw it uses uniform network requests that you can modify to capture the data you need directly instead of scraping the raw HTML.
I built a solution that allows you to dynamically pass whatever parameters you want to the getHomes() function. Passing nothing will use the default params that you can use as a baseline while you try to adjust the request for any additional modifications/use cases.
Install the below solution and run getHomes() from the service worker.
Here's a brief video I made explaining the solution: https://vimeo.com/772275354/07fd1025ed
--- manifest.JSON ---
{
"name": "UK Housing - Stackoverflow",
"description": "Example for how to make network requests and mimic them in background.js to avoid web scraping raw text",
"version": "1.0.0",
"manifest_version": 3,
"background": {
"service_worker": "background.js"
},
"host_permissions": [
"*://*.rightmove.co.uk/*"
]
}
--- background.js ---
async function getHomes(passedParams) {
const newParams = passedParams ? passedParams : {}; // set to an empty object if no new params passed - avoid error in object.entries loop.
// starts with default params for a working request, you can then pass params to override the defaults to test new requests.
var params = {
"locationIdentifier": "REGION%5E27675",
"maxBedrooms": "2",
"minBedrooms": "1",
"numberOfPropertiesPerPage": "25",
"radius": "0.0",
"sortType": "6",
"index": "0",
"viewType": "LIST",
"channel": "BUY",
"areaSizeUnit": "sqft",
"currencyCode": "GBP",
"isFetching": "false"
}
Object.entries(params).forEach(([key, value]) => {
// we iterate through each key in our default params to check if we passed a new value for that key.
// we then set the params object to the new value if we did.
if (newParams[key]) params[key] = newParams[key];
});
const rightMoveAPISearch = `https://www.rightmove.co.uk/api/_search?
locationIdentifier=${params['locationIdentifier']}
&maxBedrooms=${params['maxBedrooms']}
&minBedrooms=${params['minBedrooms']}
&numberOfPropertiesPerPage=${params['numberOfPropertiesPerPage']}
&radius=${params['radius']}
&sortType=${params['sortType']}
&index=${params['index']}
&viewType=${params['viewType']}
&channel=${params['channel']}
&areaSizeUnit=${params['areaSizeUnit']}
&currencyCode=${params['currencyCode']}
&isFetching=${params['isFetching']}
`.replace(/\s/g, ''); // removes the whitespaces we added to make it look nicer / easier to adjust.
const data = await
fetch(rightMoveAPISearch, {
"method": "GET",
})
.then(data => data.json())
.then(res => { return res })
if (data.resultCount) {
console.log('\x1b[32m%s\x1b[0m', 'Request successful! Result count: ', parseInt(data.resultCount));
console.log('All data: ', data);
console.log('Properties: ', data.properties);
}
else console.log('\x1b[31m%s\x1b[0m', `Issue with the request:`, data)
return data
}
Hope this is helpful. Let me know if you need anything else.

Http PUT deletes data that hasn't been changed

I am trying to edit a JSON data base of shifts. I am writing in Javascript using react.
This is my understanding of the PUT syntax:
const editShift = async (changed, id) => {
const res = await fetch(`http://localhost:5000/shifts/${id}`, {
method: 'PUT',
headers: {
'Content-type': 'application/json'
},
body: JSON.stringify(changed)
})
const data = await res.json()
setShifts([...shifts, data])
}
data.json:
{
"shifts": [
{
"title": "test",
"startDate": "2018-06-25T07:30:00.000Z",
"endDate": "2018-06-25T08:00:00.000Z",
"allDay": false,
"id": 1
},
{
"title": "test2",
"startDate": "2018-06-28T07:30:00.000Z",
"endDate": "2018-06-28T08:00:00.000Z",
"allDay": false,
"id": 2
}
]
}
The result is that the new shift will hold only the fields that have been changed and delete the rest. Any ideas why?

To get previous shifts you should use a callback inside setShifts like:
setShifts(prevShifts => ([...prevShifts, data]))

The PUT request should work like that.
A PUT request should PUT the data into place of the old record.
You would want to use a PATCH request for only PATCHing data upon the old record.
That set aside, it depends on your /shifts endpoint on how this is actually handled.

I had to send the whole updated shift into the changed field, with the fields that were not changed. I don't know why but it works so okay.

For a question like this, it'd be helpful if you posted before and after examples of data
database empty
request #1, payload equals X
request #2, payload equals Y
Also, knowing what your Server handler is doing would help determine what's going on. I have a feeling what's happening is something like this
// API handler
(req) => {
const dbData = getDBData();
dbData.shifts = req.shifts; // overwrite old with new
}
When what you need is something like
(req) => {
const dbData = getDBData();
req.shifts.forEach((newShiftData) => {
const shiftId = newShiftData.id;
if (dbData[shiftId]) {
dbData[shiftId] = { ...dbData[shiftId], ...newShiftData }; // merge old with new
}
});
// - check if data changed
// - update DB
}

PUT requests replace all the data at the target URI. They don't automatically merge changes.
It's not clear if you are the author of this API or just a consumer, but if you are consuming the API you should either look for a different API that lets you only send edits (HTTP method might be POST or PATCH), or you need to send the entire list of shifts.
Alternatively it might also be possible that there's a specific endpoint for each shift, so you can send exactly 1 PUT request to a shift endpoint just to edit that specific endpoint.

Sending "CC" and "BCC" in substitution data when calling Node SparkPost API

I have templates created in SparkPost dashboard. But the problem I face is that I am not able to send "CC" or "BCC" by making the api calls. The code snippet below will help you understand what I am trying to do.
var SPARKPOST_KEY = "KEY"
var sparkpost = require('sparkpost');
var sparkclient = new sparkpost(SPARKPOST_KEY);
var req_opts = {
transmissionBody : {
content: {
template_id: 'order-confirmation',
from: 'support#domain.in',
subject: 'Order confirmation',
headers: {
"CC": "<anon2#gmail.com>"
}
},
substitution_data: {
"CC": "anon2#gmail.com",
"customer": "Aravind",
"order": 123532
},
recipients: [
{address: {email:'anon1#domain1.in'}},
{address: {email: 'anon2#gmail.com'}}
],
"return_path": "support#domain.in",
}
};
sparkclient.transmissions.send(req_opts, function(err, res){
if(err){
console.log("ERROR");
console.log(err)
}else {
console.log(res.body);
console.log("Mail has been successfully sent");
}
});

As mentioned in the reply on your github issue, you must use either inline content or a template. So as the documentation says, use just template_id in your content.
What needs to happen for this to work is that the headers in the template include a CC header, as described here. Currently there is no way to set the headers of a template in the UI -- it must be done using the API.
To do this execute a PUT against the templates endpoint, in your case https://api.sparkpost.com/api/v1/templates/order-confirmation, with a JSON payload containing the following:
{
"content": {
<other content parts>
"headers": {
"CC": "{{CC}}"
}
}
}
Note that you will also need to use the header_to parameter for your CC recipient to prevent their address showing up in the To: header. In your example this means replacing:
{address: {email: 'anon2#gmail.com'}}
with this:
{address: {email: 'anon2#gmail.com', header_to: 'anon1#domain1.in'}}
You also do not need the return_path parameter.
Hope this helps!

Creating an envelope from a template returning "UNSPECIFIED_ERROR"

When I try to create an envelope from a template I get a response of:
{ errorCode: 'UNSPECIFIED_ERROR',
message: 'Non-static method requires a target.' }
Here's what I'm doing so far:
First I login, which returns
{ loginAccounts:
[ { name: '*****',
accountId: '*****',
baseUrl: 'https://demo.docusign.net/restapi/v2/accounts/******',
isDefault: 'true',
userName: '***** ********',
userId: '*******-*****-*****-*****-*********',
email: '********#*******.com',
siteDescription: '' } ] }
So then I take the baseUrl out of that response and I attempt to create the envelope. I'm using the hapi framework and async.waterfall of the async library, so for anyone unfamiliar with either of these my use of the async library uses the next callback to call the next function which in this case would be to get the url for the iframe, and with our usage of the hapi framework AppServer.Wreck is roughy equivalent to request:
function prepareEnvelope(baseUrl, next) {
var createEntitlementTemplateId = "99C44F50-2C97-4074-896B-2454969CAEF7";
var getEnvelopeUrl = baseUrl + "/envelopes";
var options = {
headers: {
"X-DocuSign-Authentication": JSON.stringify(authHeader),
"Content-Type": "application/json",
"Accept": "application/json",
"Content-Disposition": "form-data"
},
body : JSON.stringify({
status: "sent",
emailSubject: "Test email subject",
emailBlurb: "My email blurb",
templateId: createEntitlementTemplateId,
templateRoles: [
{
email: "anemailaddress#gmail.com",
name: "Recipient Name",
roleName: "Signer1",
clientUserId: "1099", // TODO: replace with the user's id
tabs : {
textTabs : [
{
tabLabel : "acct_nmbr",
value : "123456"
},
{
tabLabel : "hm_phn_nmbr",
value : "8005882300"
},
{
tabLabel : "nm",
value : "Mr Foo Bar"
}
]
}
}
]
})
};
console.log("--------> options: ", options); // REMOVE THIS ====
AppServer.Wreck.post(getEnvelopeUrl, options, function(err, res, body) {
console.log("Request Envelope Result: \r\n", JSON.parse(body));
next(null, body, baseUrl);
});
}
And what I get back is:
{ errorCode: 'UNSPECIFIED_ERROR',
message: 'Non-static method requires a target.' }
From a little googling it look like 'Non-static method requires a target.' is a C# error and doesn't really give me much indication of what part of my configuration object is wrong.
I've tried a simpler version of this call stripping out all of the tabs and clientUserId and I get the same response.
I created my template on the Docusign website and I haven't ruled out that something is set up incorrectly there. I created a template, confirmed that Docusign noticed the named form fields, and created a 'placeholder' templateRole.
Here's the templateRole placeholder:
Here's one of the named fields that I want to populate and corresponding data label:
As a side note, I was able to get the basic vanilla example working without named fields nor using a template using the docusign node package just fine but I didn't see any way to use tabs with named form fields with the library and decided that I'd rather have more fine-grained control over what I'm doing anyway and so I opted for just hitting the APIs.
Surprisingly when I search SO for the errorCode and message I'm getting I could only find one post without a resolution :/
Of course any help will be greatly appreciated. Please don't hesitate to let me know if you need any additional information.

Once I received feedback from Docusign that my api call had an empty body it didn't take but a couple minutes for me to realize that the issue was my options object containing a body property rather than a payload property, as is done in the hapi framework.

twitter typeahead 0.9.3 remote return URL and json object

in bootstrap 2, I used the following code to post a json object,
$('#typeahead').typeahead({
source: function (query, process) {
var URL = 'http://localhost:8080/autocomplete/search/';
var query = {"t:jsonStringField": {
"name": "model",
"value": "fusion"
},
"t:jsonStringFilter": [
{"name": "year","value": "2009"},
{"name": "make","value": "ford"}
]
};
return $.getJSON(URL,
{ query: JSON.stringify(query)},
function (data) {
return process(data);
});
}
});
Now in twitter tyeahead 0.9.3 they have done away with the source concept and moved to a remote concept, but unfortunately I do no know how to work with it.
$(".typeahead").typeahead({
remote : {
url: 'http://localhost:8080/autocomplete/search/',
replace: function(uri, query) {
var query = {"t:jsonStringField": {
"name": "model",
"value": "fusion"
},
"t:jsonStringFilter": [
{"name": "year","value": "2009"},
{"name": "make","value": "ford"}
]
};
return uri + JSON.stringify(query);
},
filter: function(response) {
return response.matches;
}
return process(resultList);
}
}
Unfortunately it doesn't work, how do I just post the JSON object rather than appending it to the string? Thanks.

In your original code you use $.getJSON. This will send a request (and expects json as result) to: http://localhost:8080/autocomplete/search/?query=%7B%22t%3AjsonStringField%22%3A%7B%22name%22%3A%22model%22%2C%22value%22%3A%22fusion%22%7D%2C%22t%3AjsonStringFilter%22%3A%5B%7B%22name%22%3A%22year%22%2C%22value%22%3A%222009%22%7D%2C%7B%22name%22%3A%22make%22%2C%22value%22%3A%22ford%22%7D%5D%7D
To do the same for Twitter's Typeahead call your replace function of your remote data should return a valid url. In your function the ?query= part of the url is missing.
You will have to set: url: 'http://localhost:8080/autocomplete/search/?query=',
You also will have to urlencode you json input maybe.
Note: you will not need the line return process(resultList); You will have to use the filter function to convert your json results to valid data:
The individual units that compose datasets are called datums. The
canonical form of a datum is an object with a value property and a
tokens property.
you could use templates to style your dropdown results, see: http://twitter.github.io/typeahead.js/examples/
By default, the dropdown menu created by typeahead.js is going to look
ugly and you'll want to style it to ensure it fits into the theme of
your web page.
You will need the additional CSS from https://github.com/jharding/typeahead.js-bootstrap.css to style the default dropdown for Bootstrap.