I'm using fetch to make API calls and everything works but in this particular instance I'm running into an issue because the API simply returns a string -- not an object.
Typically, the API returns an object and I can parse the JSON object and get what I want but in this case, I'm having trouble finding the text I'm getting from the API in the response object.
Here's what the response object looks like.
I thought I'd find the text inside the body but I can't seem to find it. Where do I look?
Using the fetch JavaScript API you can try:
response.text().then(function (text) {
// do something with the text response
});
Also take a look at the docs on fetch > response > body interface methods
ES6 Syntax:
fetch("URL")
.then(response => response.text())
.then((response) => {
console.log(response)
})
.catch(err => console.log(err))
You can do this in two different ways:
The first option is to use the response.text() method, but be aware that, at Dec/2019, its global usage is only 36.71%:
async function fetchTest() {
let response = await fetch('https://httpbin.org/encoding/utf8');
let responseText = await response.text();
document.getElementById('result').innerHTML = responseText;
}
(async() => {
await fetchTest();
})();
<div id="result"></div>
The second option is to use the response.body property instead, which requires a little more work but has 73.94% of global usage:
async function fetchTest() {
let response = await fetch('https://httpbin.org/encoding/utf8');
let responseText = await getTextFromStream(response.body);
document.getElementById('result').innerHTML = responseText;
}
async function getTextFromStream(readableStream) {
let reader = readableStream.getReader();
let utf8Decoder = new TextDecoder();
let nextChunk;
let resultStr = '';
while (!(nextChunk = await reader.read()).done) {
let partialData = nextChunk.value;
resultStr += utf8Decoder.decode(partialData);
}
return resultStr;
}
(async() => {
await fetchTest();
})();
<div id="result"></div>
Related
This is probably dead simple, but I can't quite figure it out. I simply want to read the contexts of a text file into a variable. What I have is:
async function load(path) {
try {
const response = await fetch(path);
const text = await response.text();
return text;
} catch (err) {
console.error(err);
}
}
var source_text = load(source_text_path);
console.log(source_text);
To my mind, this should work but only the pending promise is returned and not the text, thought I thought it was awaiting properly.
You need to wait for the load method.
var source_text = await load(source_text_path);
OR
load(source_text_path).then(e => console.log(e))
The function is indeed awaiting as it should, but since it's async, it will always result in a Promise<things> when you return thing.
Which means you should either await it elsewhere:
var source_text = await load(source_text_path);
or then it:
load(source_text_path).then((source_text) => {
console.log(source_text);
})
How may I get information from a ReadableStream object?
I am using the Fetch API and I don't see this to be clear from the documentation.
The body is being returned as a ReadableStream and I would simply like to access a property within this stream. Under Response in the browser dev tools, I appear to have this information organised into properties, in the form of a JavaScript object.
fetch('http://192.168.5.6:2000/api/car', obj)
.then((res) => {
if(!res.ok) {
console.log("Failure:" + res.statusText);
throw new Error('HTTP ' + res.status);
} else {
console.log("Success :" + res.statusText);
return res.body // what gives?
}
})
In order to access the data from a ReadableStream you need to call one of the conversion methods (docs available here).
As an example:
fetch('https://jsonplaceholder.typicode.com/posts/1')
.then(function(response) {
// The response is a Response instance.
// You parse the data into a useable format using `.json()`
return response.json();
}).then(function(data) {
// `data` is the parsed version of the JSON returned from the above endpoint.
console.log(data); // { "userId": 1, "id": 1, "title": "...", "body": "..." }
});
EDIT: If your data return type is not JSON or you don't want JSON then use text()
As an example:
fetch('https://jsonplaceholder.typicode.com/posts/1')
.then(function(response) {
return response.text();
}).then(function(data) {
console.log(data); // this will be a string
});
Some people may find an async example useful:
var response = await fetch("https://httpbin.org/ip");
var body = await response.json(); // .json() is asynchronous and therefore must be awaited
json() converts the response's body from a ReadableStream to a json object.
The await statements must be wrapped in an async function, however you can run await statements directly in the console of Chrome (as of version 62).
response.json() returns a Promise. Try ...
res.json().then(body => console.log(body));
where response is the result of the fetch(...)
Little bit late to the party but had some problems with getting something useful out from a ReadableStream produced from a Odata $batch request using the Sharepoint Framework.
Had similar issues as OP, but the solution in my case was to use a different conversion method than .json(). In my case .text() worked like a charm. Some fiddling was however necessary to get some useful JSON from the textfile.
Note that you can only read a stream once, so in some cases, you may need to clone the response in order to repeatedly read it:
fetch('example.json')
.then(res=>res.clone().json())
.then( json => console.log(json))
fetch('url_that_returns_text')
.then(res=>res.clone().text())
.then( text => console.log(text))
If you just want the response as text and don't want to convert it into JSON, use https://developer.mozilla.org/en-US/docs/Web/API/Body/text and then then it to get the actual result of the promise:
fetch('city-market.md')
.then(function(response) {
response.text().then((s) => console.log(s));
});
or
fetch('city-market.md')
.then(function(response) {
return response.text();
})
.then(function(myText) {
console.log(myText);
});
You may have asked the wrong question to solve your problem, but here is an answer to your actual question. An inspiration may be the source code of the Node.js stream/consumers module.
res.body is a ReadableStream that emits chunks as Uint8Arrays. The following function will collect all the chunks in a single Uint8Array:
export async function streamToArrayBuffer(stream: ReadableStream<Uint8Array>): Promise<Uint8Array> {
let result = new Uint8Array(0);
const reader = stream.getReader();
while (true) { // eslint-disable-line no-constant-condition
const { done, value } = await reader.read();
if (done) {
break;
}
const newResult = new Uint8Array(result.length + value.length);
newResult.set(result);
newResult.set(value, result.length);
result = newResult;
}
return result;
}
You can then use TextDecoder to convert the array to a string. You can then parse this string using JSON.parse():
const buffer = await streamToArrayBuffer(res.body);
const text = new TextDecoder().decode(buffer);
const json = JSON.parse(text);
In the future when browsers support it, you will also be able to use TextDecoderStream to collect the stream content as a string directly:
export async function streamToText(stream: ReadableStream<Uint8Array>): Promise<string> {
let result = '';
const reader = stream.pipeThrough(new TextDecoderStream()).getReader();
while (true) { // eslint-disable-line no-constant-condition
const { done, value } = await reader.read();
if (done) {
break;
}
result += value;
}
return result;
}
For those who have a ReadableStream and want to get the text out of it, a short hack is to wrap it in a new Response (or Request) and then use the text method:
let text = await new Response(yourReadableStream).text();
I dislike the chaining thens. The second then does not have access to status. As stated before 'response.json()' returns a promise. Returning the then result of 'response.json()' in a acts similar to a second then. It has the added bonus of being in scope of the response.
return fetch(url, params).then(response => {
return response.json().then(body => {
if (response.status === 200) {
return body
} else {
throw body
}
})
})
I just had the same problem for over 12 hours before reading next, just in case this helps anyone. When using nextjs inside your _api page you will need to use JSON.stringify(whole-response) and then send it back to your page using res.send(JSON.stringify(whole-response)) and when it's received on the client side you need to translate it back into json format so that it's usable. This can be kinda figured out by reading their serialization section. Hope it helps.
The code is being fetched but it is not being fully displayed, as in some parts are missing
https://github.com/KushRohra/GFG_Codes/blob/master/School/Leap%20year.cpp
You can see the code here and the one being displayed below.
However full code is being displayed in the console. dont know whats the problem
async function call() {
let url = "https://api.github.com/repos/KushRohra/GFG_Codes/contents/School/Leap year.cpp";
let response = await fetch(url);
console.log(data);
s = atob(data.content);
document.getElementById("code").innerHTML = s.replace(/\n/g, '<br>').replace(/\s/g, ' ');
}
call();
<div id="code"></div>
Heres how your code should look like:
async function call() {
let url = "https://api.github.com/repos/KushRohra/GFG_Codes/contents/School/Leap year.cpp";
let response = await fetch(url);
let data = await response.json();
console.log(data);
let s = window.atob(data.content);
document.getElementById("code").innerHTML = s.replace(/\n/g, '<br>').replace(/\s/g, ' ');
}
call();
You didn't declare data variable.
you didn't asked for a response to fetch body(response data) as a JSON() nor TEXT()
for atob function you better explicitly say window.atob() instead of atob()
When I run this code I get undefined but it is clear that the ski and product_id are in the value form.
I want:
value="BTdtb4CBz3uSJ2qv"
value="adi-ss20-042"
but I get "undefined"
class TresBien {
async scrapeRaffleInfo() {
// scrape the form_key and sku values
const response = await axios(
"https://tres-bien.com/adidas-yeezy-boost-380-mist-fx9764-ss20"
);
console.log("response: ", response);
const html = await response.data;
const $ = cheerio.load(html);
const res = $('input[name="sku"]').val();
const ans = $('input[name="form_key"]').val();
console.log(res && ans);
}
}
const main = async () => {
const tb = new TresBien(
"https://tres-bien.com/adidas-yeezy-700-v3-alvah-h67799-ss20"
);
let checkoutSucc = await tb.scrapeRaffleInfo();
if (checkoutSucc) {
Logger.logEventSuccess("Raffle successfully entered");
}
};
main();
There are a few things wrong in your code:
as #Pointy mentioned, scrapeRaffleInfo() is returning undefined and you're trying to use it in checkoutSucc
Tresbien class doesn't offer any constructor, yet you're passing your url as a parameter to the constructor of that class.
You're not using any of axios library methods (like: get(), post(), put(), delete()). typically you need axios.get() but there in your code, you're just using axios()
I am trying to scrap wikipedia page to fetch list of airlines by first scrapping first page and then going to each individual page of airline to get the website url. I have divided the code in two functions. One to scrap main page and get a new url, and second function to scrap another page from the created url to get the website name from that page. I have used request-promise module for getting the html and then cheerio to parse the data.
export async function getAirlinesWebsites(req,res) {
let response = await request(options_mainpage);
console.log(`Data`);
let $ = cheerio.load(response);
console.log('Response got');
$('tr').each((i,e)=>{
let children = '';
console.log('inside function ', i);
if($(e).children('td').children('a').attr('class') !== 'new') {
children = $(e).children('td').children('a').attr('href');
let wiki_url = 'https://en.wikipedia.org' + children;
console.log(`wiki_url = ${wiki_url}`);
let airline_url = getAirlineUrl(wiki_url);
console.log(`airline_url = ${airline_url}`);
}
})
And then the getAirlineUrl() function will parse another page based on the provided url.
async function getAirlineUrl(url){
const wiki_child_options = {
url : url,
headers : headers
}
let child_response = await request(wiki_child_options);
let $ = cheerio.load(child_response);
let answer = $('.infobox.vcard').children('tbody').children('tr').children('td').children('span.url').text();
return answer;
})
However when I console log the answer variable in the parent function, I get a [object Promise] value instead of a String. How do I resolve this issue?
Async function return promise.In case of that,you need to use then to get resolved response or use await.
This should work if other part of your code is ok.
export async function getAirlinesWebsites(req, res) {
let response = await request(options_mainpage);
console.log(`Data`);
let $ = cheerio.load(response);
console.log("Response got");
$("tr").each(async (i, e) => {
let children = "";
console.log("inside function ", i);
if ($(e).children("td").children("a").attr("class") !== "new") {
children = $(e).children("td").children("a").attr("href");
let wiki_url = "https://en.wikipedia.org" + children;
console.log(`wiki_url = ${wiki_url}`);
let airline_url = await getAirlineUrl(wiki_url);
console.log(`airline_url = ${airline_url}`);
}
});
}
Since your getAirlineUrl function returns a promise, you need to await that promise. You can't have await nested inside of the .each callback because the callback is not an async function, and if it was it wouldn't work still. The best fix is the avoid using .each and just use a loop.
export async function getAirlinesWebsites(req,res) {
let response = await request(options_mainpage);
console.log(`Data`);
let $ = cheerio.load(response);
console.log('Response got');
for (const [i, e] of Array.from($('tr')).entries()) {
let children = '';
console.log('inside function ', i);
if($(e).children('td').children('a').attr('class') !== 'new') {
children = $(e).children('td').children('a').attr('href');
let wiki_url = 'https://en.wikipedia.org' + children;
console.log(`wiki_url = ${wiki_url}`);
let airline_url = await getAirlineUrl(wiki_url);
console.log(`airline_url = ${airline_url}`);
}
}
}