I am crawling 5 different sites for data using node-fetch and cheerio.
everything checks out but I need to collect the returned data from these 5 separate functions in an array.
First I store function name and url for each site in an object like so
url: 'https://sampleurl.com',
crawl: FirstLinkCrawl
}
const secondLink = {
url: 'https://sampleurl.com',
crawl: secondLinkCrawl
}
}```
Then I write the function to crawl each site like so, I ran this function with and without promise, both check out
```const secondLinkCrawl = (body) =>{
return new Promise((res, rej)=>{
"this crawl function is ran here, everything works fine, Data is an object"
const error = false
if(!error){
res(Data)
}else{
rej()
}
})
}```
This here is my fetch function that takes the url and a callback, which is the crawl function
```async function Fetch(url, callback){
const response = await fetch(url)
const html = await response.text()
callback(html)
}
Then I call the fetch and crawl using promise.all() in an attempt to have it return in an array, but in an array,
const promises = [
Fetch(firstLink.url, firstLink.crawl),
Fetch(secondLink.url, secondLink.crawl)]
Promise.all(promises)
.then(values => console.log(values))
.catch(err => console.log(err))
}
When I run this, I get [ undefined, undefined ]
But when I run it without promises and simply log the result, both of them run successfully.
My aim is to get my results in a single array. what can I do?
I also tried declaring an array a the top of the page and then pushing each result into the array, after which I log the array at the bottom of the functions call. But the array returns empty
You're not returning anything from Fetch function that's why it's undefined. You can fix it by -
async function Fetch(url, callback){
const response = await fetch(url)
const html = await response.text()
const result = await callback(html);
return result;
}
As the callback, you are passing in Fetch function returns Promise so we can await it and return the result
I'm using The Movie Database API. And the problem that i can't solve is returning "keys" variable when i call the function with Movie's id.I'm new on JavaScript that's why i can't solve this. Hope someone can help me, thanks in advance.
const APIURL = "https://api.themoviedb.org/3/discover/movie?sort_by=popularity.desc&api_key=[MY API KEY HERE]&page-1";
getMovies(APIURL)
async function getMovies(url)
{
const resp = await fetch(url);
const respData = await resp.json();
showMovies(respData.results)
}
async function getTrailer(id)
{
const resp = await fetch(`https://api.themoviedb.org/3/movie/${id}/videos?api_key=[MY API KEY HERE]&language=en-US`);
const respDataa = await resp.json();
let results = respDataa.results;
let keys = results[0].key;
return keys;
}
function showMovies(movies){
movies.forEach(movie => {
const modals = document.createElement('div');
modals.classList.add('modal');
modals.innerHTML = ` <a target="_blank" href ="https://www.youtube.com/watch?v=${getTrailer(movie.id)}">Watch Trailer</a>`
}
}
Well, first of all, always hide your api keys if you're posting your code somewhere (even if it's a private repository, you shouldn't do it).
Secondly, if you want to return multiple keys, you can map the results to extract id from each of them and do a return:
async function getTrailer(id)
{
const resp = await fetch(`https://api.themoviedb.org/3/movie/${id}/videos?api_key=04c35731a5ee918f014970082a0088b1&language=en-US`);
const respDataa = await resp.json();
let results = respDataa.results;
return results.map(({ key }) => key);
}
Async functions return a Promise in JavaScript.
Simply add return keys at the end of your function.
Then you can do:
getTrailer(528085).then(data => {
// do something with the data
})
You can also handle errors:
getTrailer(528085)
.then(data => {
// do something with the data
})
.catch(err => {
/*
An error occured. You can use err.toString() to convert the error into a string
*/
})
If you want to get the returned data immediately from an async function(or a promise), put your logic inside an async function, then you can simply do:
let keys = await getTrailer(528085)
And, here is how to handle errors in async/await:
try {
let keys = await getTrailer(528085)
}
catch(err){
/*
An error occured. You can use err.toString() to convert the error into a string
*/
}
By the way, like Desiigner said, don't keep your API keys in the client. Anyone can see your API key. Use a server to return the API response to the client.
We have to await or.then the return (it’s a Promise).
function showMovies(movies) {
// make the forEach callback async
movies.forEach(async (movie) => {
console.log(getTrailer(movie.id)) // Promise {<pending>}
const trailer = await getTrailer(movie.id);
console.log({ trailer });
const modals = document.createElement("div");
modals.classList.add("modal");
modals.innerHTML = ` <a target="_blank" href ="https://www.youtube.com/watch?v=${trailer}">Watch Trailer</a>`;
});
}
So I know this question is asked a lot, but I'm trying to retrieve a variable that is created within a promise. The examples I've seen on here involve calling .then and using the data there, however what I'm trying to do involves an async function--which i cant use within the .then block.
Here's my code. I'm using the Asana API To call out a lists of tasks that are due. It successfuly logs it. But I want to save the list value from the last block as a variable that I can use elsewhere.
const asana = require('asana');
const client = asana.Client.create().useAccessToken("xxx");
client.users.me()
.then(user => {
const userId = user.id;
// The user's "default" workspace is the first one in the list, though
// any user can have multiple workspaces so you can't always assume this
// is the one you want to work with.
const workspaceId = user.workspaces[0].id;
return client.tasks.findAll({
assignee: userId,
workspace: workspaceId,
completed_since: 'now',
opt_fields: 'id,name,assignee_status,completed'
});
})
.then(response => {
// There may be more pages of data, we could stream or return a promise
// to request those here - for now, let's just return the first page
// of items.
return response.data;
})
.filter(task => {
return task.assignee_status === 'today' ||
task.assignee_status === 'new';
})
.then(list => {
console.log (util.inspect(list, {
colors: true,
depth: null
}));
})
.catch(e => {
console.log(e);
});
If you're open to rewriting your .then()'s as async/await something like this could work for you:
const fetch = require('node-fetch');
async function doit() {
const response = await fetch('https://jsonplaceholder.typicode.com/todos/1');
const json = await response.json();
console.log(json);
}
doit();
This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 3 months ago.
I am trying to use a Node.JS application to make and receive API requests. It does a get request to another server using Axios with data it receives from an API call it receives. The second snippet is when the script returns the data from the call in. It will actually take it and write to the console, but it won't send it back in the second API.
function axiosTest() {
axios.get(url)
.then(function (response) {
console.log(response.data);
// I need this data here ^^
return response.data;
})
.catch(function (error) {
console.log(error);
});
}
...
axiosTestResult = axiosTest();
response.json({message: "Request received!", data: axiosTestResult});
I'm aware this is wrong, I'm just trying to find a way to make it work. The only way I can seem to get data out of it is through console.log, which isn't helpful in my situation.
The issue is that the original axiosTest() function isn't returning the promise. Here's an extended explanation for clarity:
function axiosTest() {
// create a promise for the axios request
const promise = axios.get(url)
// using .then, create a new promise which extracts the data
const dataPromise = promise.then((response) => response.data)
// return it
return dataPromise
}
// now we can use that data from the outside!
axiosTest()
.then(data => {
response.json({ message: 'Request received!', data })
})
.catch(err => console.log(err))
The function can be written more succinctly:
function axiosTest() {
return axios.get(url).then(response => response.data)
}
Or with async/await:
async function axiosTest() {
const response = await axios.get(url)
return response.data
}
Guide on using promises
Info on async functions
I know this post is old. But i have seen several attempts of guys trying to answer using async and await but getting it wrong. This should clear it up for any new references
UPDATE: May 2022
This answer is still having lots of interest and have updated it to use arrow functions
const axiosTest = async () {
try {
const {data:response} = await axios.get(url) //use data destructuring to get data from the promise object
return response
}
catch (error) {
console.log(error);
}
}
you can populate the data you want with a simple callback function,
let's say we have a list named lst that we want to populate,
we have a function that pupulates pupulates list,
const lst = [];
const populateData = (data) => {lst.push(data)}
now we can pass the callback function to the function which is making the axios call and we can pupulate the list when we get data from response.
now we make our function that makes the request and pass populateData as a callback function.
function axiosTest (populateData) {
axios.get(url)
.then(function(response){
populateData(response.data);
})
.catch(function(error){
console.log(error);
});
}
The axios library creates a Promise() object. Promise is a built-in object in JavaScript ES6. When this object is instantiated using the new keyword, it takes a function as an argument. This single function in turn takes two arguments, each of which are also functions — resolve and reject.
Promises execute the client side code and, due to cool Javascript asynchronous flow, could eventually resolve one or two things, that resolution (generally considered to be a semantically equivalent to a Promise's success), or that rejection (widely considered to be an erroneous resolution). For instance, we can hold a reference to some Promise object which comprises a function that will eventually return a response object (that would be contained in the Promise object). So one way we could use such a promise is wait for the promise to resolve to some kind of response.
You might raise we don't want to be waiting seconds or so for our API to return a call! We want our UI to be able to do things while waiting for the API response. Failing that we would have a very slow user interface. So how do we handle this problem?
Well a Promise is asynchronous. In a standard implementation of engines responsible for executing Javascript code (such as Node, or the common browser) it will resolve in another process while we don't know in advance what the result of the promise will be. A usual strategy is to then send our functions (i.e. a React setState function for a class) to the promise, resolved depending on some kind of condition (dependent on our choice of library). This will result in our local Javascript objects being updated based on promise resolution. So instead of getters and setters (in traditional OOP) you can think of functions that you might send to your asynchronous methods.
I'll use Fetch in this example so you can try to understand what's going on in the promise and see if you can replicate my ideas within your axios code. Fetch is basically similar to axios without the innate JSON conversion, and has a different flow for resolving promises (which you should refer to the axios documentation to learn).
GetCache.js
const base_endpoint = BaseEndpoint + "cache/";
// Default function is going to take a selection, date, and a callback to execute.
// We're going to call the base endpoint and selection string passed to the original function.
// This will make our endpoint.
export default (selection, date, callback) => {
fetch(base_endpoint + selection + "/" + date)
// If the response is not within a 500 (according to Fetch docs) our promise object
// will _eventually_ resolve to a response.
.then(res => {
// Lets check the status of the response to make sure it's good.
if (res.status >= 400 && res.status < 600) {
throw new Error("Bad response");
}
// Let's also check the headers to make sure that the server "reckons" its serving
//up json
if (!res.headers.get("content-type").includes("application/json")) {
throw new TypeError("Response not JSON");
}
return res.json();
})
// Fulfilling these conditions lets return the data. But how do we get it out of the promise?
.then(data => {
// Using the function we passed to our original function silly! Since we've error
// handled above, we're ready to pass the response data as a callback.
callback(data);
})
// Fetch's promise will throw an error by default if the webserver returns a 500
// response (as notified by the response code in the HTTP header).
.catch(err => console.error(err));
};
Now we've written our GetCache method, lets see what it looks like to update a React component's state as an example...
Some React Component.jsx
// Make sure you import GetCache from GetCache.js!
resolveData() {
const { mySelection, date } = this.state; // We could also use props or pass to the function to acquire our selection and date.
const setData = data => {
this.setState({
data: data,
loading: false
// We could set loading to true and display a wee spinner
// while waiting for our response data,
// or rely on the local state of data being null.
});
};
GetCache("mySelelection", date, setData);
}
Ultimately, you don't "return" data as such, I mean you can but it's more idiomatic to change your way of thinking... Now we are sending data to asynchronous methods.
Happy Coding!
axiosTest() needs to return axios.get, which in turn returns a Promise.
From there, then can be used to execute a function when said Promise resolves.
See Promise for more info.
Alternatively, await can be used from within the scope of some async function.
// Dummy Url.
const url = 'https://jsonplaceholder.typicode.com/posts/1'
// Axios Test.
const axiosTest = axios.get
// Axios Test Data.
axiosTest(url).then(function(axiosTestResult) {
console.log('response.JSON:', {
message: 'Request received',
data: axiosTestResult.data
})
})
<script src="https://cdnjs.cloudflare.com/ajax/libs/axios/0.18.0/axios.js"></script>
IMO extremely important rule of thumb for your client side js code is to keep separated the data handling and ui building logic into different funcs, which is also valid for axios data fetching ... in this way your control flow and error handlings will be much more simple and easier to manage, as it could be seen from this
ok fetch
and this
NOK fetch
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
<script>
function getUrlParams (){
var url_params = new URLSearchParams();
if( window.location.toString().indexOf("?") != -1) {
var href_part = window.location.search.split('?')[1]
href_part.replace(/([^=&]+)=([^&]*)/g,
function(m, key, value) {
var attr = decodeURIComponent(key)
var val = decodeURIComponent(value)
url_params.append(attr,val);
});
}
// for(var pair of url_params.entries()) { consolas.log(pair[0]+ '->'+ pair[1]); }
return url_params ;
}
function getServerData (url, urlParams ){
if ( typeof url_params == "undefined" ) { urlParams = getUrlParams() }
return axios.get(url , { params: urlParams } )
.then(response => {
return response ;
})
.catch(function(error) {
console.error ( error )
return error.response;
})
}
// Action !!!
getServerData(url , url_params)
.then( response => {
if ( response.status === 204 ) {
var warningMsg = response.statusText
console.warn ( warningMsg )
return
} else if ( response.status === 404 || response.status === 400) {
var errorMsg = response.statusText // + ": " + response.data.msg // this is my api
console.error( errorMsg )
return ;
} else {
var data = response.data
var dataType = (typeof data)
if ( dataType === 'undefined' ) {
var msg = 'unexpected error occurred while fetching data !!!'
// pass here to the ui change method the msg aka
// showMyMsg ( msg , "error")
} else {
var items = data.dat // obs this is my api aka "dat" attribute - that is whatever happens to be your json key to get the data from
// call here the ui building method
// BuildList ( items )
}
return
}
})
</script>
After 6 hours of fluttering, I realized it was a one-line problem. If you are interfering with the axios life-cycle, you may have forgotten this line:
componentDidMount() {
this.requestInterceptor = axios.interceptors.request.use((request) => {
this.updateApiCallFor(request.url, true);
return request;
});
this.responseInterceptor = axios.interceptors.response.use((response) => {
this.updateApiCallFor(response.config.url, false);
return response; // THIS LINE IS IMPORTANT !
}, (error) => {
this.updateApiCallFor(error.config.url, false);
throw error;
});
async makes a function return a Promise
await makes a function wait for a Promise
code async/await
// https://www.npmjs.com/package/axios
const axios = require('axios')
/* --- */
async function axiosTest() {
let promiseAxios = axios.get( 'https://example.com' )
/* --- */
console.log( await promiseAxios )
}
/* --- */
axiosTest()
replit.com Stackoverflow - Returning data from Axios API
replit.com Stackoverflow - How to return values from async
code async/await with return
// https://www.npmjs.com/package/axios
const axios = require('axios')
/* --- */
async function axiosTest() {
console.log( await promiseAxios() )
}
/* --- */
axiosTest()
/* --- */
// create function for promise axios and return it
function promiseAxios() {
return axios.get( 'https://example.com' )
}
replit.com Stackoverflow - Returning data from Axios API - return
replit.com Stackoverflow - How to return values from async - return
Try this,
function axiosTest() {
axios.get(url)
.then(response => response.data)
.catch(error => error);
}
async function getResponse () {
const response = await axiosTest();
console.log(response);
}
getResponse()
It works, but each function where you want to get the response needs to be an async function or use an additional .then() callback.
function axiosTest() {
axios.get(url)
.then(response => response.data)
.catch(error => error);
}
async function getResponse () {
axiosTest().then(response => {
console.log(response)
});
}
getResponse()
If anyone knows a way to avoid this please do tell.
Also checkout Katsiaryna (Kate) Lupachova's article on Dev.to. I think it will help.
async handleResponse(){
const result = await this.axiosTest();
}
async axiosTest () {
return await axios.get(url)
.then(function (response) {
console.log(response.data);
return response.data;})
.catch(function (error) {
console.log(error);
});
}
You can find check https://flaviocopes.com/axios/#post-requests url and find some relevant information in the GET section of this post.
You can use Async - Await:
async function axiosTest() {
const response = await axios.get(url);
const data = await response.json();
}
Maybe a trivial one, but I am new with Typescript and fetch API.
In an exported class I have a public method remoteFetchSomething like:
export class className {
remoteFetchSomething = (url : string) : Promise<Response> => {
return fetch(url)
.then(
(r) => r.json()
)
.catch((e) => {
console.log("API errore fetching " + objectType);
});
}
}
export const classInstance = new className();
The method queries a remote JSON API service, and in the code, I am using it like:
import { classInstance } from ...
classInstance.remoteFetchSomething('https://example.url')
.then((json) => {
console.log(json);
}
)
The console.log is actually showing the results, but the remoteFetchSomething returns a Promise and I am unable to parse and access the JSON object values.
I would like to wait for the response before executing the remaining code, but how do I unwrap content from promise? Should I again put another .then? What am I missing?
Thank you.
By now I resolved the problem defining the return type of the remoteFetch as any:
remoteFetchSomething = (url : string) : any => {
return fetch(url)
.then(
(r) => r.json()
)
.catch((e) => {
console.log("API errore fetching " + objectType);
});
}
And now I can access JSON values like data below:
classInstance.remoteFetchSomething('https://example.url').then(
(json) => {
console.dump(json.data);
}
)
[sincerely still not clear why I cant' use the Promise<Response> type]
You can't synchronously block while waiting for a request in javascript, it would lock up the user's interface!
In regular javascript, and most versions of TypeScript, you should be / must be returning a promise.
function doRequestNormal(): Promise<any> {
return fetch(...).then(...);
}
function someOtherMethodNormal() {
// do some code here
doRequestNormal.then(() => {
// continue your execution here after the request
});
}
In newer versions of typescript, there's async/await keyword support - so instead it might look like this:
async function doRequestAsync() {
var result = await fetch(...);
// do something with request;
return result;
}
async function someOtherMethodAsync() {
// do some code here
var json = await doRequestAsync();
// continue some code here
}
Keep in mind, doRequestAsync still returns a Promise under the hood - but when you call it, you can use await to pretend that you're blocking on it instead of needing to use the .then( callback. If you call an async method from a non-async method, you'll still need to use the callbacks as normal.
this is how I do it:
type Payload = {
id: number
}
type ReturnType = number
export const functionThatHasNumberType = async (
payload: Payload
): Promise<ReturnType> => {
let url = `/api/${payload.id}`
return await axios.get(url)
}