I'm trying to use the unsplash-js module in a NodeJS/express server. As the docs say: "This library depends on fetch to make requests to the Unsplash API. For environments that don't support fetch, you'll need to provide a polyfill".
I've tried something like that, which I am quite ashamed of showing, but for the sake of learning I will
const Unsplash = require('unsplash-js').default;
const fetch = require('node-fetch');
Unsplash.prototype.fetch = fetch;
const unsplash = new Unsplash({
applicationId: process.env.UNSPLASH_API_KEY,
});
And also this
Node.prototype.fetch = fetch;
But of course, nothing worked.
How do I do that?
You need to set fetch to global variable like below:
global.fetch = require('node-fetch')
Related
I have a Next.js app that uses a serverless function to fetch data from its host. The host URL is stored in a .env file and is set to http://localhost:3000/api/data for dev and https://productionurl.com/api/data for production.
However, for testing purposes, I have hosted the app on Vercel, and the testing URL will be randomly generated, such as https://randomvercelurl_abcd1234.com/api/data.
I can use ctx or req to get the host URL in the getServerSideProps method, as shown in the example code below:
export async function getServerSideProps({query, req}){
const API_URL = req.protocol + req.headers.host + "/api/data"
const res = await axios.get(API_URL)
...
}
The problem arises in the getStaticPaths method, where I need to fetch data to generate dynamic routes. I don't have any way to get the host URL in this method, as demonstrated in the following example code:
export async function getStaticPaths() {
const host = ???
const res = await fetch(`${host}/api/data`)
const posts = await res.json()
const paths = posts.map((post) => ({
params: { id: post.id },
}))
return { paths, fallback: false }
}
How can I access the host URL in the getStaticPaths method to fetch data?
You cannot get a ctx with host like in getServerSideProps because getStaticPaths runs at build time. So the URL from which you want to fetch data should be fully known, either in the code or through an environment variable.
getStaticPaths will only run during build in production, it will not be called during runtime. You can validate code written inside getStaticPaths is removed from the client-side bundle with this tool.
The solution is to add the randomly generated URL into your environment variable on Vercel, or wherever you host your app. You may also want to read when should I use getStaticPaths.
So I am trying to get a circulating supply number of a token and use the API to do a further calculation with it.
All I could find was people handling JSON outputs but this is just a plain output:
https://avascan.info/api/v1/supply?q=circulatingSupply
Anyone that can help me how I can fetch this amount and use it to further do calculations with?
You can use the fetch API, which have a text method on it's response, for example:
const BASE_URL = 'https://avascan.info/api/v1/supply?q=circulatingSupply';
async function getCirculatingSupply() {
const response = await fetch(BASE_URL);
const data = await response.text();
return Number(data);
}
getCirculatingSupply().then(console.log);
Keep in mind that maybe you will have problems with CORS, as I had testing this API, so maybe you will need to use a proxy server like cors-anywhere to bypass this problem.
class Fetch {
async getCurrent(input) {
const apiKey = "my_api_key";
// make request to url
const response = await fetch(
`https:/api.openweathermap.org/data/2.5/weather?q=${input}&appid=${apiKey}`
);
const data = await response.json();
console.log(data);
return data;
}
}
above is a snippet of code, can someone point me to the direction of the error?
I'm assuming you are running this code in Node.js, right?
If it's in the browser, you shouldn't have any issue since the fetch() API is implemented in the browser.
If you are running in Node.js you can use node-fetch which is a great implementation of fetch in Node.js.
Your code works fine. Problem can be in that how you call function.
You need to create an object, outside the class and call the function.
let f = new Fetch();
f.getCurrent("Kiev");
You can`t use variable name fetch because this name is reserved.
This is my first phone app. I am using Ionic for the cross-platform work which uses Angular as you know I'm sure. I have a separate program which scrapes a webpage using puppeteer and cheerio and creates an array of values from the web page. This works.
I'm not sure how I get the array in my web scraping program read by my ionic/angular program.
I have a basic ionic setup and am just trying a most basic activity of being able to see the array from the ionic/angular side but after trying to put it in several places I realized I really didnt know where to import the code to ionic/angular which returns the array or where to put the webscraper code directly in one of the .ts files or ???
This is my web scraping program:
const puppeteer = require('puppeteer'); // live webscraping
let scrape = async () => {
const browser = await puppeteer.launch({
headless: true
});
const page = await browser.newPage();
await page.goto('--page url here --'); // link to page
const result = await page.evaluate(() => {
let data = []; // Create an empty array that will store our data
let elements = document.querySelectorAll('.list-myinfo-block'); // Select all Products
let photo_elements = document.getElementsByTagName('img'); //
var photo_count = 0;
for (var element of elements) { // Loop through each product getting photos
let picture_link = photo_elements[photo_count].src;
let name = element.childNodes[1].innerText;
let itype = element.childNodes[9].innerText
data.push({
picture_link,
name,
itype
}); // Push an object with the data onto our array
photo_count = photo_count + 1;
}
return data;
});
browser.close();
return result; // Return the data
};
scrape().then((value) => {
console.log(value); // Success!
});
When I run the webscraping program I see the array with the correct values in it. Its getting it into the ionic part of it. Sometimes the ionic phone page will show up with nothing in it, sometimes it says it cannot find "/" ... I've tried so many different places and looked all over the web that I have quite a combination of errors. I know I'm putting it in the wrong places - or maybe not everywhere I should. Thank you!
You need a server which will run the scraper on demand.
Any scraper that uses a real browser (ie: chromium) will have to run in a OS that supports it. There is no other way.
Think about this,
Does your mobile support chromium and nodeJS? It does not. There are no chromium build for mobile which supports automation with nodeJS (yet).
Can you run a browser inside another browser? You cannot.
Way 1: Remote wsEndpoint
There are some services which offers wsEndpoint but I will not mention them here. I will describe how you can create your own wsEndPoint and use it.
Run browser and Get wsEndpoint
The following code will launch a puppeteer instance whenever you connect to it. You have to run it inside a server.
const http = require('http');
const httpProxy = require('http-proxy');
const proxy = new httpProxy.createProxyServer();
http
.createServer()
.on('upgrade', async(req, socket, head) => {
const browser = await puppeteer.launch();
const target = browser.wsEndpoint();
proxyy.ws(req, socket, head, { target })
})
.listen(8080);
When you run this on the server/terminal, you can use the ip of the server to connect. In my case it's ws://127.0.0.1:8080.
Use puppeteer-web
Now you will need to install puppeteer-web on your mobile/web app. To bundle Puppeteer using Browserify follow the instruction below.
Clone Puppeteer repository:
git clone https://github.com/GoogleChrome/puppeteer && cd puppeteer
npm install
npm run bundle
This will create ./utils/browser/puppeteer-web.js file that contains Puppeteer bundle.
You can use it later on in your web page to drive another browser instance through its WS Endpoint:
<script src='./puppeteer-web.js'></script>
<script>
const puppeteer = require('puppeteer');
const browser = await puppeteer.connect({
browserWSEndpoint: '<another-browser-ws-endpont>'
});
// ... drive automation ...
</script>
Way 2: Use an API
I will use express for a minimal setup. Consider your scrape function is exported to a file called scrape.js and you have the following index.js file.
const express = require('express')
const scrape= require('./scrape')
const app = express()
app.get('/', function (req, res) {
scrape().then(data=>res.send({data}))
})
app.listen(8080)
This will launch a express API on the port 8080.
Now if you run it with node index.js on a server, you can call it from any mobile/web app.
Helpful Resources
I had some fun with puppeteer and webpack,
playground-react-puppeteer
playground-electron-react-puppeteer-example
To keep the api running, you will need to learn a bit about backend and how to keep the server alive etc. See these links for full understanding of creating the server and more,
Official link to puppeteer-web
Puppeteer with docker
Docker with XVFB and Puppeteer
Puppeteer with chrome extension
Puppeteer with local wsEndpoint
Avoid memory leak on server
It's basically the same question here.
I'm using a browser. The following code is compiled by webpack.
I tried this:
const axios = require('axios');
var res = await axios.get('https://api.ipify.org?format=json', {
proxy: {
host: 'proxy-url',
port: 80,
auth: {username: 'my-user', password: 'my-password'}
}
});
console.log(res.data); // gives my ip and not the proxy's one.
I also tried this with the same code, but it did not work:
const axios = require('axios-https-proxy-fix');
Then, I tried with httpsAgent:
const axios = require('axios');
const HttpsProxyAgent = require('https-proxy-agent')
var agent = new HttpsProxyAgent('http://my-user:my-pass#proxy-url:port');
var res = await axios.get('https://api.ipify.org?format=json', {
httpsAgent: agent,
});
console.log(res.data); // gives my ip and not the proxy's one.
Is this a bug? Am I cursed or maybe do I have a problem reading the documentation?
if you want to use axios and work-around the issue then consider using https-proxy-agent for proxy agent as mentioned in link
const HttpsProxyAgent = require("https-proxy-agent"),
axios = require("axios");
const httpsAgent = new HttpsProxyAgent({host: "proxyhost", port: "proxyport", auth: "username:password"})
//use axios as you normally would, but specify httpsAgent in the config
axios = axios.create({httpsAgent});
There is an open issue on axios's github page.
The issue is labeled as bug from 31 Mar and is not resolved yet.
So it seems you are not cursed, just a bug in axios.
You may add your details to that thread in order to dev team prioritize this issue.
In case you cannot wait for this issue to be resolved, you may consider using fetch API like #Sumi Straessle proposed in the comments.
axios's config.proxy is Node.js only. I think it is meaningless in browsers.
You can check lib/adapters/xhr.js and will find nothing related to proxy there.
If you are trying to access an HTTPS URL by an HTTP proxy, you need to create an HTTPS-HTTP tunnel.
I found the solution for this issue in this post: https://janmolak.com/node-js-axios-behind-corporate-proxies-8b17a6f31f9d. Now it works perfectly!