I am sending an Axios request to a web page for scraping a little string off of it, but the returned response is a large html, and I only need a little part of it, is there a way to somehow shorten the response so I can save data and make the request faster?
const longHtml = await axios.get('https://example.com');
const shortHtml = longHtml.data //get short data here
If I understand what your trying to do, you wan't to stop the request when you find the data you want, you could use htmlparser2 and feed it a stream from axios and then register listeners and when you get the element you need you can end the stream.
Web scraping is a technique used for retrieving data from websites. You fetch the page's contents, and after that extract the data you need from the page.
here an example by using axios and cheerio
const axios = require("axios")
const cheerio = require("cheerio")
async function fetchHTML(url) {
const { data } = await axios.get(url)
return cheerio.load(data)
}
const $ = await fetchHTML("https://example.com")
// Print the full HTML
console.log(`Site HTML: ${$.html()}\n\n`)
// Print some specific page content
console.log(`First h1 tag: ${$('h1').text()}`)
Related
I am sending a HTTP GET request from a browser to an external API using React. I'm getting a response containing some HTML and Javascript which I would like to render in my browser.
Here's my code so far:
const url = getExternalEndpoint()
fetch(url)
.then(function(response) {
return response.text();
}).then(function(data) {
console.log(data);
})
Based on Retrieve data from a ReadableStream object?
So I can see the HTML in the console, but I'm not sure how to render it.
For context, the external server I'm sending the request to is an OpenID Connect server.
So, there is one library called react-html-parser
To install, use the following command
npm install react-html-parser
# or
yarn add react-html-parser
Here, you can use the state to update the value from API.
import ReactHtmlParser from 'react-html-parser';
const [html_string, Sethtml_string] = useState('')
//set value in the html_string
<div> { ReactHtmlParser (html_string) } </div>
I am trying to get the latitude and longitude of user location using google api
I have tried doing this using the fetch method in javascript but I do not get any response. There's no error or anything.
The documentation says to fetch it this way:
https://www.googleapis.com/geolocation/v1/geolocate?key=MY_API_KEY
How I fetch it:
const res = await fetch(https://www.googleapis.com/geolocation/v1/geolocate?key=MY_API_KEY)
const response = await res.json()
When I first did it like above I got an error because it returned nothing which I found out after console.logging res.text()
I do not know what I'm doing wrong In the documentation they equally said something about including Request body but it's optional, So I have no need to include it. Please do you have any ideas on what I'm doing wrong?
I believe you need to send this as a POST request according to the documentation you linked. By default fetch sends a GET request.
To do that you just need to pass in the request type to the fetch function
const response = await fetch(<URI>,{method:'POST'});
const res = await response.json()
console.log(res)
So I am trying to get a circulating supply number of a token and use the API to do a further calculation with it.
All I could find was people handling JSON outputs but this is just a plain output:
https://avascan.info/api/v1/supply?q=circulatingSupply
Anyone that can help me how I can fetch this amount and use it to further do calculations with?
You can use the fetch API, which have a text method on it's response, for example:
const BASE_URL = 'https://avascan.info/api/v1/supply?q=circulatingSupply';
async function getCirculatingSupply() {
const response = await fetch(BASE_URL);
const data = await response.text();
return Number(data);
}
getCirculatingSupply().then(console.log);
Keep in mind that maybe you will have problems with CORS, as I had testing this API, so maybe you will need to use a proxy server like cors-anywhere to bypass this problem.
I've created an api server'ish environment on cloudflare using cloudflare-worker and there is no node server running(cloudflare-worker is pretty much serverless event handler service). It does provide the configurations to handle any subdomain calls much like how api works. I've used a package called cf-worker-router to do so.
My cloud service looks like this:
import { ApiError, ApiRedirect, DomainRouter, FetchRouter } from 'cf-worker-router';
const router = new FetchRouter();
// add the cloudflare event listener
addEventListener('fetch', (event) => {
router.onFetch(event);
});
router.route('/users', 'POST', async (event) => {
// automatically converts anything not of Response type to ApiResponse
return await event.request.text();
});
And what I did was create a POST request to the url and supplied some body to the request. I was able to get the request text successfully but now I can't figure out how to parse the text I received.
When using the request as multipart/form-data request and the received body text is as follows:
"----------------------------093590450792211419875705\r\nContent-Disposition: form-data; name=\"name\"\r\n\r\nJon Doe\r\n----------------------------093590450792211419875705--\r\n"
I tried sending application/x-www-form-urlencoded and I the response text as such:
"name=Jon%20Doe"
And Similar for application/json request:
"{\n\t\"name\": \"Jon Doe\"\n}"
Since cloudflare is not using nodejs server, body-parser can't be applied here. This service is pretty much an open api so it needs to take care of all sorts of request content types. Is there any way to identify and decode the strignified contents from any of these content types to a valid object in javascript?
To handle form data uploads, you can use the request.formData() method which will return a promise of a FormData object.
For example:
addEventListener('fetch', event => {
event.respondWith(handleRequest(event.request))
})
async function handleRequest(request) {
const formData = await request.formData();
const name = formData.get('name');
return new Response(`Hello ${name}`);
}
I need to do some DOM manipulation based on some AJAX call. But I end up with the res.send on my page and I am unable to get the console.log I need to see the datas and be able to check what I need to insert in my Dom. All I see is the res.render and the JSON datas.
Even by trying to do some basic DOM creation it didnt work.
I manage to do some AJAX call already. Some Axios post, patch or delete, but I never needed to call the data when rendering the page, always through a button inside the page.
There must be something I am not understanding...
Router.get("/collection", async (req, res) => {
const dbRes = await Promise.all([
sneakerModel.find().populate("tag"),
tagModel.find()
]);
const sneakRes = dbRes[0];
const tagRes = dbRes[1];
res.send(tagRes);
});
// ===============================
// CLIENT SIDE =>
const allCollecRoutes = document.getElementById("allCollec");
allCollecRoutes.onclick = async () => {
const dbRes = await axios.get("http://localhost:9876/collection");
console.log(dbRes);
};
You appear to be expecting a JSON serialised response at the client but are sending a plain/text from the server i.e. res.send(tagRes)
Use res.json(tagRes) instead.