My question is how retrieve data using many jsonp requests? it is practical? Currently, I'm using in my CRA this fragment of code (below)(pseudocode).
import * as fetch from 'fetch-jsonp';
import * as BlueBird from 'bluebird'; // bluebird is promise library
const getData = async () => {
const urls = ['https://...', 'https://...'] // containt about 20000 urls
const response = await BlueBird.map(urls, url => fetch(url), { concurrency: 10 })
return response
}
Working version of code creates script tags in my DOM, so as you can guess it prolongs DOM rendering, my laptop starts really heating and I end up getting error "render process gone" (on small data everything works). So what I should do? move my code to the server side and use json? or it's possible to create separate react dom and use it for jsonp? (cannot use json on client side cause cors)
Use jsoup official dependency if you are working for Android or library for java.
Then use
String html="here html";
Document doc=Jsoup.parse(html);
Related
I am new to nodejs/Express.js development.
I have built my backend service with Express.js / Typescript and I have multiple routes / api endpoints defined. One is like this:
app.post('/api/issues/new', createNewIssue);
where browser will send a post request when a user submits a new photo (also called an issue in my app).
The user can send an issue to another user, and the backend will first query the database to find the number of issues that matches the conditions of "source user" and "destination user", and then give the new issue an identifying ID in the form srcUser-dstUser-[number], where number is the auto-incremented count.
The createNewIssue function is like this:
export const createNewIssue = catchErrors(async (req, res) => {
const srcUser = req.header('src_username');
const dstUser = req.header('dst_username');
// query database for number of issues matching "srcUser" and "dstUser"
...
const lastIssues = await Issue.find( {where: {"srcUser": srcUser, "dstUser": dstUser}, order: { id: 'DESC'}});
const count = lastIssues.length;
// create a new issue Entity with the ID `srcUser-dstUser-[count+1]`
const newIssue = await createEntity(Issue, {
...
id: `srcUser-dstUser-${count+1}`,
...
});
res.respond({ newIssue: newIssue});
})
Say the backend receives multiple requests with the same srcUser and dstUser attributes at the same time, will there be collisions where multiple new issues are created with the same id?
I have read some documentation about nodejs being single-threaded, but I'm not sure what that means definitely for this specific scenario.
Besides business logic in this scenario, I have some confusions in general about Express JS / Node JS:
When there is only one cpu core, Express JS process multiple concurrent requests asynchronously: it starts processing one and does not wait for it to finish, instead continues to process the next one. Is this understanding accurate?
When there are multiple cpu cores, does Express JS / Node Js utilize them all in the same manner?
Node.js will not solve this problem for you automatically.
While it will only deal with one thing at a time, it is entirely possible that Request 2 will request the latest ID in the database while Request 1 has hit the await statement at the same point and gone to sleep. This would mean they get the same answer and would each try to create a new entry with the same ID.
You need to write your JavaScript to make sure that this doesn't happen.
The usual ways to handle this would be to either:
Let the database (and not your JavaScript) handle the ID generation (usually by using a sequence.
Use transactions so that the request for the latest ID and the insertion of the new row are treated as one operation by the database (so it won't start the same operation for Request 2 until the select and insert for Request 1 are both done).
Test to make sure createEntity is successful (and doesn't throw a 'duplicate id' error) and try again if it fails (with a limit in case it keeps failing in which case it should return an error message to the client).
The specifics depend on which database you use. I linked to the Postgresql documentation for the sake of example.
This code below works in PHP but not in JS. Can you tell me why? How can I make the pure JS version work?
async function cccrFlow() {
response = await fetch('assets/club_mems.json');
club_mems = [];
club_mems = await response.json(); // DOESN"T WORK
}
function cccrFlow() {
club_mems = <?php include('assets/club_mems.json');?>; // WORKS PERFECTLY
}
club_mems.json
[{"Name":"Ahmedistan, Jamshed","clubEXP":"2022-09-15"},{"Name":"Anaizi, Paul N","clubEXP":"2021-01-27"},{"Name":"Anderson, Simon","clubEXP":"2022-06-30"},{"Name":"Ashes, Bob P","clubEXP":"2022-04-21"}]
When you use fetch, you have to make sure that whatever asset/resource you are loading - is publicly accessible. In short,
if you use this URL in browser http://yourserver.com/assets/club_mems.json and you can see the response in the browser, then your fetch/async/await will work (see screenshot).
If you don't want to expose your data in public domain, then you will have to include the JSON file using one of the module loading methods i.e. import/require.
EDIT
To achieve this: you need to have some kind of module loader in your development environment. The module loader will make sure that once you build your application for prod environment, your code includes the JSON data without any issue.
There are different options: webpack, bundlr, parcel etc. Checkout their official websites for how to set them.
Once you set this up, you can simply write in your code:
const jsonData = require("/path/to/club_mems.json");
or
import jsonData from "/path/to/club_mems.json"
try this solution:
function cccrFlow() {
club_mems = fetch('assets/club_mems.json')
.then(response => response.json())
.then(data => {
console.log(data)
alert(data);
})
}
I want to get json data from multiple url's and display it on frontend.
Following are the url's:
1) localhost:3000/api/getdata1
2) localhost:3000/api/getdata2
3) localhost:3000/api/getdata3
Instead of using .fetch() on each of the url's like below:
.fetch('localhost:3000/api/getdata1')
.fetch('localhost:3000/api/getdata2')
.fetch('localhost:3000/api/getdata3')
Can this be done in more efficent way in ReactJs ?
I was trying:
const dataurls = [
'localhost:3000/api/getdata1',
'localhost:3000/api/getdata2',
'localhost:3000/api/getdata3'
];
const promisedurl = dataurls.map(httpGet);
Promise.all(promisedurls)
.then(data=> {
for (const d of data) {
console.log(d);
}
})
.catch(reason => {
// Receives first rejection among the Promises
});
Please suggest which one should be used or is there any efficient way to do get data from multiple url's.
ReactJS is a View layer library. It has nothing to do with how you aquire any data from server.
Even state libraries, like Redux and Reflux do not implement any method of fetching data. In most cases you do that in your custom app code. Sometimes using extra libraries (e.g. Redux middlewares).
So, yes: your Promise.all(<several https requests here>) is the most natural way to achieve that.
I have just started learning to code in ReactJS and have come to the part where I need to hit an API to get back JSON data.
I am running this on a node server on my local which tries to build it each time I make a change - this helps because it shows me where my errors are. Unfortunately it doesn't mean I know why I am getting an error.
This is my code so far:
import React from 'react';
import './App.css';
var GetUnassignedClients = React.createClass({
unassignedClients: function () {
return $.getJSON('http://localhost/backoffice/?action=unassignedClients.getUnassingedClients')
.then((data) => {
this.setState({resultMe: data.results});
});
},
render: function() {
return (
<div>this.state.resultMe</div>
);
}
});
module.exports = GetUnassignedClients;
I will re-paste the bit that seems to be throwing the error:
return $.getJSON('http://localhost/backoffice/?action=unassignedClients.getUnassingedClients')
.then((data) => {
this.setState({resultMe: data.results});
});
The error seems to be pointing at the $ - when I wrap this all in {} then it points to the full stop (.).
Not sure if I am doing something wrong or missing something.
The $in the $.getJSON is referring to jQuery. Do you have it included in your page? Javascript is trying to parse the dollar sign and can't find it. Other possible solution is to use a separate http library (if you don't need jQuery), such as axiois or fetch for example.
Im not sure, but I thought you would hit in this way
fetch(`http://localhost:8088/api/this/is/an/api/call`)
.then(result=> {
this.setState({items:result.json()});
});
Okay turns out I was trying to use $.getJSON which is JQuery - due to building this on my local using a node server JQuery wasn't included on my local development before the production code was built.
So instead I looked into using fetch that seems to be the go to when coding in react.
I also moved this section into a componentDidMount function so the app would be loaded and then the API call would be made - not that I have a lot of data to load but it's good practice for when you do.
My working code for this section:
componentDidMount () {
fetch("http://localhost/dashboard/?action=unassignedUsers.getUnassingedUsers", {
credentials: 'same-origin'
})
.then(response => response.json())
.then(json => this.setState({data: json}));
}
I also came across a couple issues using fetch - if you need a session available to access said data then you need to specifically pass through the credentials or the API call will re-direct to the login page. At first I saw a status 200 but no data - as it successfully re-directed to the login page, but obviously there was no JSON to access here.
I'm looking for a simple way to cache HTML that I pull using the request-promise library.
The way I've done this in the past is specify a time-to-live say one day. Then I take the parameters passed into request and I hash them. Then whenever a request is made I save the HTML contents on the file-system in a specific folder and name the file the name of the hash and the unix timestamp. Then when a request is made for the using the same parameters I check if the cache is still relevant via timestamp and pull it or make a new request.
Is there any library that can help with this that can wrap around request? Does request have a method of doing this natively?
I went with the recco in the comments and used Redis. Note this only works for get requests.
/* cached requests */
async function cacheRequest(options){
let stringOptions = JSON.stringify(options)
let optionsHashed = crypto.createHash('md5').update(stringOptions).digest('hex')
let get = await client.getAsync(optionsHashed)
if (get) return get
let HTML = await request.get(options)
await client.setAsync(optionsHashed, HTML)
return HTML
}