Web-scraping images with JavaScript Cheerio - javascript

I receive an error when attempting to web-scrape images off amazon. The goal is to download the book cover.
(node:26904) UnhandledPromiseRejectionWarning: Error: undefined is not a valid uri or options object.
at request (C:\Users\Isaac\Desktop\webscrape_with_cheerio\node_modules\request\index.js:44:11)
at PromisseHandle._RejectOrResolve (C:\Users\Isaac\Desktop\webscrape_with_cheerio\node_modules\node-image-downloader\src\image-downloader.js:86:5)
at new Promise (<anonymous>)
at ImageDownloader (C:\Users\Isaac\Desktop\webscrape_with_cheerio\node_modules\node-image-downloader\src\image-downloader.js:98:26)
at Request._callback (C:\Users\Isaac\Desktop\webscrape_with_cheerio\index.js:22:13)
at Request.self.callback (C:\Users\Isaac\Desktop\webscrape_with_cheerio\node_modules\request\request.js:185:22)
at Request.emit (events.js:210:5)
at Request.<anonymous> (C:\Users\Isaac\Desktop\webscrape_with_cheerio\node_modules\request\request.js:1154:10)
at Request.emit (events.js:210:5)
at IncomingMessage.<anonymous> (C:\Users\Isaac\Desktop\webscrape_with_cheerio\node_modules\request\request.js:1076:12)
(node:26904) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:26904) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with
a non-zero exit code.
Here is my code thus far. I believe my error stems from the line;
var imagesrc = $(".a-row center-align litb-on-click img").attr('src')
I need to access the Class over where the book url is, though i might be wrong.
const express = require('express');
const cheerio = require('cheerio');
const download = require('node-image-downloader');
const request = require('request');
const app = express();
app.get('/', (req, res) => {
var url = "https://www.amazon.com/Black-Swan-Improbable-Robustness-Fragility/dp/081297381X"
// makeing a request
request(url,(error, response,html) => {
if (!error){
//console.log(html);
var $ = cheerio.load(html)
var imagesrc = $(".a-row center-align litb-on-click img").attr('src')
//download the image
download({
imgs: [
{
uri:imagesrc
}
],
dest:'./downloads'
})
.then((info) => {
console.log("Download Complete")
process.exit(1)
})
}
})
})
app.listen(5000)

Here, a-row center-align litb-on-click are classes
To select a class we need to use class-selector . (dot)
please try to update your imagesrc variable as below
var imagesrc = $(".a-row.center-align.litb-on-click img").attr('src');
Spaces are not compulsory.

Related

Axios Get request with Looker API

I am trying to create a lookerbot. I want my node app to reach looker but I keep getting this error:
node:internal/process/promises:279
triggerUncaughtException(err, true /* fromPromise */);
^
[UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "AxiosError: Request failed with status code 401".] {
code: 'ERR_UNHANDLED_REJECTION'
}
Node.js v17.9.0
error Command failed with exit code 1.
This is my code:
const axios = require('axios');
const qs = require('qs');
const SLACK_KEY = 'xoxb-1234';
const LOOKER_URL = 'https://xyz.abc.looker.com:19999/api/3.0/';
const LOOKER_CLIENT_ID = 'mnbv';
const LOOKER_CLIENT_SECRET = 'lkjh'
axios.get(LOOKER_URL, {
params: {
'clientId': LOOKER_CLIENT_ID,
'clientSecret': LOOKER_CLIENT_SECRET
}
})
.then(function (res) {
console.log(res);
})
I am not sure what my error is and how to make my node app reach my looker instance.

Firebase: Cloud function changing data in firestore - Permission denied on resource project [X:YYYYYYYYYY:web:KKKKKKKKKKKK]

I am trying to add data to my firestore collection via firebase cloud functions. However, when executing the function the following error appears.
My cloud function code is:
import * as functions from "firebase-functions";
import * as admin from "firebase-admin";
admin.initializeApp();
const Firestore = require("#google-cloud/firestore");
const PROJECTID = "[...]";
const firestore = new Firestore({
projectId: PROJECTID,
timestampsInSnapshots: true,
});
exports.createNewChat = functions.region("europe-west1")
.https.onRequest((request, response) => {
console.log("!!!!!!");
console.log("THe BodY: " + request);
console.log("THe BodY: " + request.rawBody);
console.log("THe BodY: " + request.body);
try {
const groupName = request.body.groupName;
const members: string[] = request.body.members;
firestore.collection("chats").add({
groupName: groupName,
members: members,
lastMessage: new Date(),
messages: [],
}).then((doc: any) => {
const chatId = doc.id;
members.forEach((member) => {
firestore.collection("myChats/" + member).add({
chatKey: chatId,
});
});
return response.status(200).send(doc);
});
} catch (e) {
functions.logger.log("catch clause " + e);
response.status(500).send(e);
}
});
My postman request looks like this:
Header: Content-Type -> application/json
Body (raw, json):
{
"groupName": "someGroupName",
"members": [
"123321aklslasl"
]
}
The exception which is thrown is:
! Google API requested!
URL: "https://oauth2.googleapis.com/token"
Be careful, this may be a production service.
(node:23224) UnhandledPromiseRejectionWarning: Error: 7 PERMISSION_DENIED: Permission denied on resource project [...].
at Object.callErrorFromStatus (E:\Workspaces\cloudFunctions\functions\node_modules#grpc\grpc-js\build\src\call.js:31:26)
at Object.onReceiveStatus (E:\Workspaces\cloudFunctions\functions\node_modules#grpc\grpc-js\build\src\client.js:179:52)
at Object.onReceiveStatus (E:\Workspaces\cloudFunctions\functions\node_modules#grpc\grpc-js\build\src\client-interceptors.js:336:141)
at Object.onReceiveStatus (E:\Workspaces\cloudFunctions\functions\node_modules#grpc\grpc-js\build\src\client-interceptors.js:299:181)
at E:\Workspaces\cloudFunctions\functions\node_modules#grpc\grpc-js\build\src\call-stream.js:145:78
at processTicksAndRejections (internal/process/task_queues.js:79:11)
Caused by: Error
at WriteBatch.commit (E:\Workspaces\cloudFunctions\functions\node_modules#google-cloud\firestore\build\src\write-batch.js:414:23)
at DocumentReference.create (E:\Workspaces\cloudFunctions\functions\node_modules#google-cloud\firestore\build\src\reference.js:291:14)
at CollectionReference.add (E:\Workspaces\cloudFunctions\functions\node_modules#google-cloud\firestore\build\src\reference.js:1967:28)
at E:\Workspaces\cloudFunctions\functions\lib\index.js:39:39
at C:\Users\XXXXX\AppData\Roaming\npm\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:560:16
at runFunction (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:533:15)
at runHTTPS (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:559:11)
at handler (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:479:23)
at Layer.handle [as handle_request] (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\firebase-tools\node_modules\express\lib\router\layer.js:95:5)
at next (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\firebase-tools\node_modules\express\lib\router\route.js:137:13)
(node:23224) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). T
o terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 3)
(node:23224) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
In a Cloud Function for Firebase you should use the Admin SDK for Node.js and not the standard JS SDK.
In addition, you need to use Promise.all() to execute in parallel the calls to the asynchronous add() method in the forEach() loop.
So the following set of adaptations should solve your problem:
import * as functions from "firebase-functions";
import * as admin from "firebase-admin";
admin.initializeApp();
exports.createNewChat = functions.region("europe-west1")
.https.onRequest((request, response) => {
const groupName = request.body.groupName;
const members: string[] = request.body.members;
const db = admin.firestore();
db.collection("chats").add({
groupName: groupName,
members: members,
lastMessage: new Date(),
messages: [],
}).then((doc: any) => {
const chatId = doc.id;
const promises = [];
members.forEach((member) => {
promises.push(db.collection("myChats/" + member).add({
chatKey: chatId,
}));
});
return Promise.all(promises);
})
.then(() => {
response.status(200).send(JSON.stringify(doc));
})
.catch(e => {
functions.logger.log("catch clause " + e);
response.status(500).send(JSON.stringify(e));
});
});

MongoDB: How do I delete nested object by index?

so I want to make a delete reply system but I don't know how to do it. Can you guy help me?
This is my table
image of my mongodb
and the code I've tried is but it doesn't work
app.post("/remove-reply", async (req, res) => {
const index = parseInt(req.body.index) - 1
await Posts.findOneAndUpdate({ _id: req.body.id }, [
{
$set: {
"Comments.$[].replies": {
$concatArrays: [
{ $slice: ["$Comments.$[].replies", index] },
{ $slice: ["$Comments.$[].replies", { $add: [1, index] }, { $size: "$Comments.$[].replies" }] }
]
}
}
}
])
res.redirect("/")
})
and it gives me this error
(node:17376) UnhandledPromiseRejectionWarning: MongoServerError: Invalid $set :: caused by :: FieldPath field names may not start with '$'.
at MessageStream.messageHandler (C:\Users\The.Peerapon\Desktop\programming\Express-app\node_modules\mongodb\lib\cmap\connection.js:467:30)
at MessageStream.emit (events.js:400:28)
at processIncomingData (C:\Users\The.Peerapon\Desktop\programming\Express-app\node_modules\mongodb\lib\cmap\message_stream.js:108:16)
at MessageStream._write (C:\Users\The.Peerapon\Desktop\programming\Express-app\node_modules\mongodb\lib\cmap\message_stream.js:28:9)
at writeOrBuffer (internal/streams/writable.js:358:12)
at MessageStream.Writable.write (internal/streams/writable.js:303:10)
at TLSSocket.ondata (internal/streams/readable.js:731:22)
at TLSSocket.emit (events.js:400:28)
at addChunk (internal/streams/readable.js:293:12)
at readableAddChunk (internal/streams/readable.js:267:9)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:17376) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:17376) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
my goal is to click delete button at reply to delete it by index.
for each comment\reply, save its index in the database and use it as a sort of id. then, when someone wants to delete a comment, you do the following:
app
.post('/remove-reply', async (req, res) => {
const postId = req.body.id;
const commentIndex = parseInt(req.body.commentIndex) - 1;
const replyId = req.body.replyIndex;
await Posts.findOneAndUpdate({
_id: postId
}, {
$pull: {
[`Comments.${ commentIndex }.replies`]: {
_id: replyId
}
}
});
res.redirect('/');
});
so, when a user clicks on a reply to delete, you should have all 3 fields you need for finding the exact reply you want to delete.
You can also just save an auto-generated unique id for each item to use as an identifier\index since we treat it as a collection more than an array in this use-case. It will probably be easier to maintain since after deleting a comment, the index is not aligned anymore with the underlying array.

Trying to susbcribe to PairCreated events from UniswapV2 Factory in Nodejs

I have previously used this code to get events from PancakeSwapV2 factory on the Binance Smart Chain. I'd like now to use this code to get events from UniswapV2 factory on the Ethereum blockchain but I get the following error :
(node:3544) UnhandledPromiseRejectionWarning: Error: resolver or addr is not configured for ENS name (argument="name", value="0x5C69bEe701ef814a2B6a3EDD4B1652CB9cc5aA6f ", code=INVALID_ARGUMENT, version=contracts/5.4.0)
at Logger.makeError (C:\Users\aaaa\WebstormProjects\web3test\node_modules\#ethersproject\logger\lib\index.js:187:21)
at Logger.throwError (C:\Users\aaaa\WebstormProjects\web3test\node_modules\#ethersproject\logger\lib\index.js:196:20)
at Logger.throwArgumentError (C:\Users\aaaa\WebstormProjects\web3test\node_modules\#ethersproject\logger\lib\index.js:199:21)
at C:\Users\aaaa\WebstormProjects\web3test\node_modules\#ethersproject\contracts\lib\index.js:101:32
at step (C:\Users\aaaa\WebstormProjects\web3test\node_modules\#ethersproject\contracts\lib\index.js:48:23)
at Object.next (C:\Users\aaaa\WebstormProjects\web3test\node_modules\#ethersproject\contracts\lib\index.js:29:53)
at fulfilled (C:\Users\aaaa\WebstormProjects\web3test\node_modules\#ethersproject\contracts\lib\index.js:20:58)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:3544) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To
terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:3544) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Process finished with exit code 0
Here's the source code I'm trying to reuse :
const Web3 = require('web3');
const ethers = require('ethers');
const INFURA_BASE_URL = 'https://mainnet.infura.io/v3/';
const INFURA_API_KEY = 'REPLACE';
web3 = new Web3(new Web3.providers.HttpProvider(INFURA_BASE_URL + INFURA_API_KEY));
const privateKey = "REPLACE";
const account = web3.eth.accounts.privateKeyToAccount(privateKey)
console.log(account.address)
const provider = new ethers.providers.JsonRpcProvider('https://mainnet.infura.io/v3/REPLACE');
const wallet = new ethers.Wallet(privateKey);
const account2 = wallet.connect(provider);
const addresses = {
WETH: '0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2',
factory: '0x5C69bEe701ef814a2B6a3EDD4B1652CB9cc5aA6f ',
router: '0x7a250d5630B4cF539739dF2C5dAcb4c659F2488D ',
recipient: account.address
};
//console.log(provider)
const factory = new ethers.Contract(
addresses.factory,
[
'event PairCreated(address indexed token0, address indexed token1, address pair, uint)',
'function getPair(address tokenA, address tokenB) external view returns (address pair)'
],
account2
);
factory.on('PairCreated', async (token0, token1, pairAddress) => { }
Would you know what I'm doing wrong ? Thank you.
It seems that I'm approaching the solution with this piece of code :
const Web3 = require("web3");
let web3 = new Web3(
new Web3.providers.WebsocketProvider("wss://mainnet.infura.io/ws/v3/cc44823998a0412294a47680xxxxxxxx")
);
let abi = JSON.parse('[{"inputs":[{"internalType":"address","name":"_feeToSetter","type":"address"}],"payable":false,"stateMutability":"nonpayable","type":"constructor"},{"anonymous":false,"inputs":[{"indexed":true,"internalType":"address","name":"token0","type":"address"},{"indexed":true,"internalType":"address","name":"token1","type":"address"},{"indexed":false,"internalType":"address","name":"pair","type":"address"},{"indexed":false,"internalType":"uint256","name":"","type":"uint256"}],"name":"PairCreated","type":"event"},{"constant":true,"inputs":[{"internalType":"uint256","name":"","type":"uint256"}],"name":"allPairs","outputs":[{"internalType":"address","name":"","type":"address"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"allPairsLength","outputs":[{"internalType":"uint256","name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"internalType":"address","name":"tokenA","type":"address"},{"internalType":"address","name":"tokenB","type":"address"}],"name":"createPair","outputs":[{"internalType":"address","name":"pair","type":"address"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"feeTo","outputs":[{"internalType":"address","name":"","type":"address"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"feeToSetter","outputs":[{"internalType":"address","name":"","type":"address"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"internalType":"address","name":"","type":"address"},{"internalType":"address","name":"","type":"address"}],"name":"getPair","outputs":[{"internalType":"address","name":"","type":"address"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"internalType":"address","name":"_feeTo","type":"address"}],"name":"setFeeTo","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"internalType":"address","name":"_feeToSetter","type":"address"}],"name":"setFeeToSetter","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"}]')
const instance = new web3.eth.Contract(abi, '0x5C69bEe701ef814a2B6a3EDD4B1652CB9cc5aA6f');
//web3.eth.getBlockNumber().then(console.log)
instance.getPastEvents(
"allEvents",
{fromBlock: "12908000", toBlock: "12908094"},
(errors, events) => {
if (!errors) {
//console.log('it is ok')
//console.log(events)
}
}
).then(r => {
console.log(r)
});

Discord.js embed image not working. (Could not interpret "{'url': 'https://cdn.nekos.life/boobs/boobs105.gif'}" as string.)

My command.js
based on some other bots like HarutoHiroki Bot thst open source on Github and the Nekos.life documentation
const Discord = require('discord.js');
const client = require('nekos.life');
const neko = new client();
module.exports.run = async (bot, message, args) => {
if (message.channel.nsfw === true) {
link = await neko.nsfw.boobs()
console.log(link)
const embed = new Discord.MessageEmbed()
.setAuthor(`Some neko boobs`)
.setColor('#00FFF3')
.setImage(link)
.setFooter(`Bot by`);
message.channel.send(embed);
}
else {
message.channel.send("This isn't NSFW channel!")
}
};
module.exports.config = {
name: "boobs",
description: "",
usage: "*boobs",
accessableby: "Members",
aliases: []
}
Error:
> node .
(node:25052) ExperimentalWarning: Conditional exports is an experimental feature. This feature could change at any time
Logged in and connected as Bot#1234
{ url: 'https://cdn.nekos.life/boobs/boobs105.gif' }
(node:25052) UnhandledPromiseRejectionWarning: DiscordAPIError: Invalid Form Body
embed.image.url: Could not interpret "{'url': 'https://cdn.nekos.life/boobs/boobs105.gif'}" as string.
at RequestHandler.execute (C:\Users\User\Documents\GitHub\Bot\node_modules\discord.js\src\rest\RequestHandler.js:170:25)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
(node:25052) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)(node:25052) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
My Question
how to fix this? (I tried the hole day but didn't got it working need help soon as possible)
I fixed it:
const Discord = require('discord.js');
const client = require('nekos.life');
const neko = new client();
module.exports.run = async (bot, message, args) => {
if (message.channel.nsfw === true) {
link = await neko.nsfw.boobs()
const embed = new Discord.MessageEmbed()
.setAuthor(`Some neko boobs`)
.setColor('#00FFF3')
.setImage(link.url)
.setFooter(`Bot by`);
message.channel.send(embed);
}
else {
message.channel.send("This isn't NSFW channel!")
}
};
module.exports.config = {
name: "boobs",
description: "",
usage: "*boobs",
accessableby: "Members",
aliases: []
}
It looks like neko.nsfw.boobs() returns an object, not a string. Try this instead:
const { url } = await neko.nsfw.boobs(); // get only the URL, not the whole object
const embed = new Discord.MessageEmbed()
.setAuthor(`Some neko boobs`)
.setColor('#00FFF3')
.setImage(link.url)
.setFooter(`Bot by`);
message.channel.send(embed);
Here's a code snippet example:
const link = {
url: 'https://cdn.nekos.life/boobs/boobs105.gif'
};
console.log(link); // this will not give the link. it will give the whole object
/* using object destructuring, you can use brackets to capture one
property of an array. in this case, the url.
Object Destructuring - https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Operators/Destructuring_assignment
*/
const {
url
} = link;
console.log(url); // this will give *only* the link
console.log(link.url); // another method to give only the link

Categories

Resources