I am just trying to build a crawler with chrome-remote-interface but i don't know how to get multiple dom elements like specific targets id,classes.
for Ex:
price = document.getelementbyid('price')
name= document.getelementbyid('name')
Code
const CDP = require('chrome-remote-interface');
CDP((client) => {
// Extract used DevTools domains.
const {Page, Runtime} = client;
// Enable events on domains we are interested in.
Promise.all([
Page.enable()
]).then(() => {
return Page.navigate({url: 'http://example.com'})
});
// Evaluate outerHTML after page has loaded.
Page.loadEventFired(() => {
Runtime.evaluate({expression: 'document.body.outerHTML'}).then((result) => {
//How to get Multiple Dom elements
console.log(result.result.value);
client.close();
});
});
}).on('error', (err) => {
console.error('Cannot connect to browser:', err);
});
Update
const CDP = require('chrome-remote-interface');
CDP((client) => {
// Extract used DevTools domains.
const {DOM,Page, Runtime} = client;
// Enable events on domains we are interested in.
Promise.all([
Page.enable()
]).then(() => {
return Page.navigate({url: 'https://someDomain.com'});
})
Page.loadEventFired(() => {
const expression = `({
test: document.getElementsByClassName('rows')),
})`
Runtime.evaluate({expression,returnByValue: true}).then((result) => {
console.log(result.result) // Error
client.close()
})
})
}).on('error', (err) => {
console.error('Cannot connect to browser:', err);
});
Error
{ type: 'object',
subtype: 'error',
className: 'SyntaxError',
description: 'SyntaxError: Unexpected token )',
objectId: '{"injectedScriptId":14,"id":1}' }
Actually I want to iterate over the list of elements But I don't know where it goes wrong
You cannot move DOM object from the browser context to the Node.js context, all you can do is pass a property or whatever can be considered a JSON object. Here I'm assuming you're interested in the computed HTML.
A possible solution is:
const CDP = require('chrome-remote-interface');
CDP((client) => {
// Extract used DevTools domains.
const {Page, Runtime} = client;
// Enable events on domains we are interested in.
Promise.all([
Page.enable()
]).then(() => {
return Page.navigate({url: 'http://example.com'});
});
// Evaluate outerHTML after page has loaded.
Page.loadEventFired(() => {
const expression = `({
name: document.getElementById('name').outerHTML,
price: document.getElementById('price').outerHTML
})`;
Runtime.evaluate({
expression,
returnByValue: true
}).then(({result}) => {
const {name, price} = result.value;
console.log(`name: ${name}`);
console.log(`price: ${price}`);
client.close();
});
});
}).on('error', (err) => {
console.error('Cannot connect to browser:', err);
});
The key point is returning a JSON object using returnByValue: true.
Update: You have an error in your expression, a trailing ) in ...('rows')),. But even if you fix it you'd still end up in a wrong situation because you're attempting to pass an array of DOM objects (see the first paragraph of this answer). Again, if you want just the outer HTML you can do something like:
// Evaluate outerHTML after page has loaded.
Page.loadEventFired(() => {
const expression = `
// fetch an array-like of DOM elements
var elements = document.getElementsByTagName('p');
// create and return an array containing
// just a property (in this case `outerHTML`)
Array.prototype.map.call(elements, x => x.outerHTML);
`;
Runtime.evaluate({
expression,
returnByValue: true
}).then(({result}) => {
// this is the returned array
const elements = result.value;
elements.forEach((html) => {
console.log(`- ${html}`);
});
client.close();
});
});
Related
I am developing the backend of an application using Node JS, Sequelize and Postgres database.
When the course is registered, the user must inform which organizations, companies and teachers will be linked to it.
The organization IDs are passed through an array to the backend, I am trying to do a check to make sure that the passed IDs exist.
What I've done so far is this:
const { organizations } = req.body;
const organizationsArray = organizations.map(async (organization) => {
const organizationExists = await Organization.findByPk(organization);
if (!organizationExists) {
return res
.status(400)
.json({ error: `Organization ${organization} does not exists!` });
}
return {
course_id: id,
organization_id: organization,
};
});
await CoursesOrganizations.bulkCreate(organizationsArray);
This link has the complete controller code, I believe it will facilitate understanding.
When !OrganizationExists is true, I am getting the return that the organization does not exist. The problem is when the organization exists, I am getting the following message error.
The Array.map() is returning an array of promises that you can resolve to an array using Promise.all(). Inside the map you should use throw new Error() to break out of the map - this error will be raised by Promise.all() and you can then catch it and return an error to the client (or swallow it, etc).
This is a corrected version of your pattern, resolving the Promise results.
const { organizations } = req.body;
try {
// use Promise.all to resolve the promises returned by the async callback function
const organizationsArray = await Promise.all(
// this will return an array of promises
organizations.map(async (organization) => {
const organizationExists = await Organization.findByPk(organization, {
attributes: ['id'], // we only need the ID
raw: true, // don't need Instances
});
if (!organizationExists) {
// don't send response inside the map, throw an Error to break out
throw new Error(`Organization ${organization} does not exists!`);
}
// it does exist so return/resolve the value for the promise
return {
course_id: id,
organization_id: organization,
};
})
);
// if we get here there were no errors, create the records
await CoursesOrganizations.bulkCreate(organizationsArray);
// return a success to the client
return res.json({ success: true });
} catch (err) {
// there was an error, return it to the client
return res.status(400).json({ error: err.message });
}
This is a refactored version that will be a bit faster by fetching all the Organizations in one query and then doing the checks/creating the Course inserts.
const { Op } = Sequelize;
const { organizations } = req.body;
try {
// get all Organization matches for the IDs
const organizationsArray = await Organization.findAll({
attributes: ['id'], // we only need the ID
where: {
id: {
[Op.in]: organizations, // WHERE id IN (organizations)
}
},
raw: true, // no need to create Instances
});
// create an array of the IDs we found
const foundIds = organizationsArray.map((org) => org.id);
// check to see if any of the IDs are missing from the results
if (foundIds.length !== organizations.length) {
// Use Array.reduce() to figure out which IDs are missing from the results
const missingIds = organizations.reduce((missingIds, orgId) => {
if (!foundIds.includes(orgId)){
missingIds.push(orgId);
}
return missingIds;
}, []); // initialized to empty array
throw new Error(`Unable to find Organization for: ${missingIds.join(', ')}`);
}
// now create an array of courses to create using the foundIds
const courses = foundIds.map((orgId) => {
return {
course_id: id,
organization_id: orgId,
};
});
// if we get here there were no errors, create the records
await CoursesOrganizations.bulkCreate(courses);
// return a success to the client
return res.json({ success: true });
} catch (err) {
// there was an error, return it to the client
return res.status(400).json({ error: err.message });
}
If you have an array of Ids and you want to check if they exist you should you use the (in) operator, this makes it so that you are hitting the DB only once and getting all the records in one hit (instead of getting them one by one in a loop), after you get these records you can check their lengths to determine if they all exist or not.
const { Op } = require("sequelize");
let foundOrgs = await Organization.findAll({
where: {
id: {
[Op.in]: organizationsArray,
}
}
});
I have the following code in one of my endpoints:
let setText = ''
for (const [ key, value ] of Object.entries(req.body.info)) {
setText = setText.concat(`${key} = ${value}, `)
}
// Last character always an extra comma and whitespace
setText = setText.substring(0, setText.length - 2)
db.one('UPDATE students SET ${setText} WHERE id = ${id} RETURNING *', { setText, id: req.body.id })
.then(data => {
res.json({ data })
})
.catch(err => {
res.status(400).json({'error': err.message})
})
It is supposed to dynamically generate the SQL from the request body. When I log the created SQL, it generates correctly. It even works when I directly query the database. However, whenever I run the endpoint, I get a syntax error that's "at or near" whatever setText is. I've tried using slice instead of substring with no change.
You should never concatenate values manually, as they are not escaped properly then, and open your code to possible SQL injections.
Use the tools that the library offers. For an UPDATE from a dynamic object, see below:
const cs = new pgp.helpers.ColumnSet(req.body.info, {table: 'students'});
const query = pgp.helpers.update(req.body.info, cs) +
pgp.as.format(' WHERE id = ${id} RETURNING *', req.body);
db.one(query)
.then(data => {
res.json({ data });
})
.catch(err => {
res.status(400).json({error: err.message})
});
I am creating a program in which users can preserve search terms from session to session. If a search term is preserved by the user, then the data that corresponds to that search term is also preserved. At the beginning of each session, the script discards any old data that should be preserved if it no longer corresponds to an active or connected drive.
However, I am new to working with JSON and such objects in general, so I am wondering if there is a better way to accomplish this than the way below? Specifically, is there an approach that is more efficient or even prettier than the first for of loop and the heavily nested if,for,for,if block of code?
async function guaranteeData(drives){
const
config = await readJson('./config.json'),
data = await readJson('./data.json')
// Is there a better way to write this code?
let
json = {}
for (const [drive] of drives) {
json[drive] = {}
}
// if tags have been preserved
if (config.fileTypes.length > 0) {
// loop thru all current system drives
for (const [drive] of drives) {
// loop thru all preserved tags
for (const fileType of config.fileTypes) {
// if current drive has current tag data
if (data[drive].hasOwnProperty(fileType)) {
// preserve this data: data[drive][fileType]
json[drive][fileType] = data[drive][fileType]
}
}
}
}
////////////////////////////////////////////
json = JSON.stringify(json, null, 2)
await fsp.writeFile('./data.json', json, {
flag: 'w',
encoding: 'utf8'
})
.then(() => {
return true
})
.catch(error => {
if (error.code === 'EEXIST') {
return false
} else {
throw error
}
})
}
I would change two things in this
Remove the if (config.fileTypes.length > 0) check as it will be handled the for loop which is afterwards.
Remove the for loop which assigns the json[drive] to empty object inside the for loop where you have nested for loops. This will remove that for loop as well.
It will looks something like
async function guaranteeData(drives) {
const config = await readJson("./config.json");
const data = await readJson("./data.json");
const json = {};
// if tags have been preserved
// loop thru all current system drives
for (const [drive] of drives) {
json[drive] = {};
// loop thru all preserved tags
for (const fileType of config.fileTypes) {
// if current drive has current tag data
if (data[drive].hasOwnProperty(fileType)) {
// preserve this data: data[drive][fileType]
json[drive][fileType] = data[drive][fileType];
}
}
}
////////////////////////////////////////////
json = JSON.stringify(json, null, 2);
await fsp
.writeFile("./data.json", json, {
flag: "w",
encoding: "utf8"
})
.then(() => {
return true;
})
.catch(error => {
if (error.code === "EEXIST") {
return false;
} else {
throw error;
}
});
}
How do I store the return value of dns.lookup in a variable? If I store it in an array it shows blank.
Following is the code I'm working on:
const dns = require('dns');
class Network
{
search(host)
{
let options = {
hints: dns.ADDRCONFIG | dns.V4MAPPED,
all: true,
verbatim: true
}
let addr = [];
dns.lookup(host, options, (error, address) =>
{
if(error)
{
console.log("Could not resolve %s\n", host);
console.log(error);
}
address.forEach(ip => {
addr.push({
address: ip.address,
family: ip.family
});
});
});
console.log(addr);
return addr;
}
}
const network = new Network();
let output = network.search("www.google.com");
console.log(output);
The output of the code is Blank i.e. []
Please suggest a rememdy.
Thanks
I would suggest using the async/await syntax along with the very useful util.promisify function.
This gives us code that is more compact, it will also allow you to write your code in a way that's very similar to synchronous code.
We'll then use some destructuring to pick only the properties in the address object(s) we're interested in.
const dns = require('dns');
const promisify = require('util').promisify;
class Network
{
async search(host)
{
let options = {
hints: dns.ADDRCONFIG | dns.VMAPPED,
all: true,
verbatim: true
}
let address = await promisify(dns.lookup)(host, options);
return address.map( ({address, family}) => {
return { address, family };
});
}
}
async function testSearch() {
const network = new Network();
let output = await network.search("www.google.com");
console.log(output);
}
testSearch();
The last parameter to dns.lookup must be a callback as you have already noticed. The method is asynchronous but this line return addr; is not - addr is returned before getting populated.
The method definition of search could be changed to take as second parameter a callback that will be called with the result of dns.lookup.
Something like this:
search(host, done) {
...
dns.lookup(host, options, (err, addresses) => {
...
const finalAddresses = [];
addresses.forEach((ip) => {
finalAddresses.push({
address: ip.address,
family: ip.family
});
});
done(null, finalAddresses);
});
}
const network = new Network();
network.search("www.google.com", (err, result) => {
if (err) {
return console.log(err);
}
console.log(result);
});
I'm looking a good and right way to set/ initalize values with http trigger.
what I did is ref to the node in firebase and get data then update it.
module.exports.initializeAnswers = functions.https.onRequest(async(req,res)=>{
try{
// i want to initalize each key
await firebase.ref('/CurrentGame').once('value',snapshot=>{
snapshot.forEach((childSnapshot)=>{
if(childSnapshot !=null){
childSnapshot.update(0)
}
return false
});
})
}catch(e){
console.info(e)
return res.status(400).send({error:0})
}
})
I'm looking for a right way and not by the 'update' function
I want to initalize each value to zero with http trigger
I understand that you will have several children (variable number) under the "answers" node and only one "rightAnswer" node. If my understanding is correct, the following code will do the trick:
exports.initializeAnswers = functions.https.onRequest((req, res) => {
admin.database().ref('/CurrentGame/answers').once('value', snapshot => {
const updates = {};
snapshot.forEach((child) => {
updates['/answers/' + child.key] = 0;
});
updates['/rightAnswer'] = 0;
return admin.database().ref('/CurrentGame').update(updates);
}).then(() => {
res.status(200).end();
}).catch((err) => {
console.log(err);
res.status(500).send(err);
});
});
You can use Firebase Cloud Functions to initialize the values on HTTP trigger. check this link