Develop Reference

How to handle empty <dt> values in Web-Scraping using JavaScript

I have a JavaScript code that scrapes data in a dl Description List.
There are possible 7 dt values with respective dd values.
Only those of the 7 dt values and their dd values are shown on the website that have at least 1 dd value - thus, it can be 1 dt incl. dd is scraped or in another webpage 7
OK, I have a working Javascript code, that does the job!
const columns = [
{ text: 'Website', name: 'Website' },
{ text: 'Phone', name: 'Phone' },
{ text: 'Industry', name: 'Industry' },
{ text: 'Company size', name: 'Companysize' },
{ text: 'Headquarters', name: 'Headquarters' },
{ text: 'Founded', name: 'Founded' },
{ text: 'Specialties', name: 'Specialties' },
{ text: 'Employees', name: 'Employees' }
]
const result = [];
const elements = document.querySelectorAll('dl dt');
elements.forEach((element) => {
const regex = new RegExp(element.innerText, 'i');
const findColumn = columns.find((column) => regex.test(column.text));
if (!findColumn) return;
const columnValue = element.nextElementSibling.innerText;
result.push({ [findColumn.name]: columnValue });
});
Problem
I want to save the scraping results in an MS Excel table that has 7 columns
BUT --> The result of the scraping can have1 up to 7 columns
Because of that, I can't simply append the results, row by row - I have to do it manually. Copying the right value in the right column for it.
I would need a code that can do the following:
The values of the 7 dt elements are the header of the 7 columns
The code always results in 7 values
If the ddis a real value and accordingly scraped by the code above, then it is put in the correct column
If there is no dt element, then the string "n/a" should be put under the respective column.
This way, the results are stored in a consistent Excel table with always the correct values in the correct column.
I cannot find any specific info or material or sample code to write JavaScript code to solve this task! I think, a JavaScript expert is needed to write that code.
Thank you for helping me out to understand JavaScript better and to learn how to write JS code.
P.S.: The website scraping from: LinkedIn companies

Creating dynamic number of columns or header in excel in nodejs/javascript

I have used exceljs module in nodejs for exporting json data to excel. It's working fine, but the names of headers/columns have to be predefined before adding rows i.e., columns are fixed. After addition of rows, I can't add columns dynamically.
I have tried a number of modules available through npm but all of them have the same features.
So, is there any way or module that, at the time of manipulation of json data, can create a new column and add the required row.

If someone is still looking into this problem then I have a decent solution.
Instead of creating columns, you can create a table as follows in the worksheet.
worksheet.addTable({
name: "MyTable",
ref: "A1",
headerRow: true,
totalsRow: false,
style: {
theme: null,
showRowStripes: true,
showColumnStripes: true,
},
columns: [
{ name: "EmployeeID" },
{ name: "First Name" },
],
rows: [/* Enter initial rows if you want to add*/],
});
After adding a table to the column A1 of your worksheet you can add new columns dynamically
const table = worksheet.getTable("MyTable");
table.addColumn({
name: "Column name",
});
table.commit();

I tried directly pushing the new columns to the worksheet.columns but it is not working. I did a workaround and working well for me.
Note: You need to make the track of already added columns in the worksheet to get the next empty columns by column index.
Here is an example:
let workbook = new excel.Workbook(); //creating workbook
let worksheet = workbook.addWorksheet('Records'); //creating worksheet
const columns = [];
columns.push({header: 'Id', key: '_id', width: 30});
columns.push({header: 'Name', key: 'name', width: 30});
//Set Headers to WorkSheet Header
worksheet.columns = columns;
//Now insert some records if you want
worksheet.addRow({_id: "1", name: "Mitchell Starc"});
worksheet.addRow({_id: "2", name: "Ab de Villiers"});
//Update or add dynamic columns
//Get the empty columns from worksheet. You can get the empty columns number by using `columns` array length
//For this you have to track all inserted columns in worksheet
//This will return the next empty columns
let newColumn = worksheet.getColumn(columns.length + 1);
//Set new key header and all other required properties
newColumn.key = "profession";
newColumn.header = "Profession";
newColumn.width = 30;
//Add the new column to `columns` to track the added headers
columns.push(newColumn);
//Now you can insert rows with new columns
worksheet.addRow({_id: "3", name: "MS Dhoni", profession: "Cricket"});
workbook.xlsx.writeFile("records.xlsx")
.then(function () {
console.log("file saved!");
});

Now not sure if this worked 2 years ago but this worked for me
var columns=[]
x="data1"
y="data2"
Columns.push({ header: x, key: x })
Columns.push({ header: y, key: y})
worksheet.columns = Columns
You must use a separate variable to dynamically create the array of structs for it to work. if you use worksheet.columns=[] and worksheet.columns.push(..) it will fail.

pouchdb put is still rejected with _rev

I'm using pouchDB for the first time, and as indicated in the docs I'm using put() so it will automatically handle revisions. However, when the code is running and there's an existing item with the same ID, it's still rejecting even when including a _rev.
Here's my code:
var db = new PouchDB('blog')
...
function saveCategory(category) {
var savedCategory = {
_id: 'category' + category.id,
_rev: '2-' + String(new Date().toISOString()),
name: category.name,
nicename: category.slug,
post_count: category.count,
description: category.description,
link: category.link,
parent: category.parent
}
return db.put(savedCategory).then((response) => {
$log.log(response)
}).catch((error) => {
$log.error('error saving category ',error)
})
}

This is not the purpose of the _rev field. It is always generated by the server and not by your code. To update a document you must pull the entire document (including the _rev field), update the desired fields, and then put the document. The value of _rev should be the same as when you got it from the server.
If you have a new record, you do not need to set _rev.
The pocketDB guide has a very useful section about this.

How can I limit the number of objects fetched from a Parse.com collection (Javascript)?

Can I do anything from here to limit the number of objects that are returned? By default, it is 100 and I don't need that many for now. Anyway to just fetch the most recent 10 objects in that class?
var Blog = Parse.Object.extend("Blog");
var Blogs = Parse.Collection.extend({
model: Blog
});
var blogs = new Blogs();
blogs.fetch({
success: function(blogs) {
var blogsView = new BlogsView({ collection: blogs });
blogsView.render();
$('.main-container').html(blogsView.el);
},
error: function(blogs, error) {
console.log(error);
}
});

From the backbone fetch doc:
jQuery.ajax options can also be passed directly as fetch options, so
to fetch a specific page of a paginated collection:
Documents.fetch({data: {page: 3}})
For setting the number of objects returned, the option name is limit, so:
blogs.fetch({ data: {limit: 10}, ...

I did it like this:
Blogs = Parse.Collection.extend({
model: Blog,
query: (new Parse.Query(Blog)).descending('createdAt').limit(10),
}),
...and showed the newest objects first by sorting by descending order.

Meteor User table value axtracting

How do I pick the email address value from meteor Mongo user table?
I have written below query to pick the element:
users=Meteor.users.find({},{emails:1})
This the code I have written to fetch the email address, but I don't know how much it's affecting performance in the code:
users = Meteor.users.find({})
users.forEach(function(key,option){
key.emails.forEach(function (key,option){
console.log(key.address)
});
});

In meteor, you should call:
users = Meteor.users.find({}, { fields: { emails: 1 } })
Reference in docs
EDIT
Please remember users is a cursor object. Cursor objects can be handled directly in templates, and must be the return of publications. You can't iterate a cursor directly in a javascript loop.
Example: (remember authorization in production publications)
Meteor.publish('user-emails', function() {
return Meteor.users.find({}, { fields: { emails: 1 } });
});
If you want to directly access the user instances, for example to iterate them in a javascript code, you need to fetch the cursor (reference in docs).
Example:
var users = Meteor.users.find({}, { fields: { emails: 1 } }).fetch();
Now users is an array of users. Feel free to iterate them.
Example (I'm using underscore.js):
var users = Meteor.users.find({}, { fields: { emails: 1 } }).fetch();
_.each(users, function(user) {
console.log(user.emails);
});
Now, if you need a vector only with emails, one on each index, you can pluck the emails from a fetched array with underscore.js (reference of pluck)
var emails = _.pluck(Meteor.users.find({}, { fields: { emails: 1 } }).fetch(), 'emails');
Hope it works :)

if its not working, dont forget to return
return users