Develop Reference

Letter / number combination in JSON (Google spreadsheet)

I need your help. I have written a small javascript that reads a Google spreadsheet. This spreadsheet is maintained by another employee. There are the following columns named last name (column A, as example "Doe"), first name (column B, as example "John"), a Chayns ID which is used for matching (column C, as example 112-73302) and a score which the person has achieved so far.
In the Chayns environment (A platform for various apps) I query the current Chayns ID, which is assigned when logged in. Matching the JSON file provided by Google is read and the score matching the Chayns ID is output accordingly. Now unfortunately something has changed in the Chayns system. The Chayns IDs used to consist of pure numbers, but now this has been replaced by a combination of numbers and letters. The script still works with pure number-based IDs, but no longer with the new IDs in combination with the letters. Now I'm wondering where I just got the thinking error. I tried to examine the JSON file provided by Google. Where the letter number combination should be output, there is only a "NULL". How can I work around this? I am not so fit in the topic JSON!
Thanks a lot for your help!
chayns.ready.then(() => {
if (chayns.env.user.isAuthenticated) { // check if person is logged-in
const chaynsid = chayns.env.user.personId; // Chayns-ID to variable
const chaynsperson = chayns.env.user.name; // Chayns-Username to variable
let chaynsBereinigt = chaynsid; // rename
chaynsBereinigt = chaynsBereinigt.replace(/-/g, ''); // replace all dashes in Chayns-ID with nothing
document.getElementById('idausspielung').innerHTML = chaynsid;
const base = 'https://docs.google.com/spreadsheets/d/XXXXXXXXXXXXXXXXXXXXXXXXXX/gviz/tq?'; // Google Sheets Info
const output = document.querySelector('.output'); // Target is Html table with class output
const query = encodeURI('Select D where C = ' + chaynsBereinigt); // Which columns should be output with
const url = base + '&tq=' + query;
fetch(url)
.then(res => res.text())
.then(rep => {
const data = JSON.parse(rep.substr(47).slice(0, -2));
const row = document.createElement('tr');
output.append(row);
data.table.cols.forEach((heading) => {
const cell = document.createElement('td');
cell.textContent = heading.label;
row.append(cell);
})
data.table.rows.forEach((main) => {
const container = document.createElement('tr');
output.append(container);
main.c.forEach((ele) => {
const cell = document.createElement('td');
cell.textContent = ele.v;
container.append(cell);
})
})
})
console.log("Hallo " + chaynsperson + " , deine Chayns-ID lautet: " + chaynsBereinigt); // Play out values to console
}
})
JSON OUTPUT
// Correctly output values number-based
{"c":[{"v":"Doe"},{"v":"Francis"},{"v":5.0928467E7,"f":"50928467"},{"v":1028.0,"f":"1028"}]}
// Incorrectly output values number and letter based (example: 75VBUA3I)
{"c":[{"v":"Doe"},{"v":"Francis"},null,{"v":100.0,"f":"100"}]}

Retrieve a CSV file received over Gmail from a specific email address to a specific Google Sheet

Goals of the problem:
Retrieve the message from Gmail using the email address.
Retrieve the CSV files from the attachment files and put them on a sheet in Google Spreadsheet. Remove 1st 2 rows from the CSV data. (Remove the past data on Google Sheets and update it with the new CSV data whenever received from the specific email address).
Achieve this using Google Apps Script.
Already tried: Automate a CSV file received over Gmail from a specific email id to a specific Google Sheet
But nothing is solving the above 3 problems in specific.
function myFunction() {
const messageId = "###"; // Please set the message ID of Gmail.
const sheetName = "Sheet1"; // Please set the sheet name you want to put the values.
const delimiter = ","; // If your CSV data uses the specific delimiter, please set this.
const skipRows = 2; // 2 is from your question.
// 1. Retrieve message.
const message = GmailApp.getMessageById(messageId);
// 2. Retrieve attachment files.
const attachments = message.getAttachments();
if (attachments.length == 0) {
console.log("No attachment files.");
return;
}
// 3. Create an array for putting to Spreadsheet from the CSV data of attachment files.
const values = attachments.reduce((ar, e) => {
if (e.getContentType() == MimeType.CSV || e.getName().includes(".csv")) {
ar = [...ar, ...Utilities.parseCsv(e.getDataAsString(), delimiter).splice(skipRows)];
}
return ar;
}, []);
if (values.length == 0) {
console.log("No values.");
return;
}
// 4. Put the values to Spreadsheet.
const sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName(sheetName); // and, you can also use. const sheet = SpreadsheetApp.openById("###spreadsheetId###").getSheetByName(sheetName);
sheet.getRange(sheet.getLastRow() + 1, 1, values.length, values[0].length).setValues(values);
}
Tried the above code but it shows an error Exception: Invalid argument: at myFunction(Code:8:28)
FYI, I also want to use Sheet ID instead of Sheet Name

Try (put the spreadsheet ID, the subject -if you have multiple words please specify as in the example hereafter- and the email of the sender hereafter):
const getGmailAttachment = () => {
const ssID = '############'
const searchQuery = 'from:#######gmail.com in:inbox has:attachment subject:##### subject:###';
const threads = GmailApp.search(searchQuery, 0, 1);
threads.forEach(thread => {
const message = thread.getMessages()[Number(thread.getMessageCount() - 1)];
const attachments = message.getAttachments();
attachments.forEach((attachment, i) => {
if (i == 0) {
console.log(attachment.getName())
const sheet = SpreadsheetApp.openById(ssID).getSheets()[0];
sheet.getDataRange().clearContent()
const csvData = Utilities.parseCsv(attachment.getDataAsString()).splice(2); // except 2 first rows
sheet.getRange(1, 1, csvData.length, csvData[0].length).setValues(csvData);
}
});
});
};
references
GmailApp.search
search syntax
getAttachments()
Utilities.parseCsv

Google Sheets, stack report from multiple workbooks

Goal: To stack data from 90+ google workbooks, all with the same sheet name, into the one master sheet for reporting
Info:
All worksheets have the same number of columns.
I have the following script but it does not run properly, I think the issue is with how I am caching / Pushing the data to the array before pasting to the output sheet.
I am trying to build an array then paste it in one go.
The tables I am stacking have 47 columns, unknown number of rows.
The part that opens the sheets is all working perfectly.
// Get the data from the worksheets
var indexsheet = SpreadsheetApp.getActive().getSheetByName("Index");
var outputsheet = SpreadsheetApp.getActive().getSheetByName("Output");
var response = SpreadsheetApp.getUi().prompt('Current Cycle', 'Enter Cycle Name Exactly in YY-MMM-Cycle# format', SpreadsheetApp.getUi().ButtonSet.OK_CANCEL)
var CurrentCycleName = response.getResponseText()
// Assign datasets to variables
var indexdata = indexsheet.getDataRange().getValues();
// For each workbook in the index sheet, open it and copy the data to a cache
indexdata.forEach(function(row, r) {
try {
//open Entity specific workbook
var workbookid = indexsheet.getRange(r + 1, 7, 1, 1).getValues();
var Entityworkbook = SpreadsheetApp.openById(workbookid)
// Open workhseet
Entitysheet.getSheetByName(CurrentCycleName)
// Add PR Data to cache - stacking for all countrys
var PRDataCache = Entitysheet.getDataRange().push()
} catch {}
})
// Set the all values of the sheet at once
outputsheet.getRange(r + 1, 14).setValue('Issue Splitting Data')
Entitysheet.getRange(2, 1, PRDataCache.length || 1, 47).setValues(PRDataCache)
};
This is the index tab where we are getting the workbookid from to open each file
This is the output file, we are stacking all data from each country

I believe your goal is as follows.
You want to retrieve the Spreadsheet IDs from the column "G" of "Index" sheet.
You want to give the specific sheet name using a dialog.
You want to retrieve all values from the specification sheet in all Spreadsheets. In this case, you want to remove the header row.
You want to put the retrieved values on "Output" sheet.
In this case, how about the following sample script?
Sample script:
function myFunction() {
var ss = SpreadsheetApp.getActive();
var indexsheet = ss.getSheetByName("Index");
var outputsheet = ss.getSheetByName("Output");
var response = SpreadsheetApp.getUi().prompt('Current Cycle', 'Enter Cycle Name Exactly in YY-MMM-Cycle# format', SpreadsheetApp.getUi().ButtonSet.OK_CANCEL);
var CurrentCycleName = response.getResponseText();
var ids = indexsheet.getRange("G1:G" + indexsheet.getLastRow()).getValues();
var values = ids.reduce((ar, [id]) => {
try {
var [, ...values] = SpreadsheetApp.openById(id).getSheetByName(CurrentCycleName).getDataRange().getValues();
ar = [...ar, ...values];
} catch (e) {
console.log(`"${id}" was not found.`);
}
return ar;
}, []);
if (values.length == 0) return;
// If the number of columns is different in all Spreadsheets, please use the following script.
// var maxLen = Math.max(...values.map(r => r.length));
// values = values.map(r => r.length < maxLen ? [...r, ...Array(maxLen - r.length).fill("")] : r);
outputsheet.getRange(outputsheet.getLastRow() + 1, 1, values.length, values[1].length).setValues(values);
}
Note:
When the number of Spreadsheet IDs is large, the processing time might be over 6 minutes. I'm worried about this. At that time, how about separating the Spreadsheet IDs?
Reference:
reduce()

Make changes of colors of icons visible in openlayers with a function

I try to accomplish that I will be able to change the color of marker icons to be visible in the map. They are in the map in different colors. Those colors are correspondenting to the json cat_id key's.
1: "http://dev.openlayers.org/img/marker.png",
2: "http://dev.openlayers.org/img/marker-blue.png",
3: "http://dev.openlayers.org/img/marker-gold.png",
4: "http://dev.openlayers.org/img/marker-green.png",
Each marker icon that is in the map is getting it's position, color and other valuable data from the json objects in the data id in the script tag. I didn't code the part of that work's close with the openlayer libary's. I wrote the following part of the total of the script:
var json = document.getElementById('data').innerHTML;
json = JSON.parse(json); // converts text into JSON
// search criteria (could be changed)
var $search_postal = "8912JB";
var $search_number = "10";
var $change_cat = 1;
function changeColor() {
for (var key in json) {
if (json.hasOwnProperty(key)) {
if (json[key].postal == $search_postal) {
if (json[key].number == $search_number) {
alert("Gonna change cat_id " + json[key].cat_id + " of the number search string " + $search_number + " of the postal search string " + $search_postal + " to cat_id " + $change_cat);
json[key].cat_id = "1";
alert('Changed!');
const posts = json; // the constant that is used to display the icons
var myJSON = json;
document.getElementById('demo').innerHTML = JSON.stringify(myJSON, undefined, 2); // changes of the json data to see on screen
}
}
}
}
}
const posts = json; //the constant that is used to display the icons
var myJSON = JSON.stringify(json);
document.getElementById('demo').innerHTML = myJSON;
This script will change the cat_id of the given search input. In the example it does a search for postal 8912JB with number 1. Once it has made the loop to find it's position it will change the cat_id value of 3 to 1. In other words the json data has been changed. Since it is stored in the const posts I replace this cons with the changed data. My problem is that I'm unable to refresh the marker icon(s) in the map with the new json data that is changed (cat_id: 1 of postal 8912JB number 10). This has to do with less knowledge about openlayers and Javascript. I'm struggling already 2 weeks with this. Can anybody help me to accomplish this? If it changes from color is the only thing that is required. I would only need to give the changes to the variables $search_postal, $search_number and $change_cat.
Here is the page for debug: https://jsfiddle.net/j4z1vpht/4/
Thanks for any help,
grid

You need to update the new category to the map feature, like this:
const updateFeatureCategory = (featureId, categoryId) => {
const feature = marker_layer.getSource().getFeatureById(featureId);
const post = feature.get("post");
const updatedPost = {
...post,
cat_id: categoryId
}
feature.set("post", updatedPost);
}
function changeColor() {
...
json[key].cat_id = "1";
updateFeatureCategory(key, "1");
...
}
Here's a working fiddle: https://jsfiddle.net/wsm2h3qz/1/

Cheerio Not Parsing HTML Correctly

I've got an array of rows that I've parsed out of a table from html, stored in a list. Each of the rows in the list is a string that looks (something) like this:
["<td headers="DOCUMENT" class="t14data"><a target="6690-Exhibit-C-20190611-1" href="http://www.fara.gov/docs/6690-Exhibit-C-20190611-1.pdf" class="doj-analytics-processed"><span style="color:blue">Click Here </span></a></td><td headers="REGISTRATIONNUMBER" class="t14data">6690</td><td headers="REGISTRANTNAME" class="t14data">SKDKnickerbocker LLC</td><td headers="DOCUMENTTYPE" class="t14data">Exhibit C</td><td headers="STAMPED/RECEIVEDDATE" class="t14data">06/11/2019</td>","<td headers="DOCUMENT" class="t14data"><a target="5334-Supplemental-Statement-20190611-30" href="http://www.fara.gov/docs/5334-Supplemental-Statement-20190611-30.pdf" class="doj-analytics-processed"><span style="color:blue">Click Here </span></a></td><td headers="REGISTRATIONNUMBER" class="t14data">5334</td><td headers="REGISTRANTNAME" class="t14data">Commonwealth of Dominica Maritime Registry, Inc.</td><td headers="DOCUMENTTYPE" class="t14data">Supplemental Statement</td><td headers="STAMPED/RECEIVEDDATE" class="t14data">06/11/2019</td>"]
The code is pulled from the page with the following page.evaluate function using puppeteer.
I'd like to then parse this code with cheerio, which I find to be simpler and more understandable. However, when I pass each of the strings of html into cheerio, it fails to parse them correctly. Here's the current function I'm using:
let data = res.map((tr) => {
let $ = cheerio.load(tr);
const link = $("a").attr("href");
const number = $("td[headers='REGISTRATIONNUMBER']").text();
const name = $("td[headers='REGISTRANTNAME']").text();
const type = $("td[headers='DOCUMENTTYPE']").text();
const date = $("td[headers='STAMPED/RECEIVEDDATE']").text();
return { link, number, name, type, date };
});
For some reason, only the "a" tag is working correctly for each row. Meaning, the "link" variable is correctly defined, but none of the other ones are. When I use $("*") to return a list of what should be all of the td's, it returns an unusual node list:
What am I doing wrong, and how can I gain access to the td's with the various headers, and their text content? Thanks!

It usually looks more like this:
let data = res.map((i, tr) => {
const link = $(tr).find("a").attr("href");
const number = $(tr).find("td[headers='REGISTRATIONNUMBER']").text();
const name = $(tr).find("td[headers='REGISTRANTNAME']").text();
const type = $(tr).find("td[headers='DOCUMENTTYPE']").text();
const date = $(tr).find("td[headers='STAMPED/RECEIVEDDATE']").text();
return { link, number, name, type, date };
}).get();
Keep in mind that cheerio map has the arguments reversed from js map.

I found the solution. I'm simply returning the full html through puppeteer instead of trying to get individual rows, and then using the above suggestion (from #pguardiario) to parse the text:
const res = await page.evaluate(() => {
return document.body.innerHTML;
});
let $ = cheerio.load(res);
let trs = $(".t14Standard tbody tr.highlight-row");
let data = trs.map((i, tr) => {
const link = $(tr).find("a").attr("href");
const number = $(tr).find("td[headers='REGISTRATIONNUMBER']").text();
const registrant = $(tr).find("td[headers='REGISTRANTNAME']").text();
const type = $(tr).find("td[headers='DOCUMENTTYPE']").text();
const date = moment($(tr).find("td[headers='STAMPED/RECEIVEDDATE']").text()).valueOf().toString();
return { link, number, registrant, type, date };
});