I've got an array question for processing in google sheets. I'm trying to map a value from one sheet into another. Thanks for any help. I've got explanation and code below...
PROBLEM: this gives me blank rows (policies2[i]).
DESIRED OUTCOME: new values from entry_top in the last row of policies2.
I'm working with two sheets. policies.values is the array from a normal google sheet of tabular values. policies2 is a copy of policies. entry_top is the array of values from a sheet with unstructured data, except that headers and values are beside each other.
So if "Line" is a header in policies, it would find "Line" in a cell/array node in entry_top and then get the next cell/array value (next column/index/to the right), take that value and put it in the last row in the same column in policies (match by header).
So the code below loops through policies.values[0], the header values. Then loops through every cell in entry_top, by rows (e) and columns (ee). Then if the headers match (h and ee), it puts the next value ([ei][eei+1]) in the matching policies2 column in the last row ([policies2.length-1][hi]).
policies.values[0].map((h, hi) => {
entry_top.map((e, ei) => {
e.map((ee, eei) => {
if (ee == h) policies2[policies2.length - 1][hi] = entry_top[ei][eei + 1];
});
});
});
MRE: oddly this example works. so not sure what above is causing an issue...
function testdp() {
policies = {
values:[[1,2,3],[4,5,6],[7,8,9]]
}
policies2=[[1,2,3],[4,5,6],[7,8,9],[,,,]];
entry_top = [[,,1,'add',,],[,'a','b',2,'add2',,'c'],['c',,,3,'add3',,]]
Logger.log(policies.values);
Logger.log(policies2);
Logger.log(entry_top);
policies.values[0].forEach((h,hi)=>{
entry_top.forEach((e,ei)=>{
e.forEach((ee,eei)=>{
if (ee == h) policies2[policies2.length - 1][hi] = entry_top[ei][eei + 1];
})
})
});
Logger.log(policies2);
// [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0], [add, add2, add3]]
}
this all works just fine. I was logging the wrong result.
Related
I have two sheets, "IMPORT" and "CASES".
In the "IMPORT" sheet, I am importing data from an external source that sometimes have more columns or existing columns are arranged each time differently.
In the "CASES" sheet, this is where I store a weekly snapshot of all last week's imported data, and I add my additional columns with more pieces of information such as comments, next steps etc.
I am looking for a way to compare both sheets without hardcoding any column ranges. I thought the most efficient way to do it is by looking up column header names in both sheets and then checking for changes in reference to the "Case Number" row. Please let me know if you can think of a better way.
I have already managed to write a code to look through headers and identify the Index number for a specific column name, "Case Number".
This column will always be present in both sheets, and it could serve as a reference point to the row that should be validated, but it could be a different row for each sheet at a time.
I will need the same time loop through all the column headers from the CASES sheet and check for updates from the IMPORT sheet.
I only need to check/loop for changes for few specific columns from the CASES sheet. Columns names such: Contact Name, Title, Priority, Status.
I am aiming to achieve 3 possible outcomes:
[ COMPLETED ] "Case Number" from the CASES sheet was NOT FOUND in the IMPORT sheet - that means the case was closed since last week.
Action: Highlight an entire row in the CASES sheet as grey (this will indicate the case is no longer open and should be removed from the list after confirmation).
"Case Number" from the IMPORT sheet was NOT FOUND in the CASES sheet - this means the case is new and needs to be added to the CASES sheet at the bottom.
Action: Copy the data from the IMPORT sheet to the CASES sheet and paste it in the correct columns at the bottom and highlight the entire row as green to indicate a new data entry.
For all non-existing columns in the CASES sheet that are in the IMPORT sheet, those should be skipped.
"Case Number" from the IMPORT sheet WAS FOUND in the CASES sheet - for the matching Case Number records, I need to validate if there were any changes in any CASES sheet columns since last week.
Action: If a change was found in any of the cells, update the cell with new data in CASES sheet and change the cell background colour to yellow to highlight the cell was updated. For cells without changes, skip.
I apologise for the lengthy problem statement.
I am new to JS and GAS, and I wrote it hoping that some JavaScript expert will understand my idea and advise maybe the easier way to complete my project.
Currently, I am stuck with finding a proper way to loop through Header Names then check cell value from the IMPORT sheet and comparing it with the CASES sheet based on the Case Name value/row.
OUTCOME 1 - Completed
OUTCOME 2 - In Progress
OUTCOME 3 - tbd...
I will continue to update this topic to show the latest progress on this project.
All the examples I found so far on the Internet were based on hardcoded ranges of cells and columns. I think my approach is interesting as it gives future-proof flexibility to the datasets.
Please let me know your thoughts or ideas for a more straightforward approach :)
Link to live sheet
UPDATED code:
// Create Top Menu
function onOpen() {
let ui = SpreadsheetApp.getUi();
ui.createMenu('>> REPORTS <<').
addItem('Highlight Closed Cases', 'closedCases').
addItem('Check for new Cases', 'addCases').addToUi();
}
// IN PROGRESS (Outcome 2) - Add and highlight new cases in CASES sheet
function addCases() {
let ss = SpreadsheetApp.getActiveSpreadsheet();
// Get column index number for Case Number
let activeImportCol = getColumnIndex("Case Number", "IMPORT");
let activeCasesCol = getColumnIndex("Case Number", "CASES");
let importHeaders = loadHeaderNames("IMPORT");
let casesHeaders = loadHeaderNames("CASES");
// Load Case Number columns values into array
let loadImportValues = getColumnValues("Case Number", "IMPORT");
let loadCasesValues = getColumnValues("Case Number", "CASES");
// Convert to 1D array
let newImportValues = loadImportValues.map(function (row) { return row[0]; });
let newCasesValues = loadCasesValues.map(function (row) { return row[0]; });
// Get number of columns
var numImportCol = ss.getSheetByName("IMPORT").getLastColumn();
// Loop through IMPORT sheet "Case Number" column to find new Case Numbers - execute OUTCOME 3 or 2
for (var line in newImportValues) {
var isMatched = newCasesValues.indexOf(newImportValues[line]);
if (isMatched !== -1) {
// "Case Number" from the IMPORT sheet WAS FOUND in the CASES sheet - EXECUTE OUTCOME 3
// ****************************************************************************************
// For the matching Case Number records, I need to validate if there were any changes in any CASES sheet columns since last week
// Action: If a change was found in any of the cells, update the cell with new data in CASES sheet
// and change the cell background colour to yellow to highlight the cell was updated. For cells without changes, skip.
} else {
// "Case Number" from the IMPORT sheet was NOT FOUND in the CASES sheet - EXECUTE OUTCOME 2
// ****************************************************************************************
// Copy the new data row from the IMPORT sheet to the CASES sheet and paste it in the correct columns
// at the bottom and highlight the entire row as green to indicate a new data entry.
// For all non-existing/not matching column names in the CASES sheet that are not in IMPORT sheet, those should be skipped.
}
}
}
// COMPLETED (Outcome 1) - Highlight entire row grey for missing values in CASES sheet
function closedCases() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
// Load all Casen Number columns values into array
var importValues = getColumnValues("Case Number", "IMPORT");
var casesValues = getColumnValues("Case Number", "CASES");
// Convert to 1D array
var newImportValues = importValues.map(function (row) { return row[0]; });
var newCasesValues = casesValues.map(function (row) { return row[0]; });
// Get column index number for Case Number
var activeCol = getColumnIndex("Case Number", "CASES");
// Get number of columns
var numCol = ss.getSheetByName("CASES").getLastColumn();
// Loop though CASES "Case Number" column and highlight closed cases (not found in IMPORT tab)
for (var line in newCasesValues) {
var isMatched = newImportValues.indexOf(newCasesValues[line]);
if (isMatched !== -1) {
// If found then...
ss.getSheetByName("CASES").getRange(+line + 2, 1, 1, numCol).setBackground(null);
} else {
// Higlight row with missing cases - grey
ss.getSheetByName("CASES").getRange(+line + 2, 1, 1, numCol).setBackground("#d9d9d9");
};
}
}
// Load column values
function getColumnValues(label, sheetName) {
var ss = SpreadsheetApp.getActiveSpreadsheet().getSheetByName(sheetName);
// Get column number for Case Number
var colIndex = getColumnIndex(label, sheetName);
// Get number of rows in Case Number
var numRows = ss.getLastRow() - 1;
// Load Case Number values into array
var colValues = ss.getRange(2, colIndex, numRows, 1).getValues();
return colValues;
}
// Load column header names
function loadHeaderNames(sheetName) {
var ss = SpreadsheetApp.getActiveSpreadsheet().getSheetByName(sheetName);
let HeaderArray = ss.getRange(1, 1, 1, ss.getLastColumn()).getValues()[0];
let colidx = {};
HeaderArray.forEach((h, i) => colidx[h] = i);
return HeaderArray;
}
// Get column name index value
function getColumnIndex(label, sheetName) {
var ss = SpreadsheetApp.getActiveSpreadsheet().getSheetByName(sheetName);
// Find last column
var lc = ss.getLastColumn();
// Load headers into array
var lookupRangeValues = ss.getRange(1, 1, 1, lc).getValues()[0];
// Search for label and return the column number
var index = lookupRangeValues.indexOf(label) + 1;
return index;
}
One way to make all this processing much easier is to reorder the columns so that they always fall in the same place, like this:
=arrayformula(
iferror(
vlookup(
hlookup("Case Number"; IMPORT!A1:G; row(IMPORT!A2:G); false);
{
hlookup("Case Number"; IMPORT!A1:G; row(IMPORT!A1:G); false) \
IMPORT!A1:G
};
match(IMPORT!A1:G1; CASES!A1:G1; 0) + 1;
false
)
)
)
The formula will reorder the columns in IMPORT so that the columns are in the same order as they are listed in CASES!A1:G1.
You can then use further formulas or script functions to work on the data, confident that a particular kind of data will always be in the same column. For instance, you can list closed cases with something like this:
=filter( 'CASES normalized'!A2:G; isna(match('CASES normalized'!C2:C; 'IMPORT normalized'!C2:C; 0)) )
...and open cases like this:
=filter( 'CASES normalized'!A2:G; match('CASES normalized'!C2:C; 'IMPORT normalized'!C2:C; 0) )
See your sample spreadsheet.
I have a requirement to remove a list of exceptions/exclusions from a large (115244 rows) data set.
The total number of exclusions is 1133.
I have the following piece of code:
function removeExclusions() {
const ss = SpreadsheetApp.getActive();
const exclusionSheet = ss.getSheetByName("Exclusion List");
const excludedAccounts = exclusionSheet
.getRange("C2:C" + exclusionSheet.getLastRow())
.getValues()
.reduce(
(o, [c]) =>
Object.assign(o, {
[c]: true
}),
{}
);
Logger.log("Total accounts to remove: " + excludedAccounts);
const dataSheet = ss.getSheetByName("Data Sheet");
const dataSheetMatches = dataSheet
.getRange("A2:A" + dataSheet.getLastRow())
.getValues()
.reduce((ar, [a], i) => {
if (excludedAccounts[a]) ar.push(i + 2);
return ar;
}, [])
.reverse();
Logger.log("Accounts left to remove: " + dataSheetMatches.length);
dataSheetMatches.forEach((r) => {
dataSheet.deleteRow(r);
Logger.log("Row:" + r + " has been deleted");
});
}
However, due to the size of the dataset/number of accounts/rows, this needs to run against - it takes an eternity and hits the timeout that Google has on Apps Script runtimes.
I need a much more efficient way to do the following:
Check the "Exclusion List" sheet (column C), then remove the row in the "Data Sheet" when it matches against column A.
Rows A-O can be cleared. Columns P-S header row (1) contains some formulas that I need to keep.
Any suggestions?
EDIT
So I have amended my code to the following:
function removeExclusions() {
const ss = SpreadsheetApp.getActive();
const exclusionSheet = ss.getSheetByName("Exclusion List");
const exclusionRange = exclusionSheet.getRange("C2:C");
const exclusionVals = exclusionRange.getDisplayValues();
const dataSheet = ss.getSheetByName("Data Sheet");
const dataSheetRange = dataSheet.getRange("A2:O");
let dataSheetVals = dataSheetRange.getValues();
dataSheetVals = dataSheetVals.filter((data) => {
return !exclusionVals.includes(data[0]);
});
Logger.log(dataSheetVals);
}
However, it's still showing the rows I would expect it to exclude...
it takes an eternity and hits the timeout that Google has on Apps
Script runtimes.
The cause of this is
dataSheet.deleteRow(r);
Even if you try it on 20 rows sheet, it's slow, you can almost observe rows being deleted one by one.
Besides that, 100K+ rows is sometimes too much for Google Sheets, the response time is slower.
The strategy you could use
Pull all data from "Data Sheet" into variable, using getValues(). (exclude formula columns if possible)
Pull "Exclusion List" column C into variable just like you did in the code
Use .filter on "Data Sheet" values array to create new array by removing the unwanted rows
Clear "Data Sheet" values (exclude formula columns if possible)
Write reduced data into "Data Sheet", using setValues().
Notes
This approach can take a long time too. It's much faster than deleteRow, but getValues and setValues on 100K+ rows and I guess 15-20 columns will take some time. The rough estimation of the execution time would be to do manual select all/copy/paste tests on the "Data Sheet" and see how long you have to wait.
If some person/process is adding/removing/moving/editing "Data Sheet" at the same time your function runs, you may have data loses. If you can prevent this by using some overnight once per day trigger to run your code you should be fine.
In general your code can break, you could make a duplicate of the sheet (to serve as a backup) at the beginning and then if everything is executed without an error delete the duplicate sheet at the end.
In general you should consider moving your data from Google Sheets to some platform that can handle larger amounts of records, I use BigQuery for such scenarios.
Edit: response to your amended code
Assuming that values you are comparing are strings:
function somethingLikeThis() {
const ss = SpreadsheetApp.getActive();
// Load "Exclusion List" column C to array of strings
const exclusionSheet = ss.getSheetByName("Exclusion List");
const lastRowExclusionSheet = exclusionSheet.getLastRow();
const exlusionList = exclusionSheet
.getRange(2, 3, lastRowExclusionSheet - 2 + 1) // Data starts at row 2, column is C
.getValues()
.map(row => row[0].toString()) // Convert array of arrays to array of strings
.filter(el => el.length > 0); // Remove empty rows if any
// Read values from "Data Sheet"
const dataSheet = ss.getSheetByName("Data Sheet");
const lastRowDataSheet = dataSheet.getLastRow();
const oldDataRange = dataSheet
.getRange(2, 1, lastRowDataSheet - 2 + 1, 15) // Data starts at row 2, columns A-O
const oldDataValues = oldDataRange.getValues();
// Clear "Data Sheet"
oldDataRange.clearContent();
// Keep rows where column A value is not on the "Exclusion List"
const newDataValues = oldDataValues
.filter(row => exlusionList.indexOf(row[0].toString()) < 0);
// Write reduced rows to "Data Sheet"
if (newDataValues.length > 0) {
dataSheet.getRange(2, 1, newDataValues.length, newDataValues[0].length)
.setValues(newDataValues);
}
}
Code is not tested, I don't have actual sheets. Try using getLastRow(), ranges like "C2:C" can pick empty rows at the end of the sheet.
I want to compare two sheets (based on header values in row 1) and delete any column with a unique value (without a match). For example, Assuming Sheet1, Row 1 data and Sheet 2, Row 1 are uniform, if a user adds/deletes a column within any sheet, I want to always match the number of columns in both sheets with their values
Screenshots of sheets headings.
IF both sheets looks like this
And a user adds a new Column N
Or delete column N
How can I ensure that both sheet matches by deleting the odd/distinct column in Sheet 1?
I have tried modifying this code below but I can't just get the unique one out. This code only look for headers with a defined value.
function deleteAloneColumns(){
var sheet = SpreadsheetApp.getActiveSheet();
var lastColumnPos = sheet.getLastColumn();
var headers = sheet.getRange( 1 ,1, 1, lastColumnPos ).getValues()[0];
for( var i = lastColumnPos ; i < 1; i--){
if( headers[i] === "alone" ) sheet.deleteColumn(i);
}
SpreadsheetApp.getUi().alert( 'Job done!' );
}
Any help to compare and delete the column with the unique value will be appreciated.
Problem
Balancing sheets based on header row values mismatch.
Solution
If I understood you correctly, you have a source sheet against which validation is run and two primary use cases: user adds a new column named differently than any other column (if you want to check that the column strictly matches the one in sheet1, it is easy to modify) in source sheet or deletes one that should be there.
const balanceSheets = (sourceShName = 'Sheet1',targetShName = 'Sheet2') => {
const ss = SpreadsheetApp.getActiveSpreadsheet();
const s1 = ss.getSheetByName(sourceShName);
const s2 = ss.getSheetByName(targetShName);
const s2lcol = s2.getLastColumn();
//keep all vals from source to reduce I/O
const s1DataVals = s1.getDataRange().getValues();
const s2Vals = s2.getRange(1, 1, 1, s2lcol).getValues();
const h1Vals = s1DataVals[0];
const h2Vals = s2Vals[0];
//assume s1 is source (validation) sheet
//assume s2 is target sheet that a user can edit
//case 1: target has value not present in source -> delete column in target
let colIdx = 0;
h2Vals.forEach(value => {
const isOK = h1Vals.some(val => val===value);
isOK ? colIdx++ : s2.deleteColumn(colIdx+1);
});
//case 2: target does not have values present in source -> append column from source
h1Vals.forEach((value,index) => {
const isOK = h2Vals.some(val => val===value);
!isOK && s2.insertColumnAfter(index);
const valuesToInsert = s1DataVals.map(row => [row[index]]);
const numRowsToInsert = valuesToInsert.length;
s2.getRange(1,index+1, numRowsToInsert,1).setValues(valuesToInsert);
});
};
Showcase
Here is a small demo of how it works as a macros:
Notes
Solving your problem with two forEach is suboptimal, but I kept number of I/O low (it can be lowered further by, for example, moving deleteColum out of the loop while only keeping track of column indices).
The script uses ES6 capabilities provided by V8, so please, be careful (although I would recommend migrating as soon as possible - even if you encounter bugs / inconsistencies , it is worth more than it costs.
UPD made script more flexible by moving sheet names to parameter list.
UPD2 after discussing the issue with deleteColumn() behaviour, the answer is updated to keep column pointer in bounds (for those curious about it - forEach kept incrementing the index, while deleteColumn reduced bounds for any given index).
Reference
insertColumnAfter() method reference
I am trying to create a simple "plug-n-play" map template, that allows user to put a csv file with geoids and values and then see the values as a choropleth.
Right now I am merging two datasets (map and values) using double loop, but wondering if there is any other option:
This chunk of code stays within the function that loads geodata (fresh_ctss) :
d3.csv("data/communities_pop.csv", function(error, comms)
{
csv = comms.map(function(d)
{
//each d is one line of the csv file represented as a json object
// console.log("Label: " + d.CTLabel)
return {"community": d.community, "population" :d.population,"label": d.tract} ;
})
csv.forEach(function(d, i) {
fresh_ctss.forEach(function(e, j) {
if (d.label === e.properties.geoid) {
e.properties.community = parseInt(d.community)
e.properties.population = parseInt(d.population)
}
})
})
You'll definitely need two loops (or a nested loop) - the most optimal way would be to just limit how much iteration needs to happen. Right now, the first loop goes through every csv row. The following nested loop goes through every csv row (as new different object) and then, as many times as there are rows in the csv, through every item in fresh_ctss.
If you mapped the rows into an object instead of an array, you could iterate through the rows once (total) and then once through the elements of fresh_ctss (again, total). Code below assumes that there are no tract duplicates in comms:
all_comms = {}
comms.forEach(function(d) {
all_comms[d.tract] = {"community": d.community, "population": d.population}
})
fresh_ctss.forEach(function(e) {
comm = all_comms[e.properties.geoid]
e.properties.community = parseInt(comm.community)
e.properties.population = parseInt(comm.population)
}
I am writing a script for google sheet validation on localization tests. I've gotten stuck on some of the logic. The purpose of the script is to 1) Iterate through all tabs. 2) Find the column on row 2 that has the text "Pass/Fail". Lastly, 3) Iterate down that column and return the rows that say Fail.
The correct script to look at is called combined(). Step 1 is close to being correct, I think. Step 2 has been hard coded for the moment and is not dynamic searching the row for the text. Step 3 is done.
Any help would be great :)!!! Thanks in advance.
https://docs.google.com/spreadsheets/d/1mJfDtAi0hHqhqNB2367OPyNFgSPa_tW9l1akByaTSEk/edit?usp=sharing
/*This function is to cycle through all spreadsheets.
On each spreadsheet, it will search the second row for the column that says "Pass/Fail".
Lastly, it will take that column and look for all the fails and return that row*/
function combined() {
var sheets = SpreadsheetApp.getActiveSpreadsheet().getSheets();
var r =[];
for (var i=0 ; i<sheets.length ; i++){//iterate through all the sheets
var sh = SpreadsheetApp.getActiveSheet();
var data = sh.getDataRange().getValues(); // read all data in the sheet
//r.push("test1"); //Testing to make sure all sheets get cycled through
/*I need something here to find which column on row two says "Pass/Fail"*/
for(i=3;i<data.length;++i){ // iterate row by row and examine data in column A
//r.push("test2"); //Testing to make sure the all
if(data[i][7]=='Fail'){ r.push(data[i])}; // if column 7 contains 'fail' then add it to the list
}
}
return r; //Return row of failed results on all tabs
}
At first, it retrieves data at column g. It retrieves a result from the data. The result is 2 dimensional array. The index of each element of the 2D array means the sheet index. If the sheet doesn't include values in column g, the element length is 0.
For example, in the case of following situation,
Sheet 0 doesn't include values in column g.
Sheet 1 includes values in column g. There are "Fail" value at the row number of 3, 4, 5.
Sheet 2 includes values in column g. There are "Fail" value at the row number of 6, 7, 8.
The result (return r) becomes below.
[[], [3, 4, 5], [6, 7, 8]]
Sample script 1:
function combined() {
var sheets = SpreadsheetApp.getActiveSpreadsheet().getSheets();
var data =[];
sheets.forEach(function(ss){
try { // In the case of check all sheets, if new sheet is included in the spreadsheet, an error occurs. This ``try...catch`` is used to avoid the error.
data.push(ss.getRange(3, 7, ss.getLastRow(), 1).getValues());
} catch(e) {
data.push([]);
}
});
var r = [];
data.forEach(function(e1, i1){
var temp = [];
e1.forEach(function(e2, i2){
if (e2[0] == "Fail") temp.push(i2 + 3);
});
r.push(temp);
});
return r;
}
If I misunderstand your question, I'm sorry.