Develop Reference

Google Sheets, stack report from multiple workbooks

Goal: To stack data from 90+ google workbooks, all with the same sheet name, into the one master sheet for reporting
Info:
All worksheets have the same number of columns.
I have the following script but it does not run properly, I think the issue is with how I am caching / Pushing the data to the array before pasting to the output sheet.
I am trying to build an array then paste it in one go.
The tables I am stacking have 47 columns, unknown number of rows.
The part that opens the sheets is all working perfectly.
// Get the data from the worksheets
var indexsheet = SpreadsheetApp.getActive().getSheetByName("Index");
var outputsheet = SpreadsheetApp.getActive().getSheetByName("Output");
var response = SpreadsheetApp.getUi().prompt('Current Cycle', 'Enter Cycle Name Exactly in YY-MMM-Cycle# format', SpreadsheetApp.getUi().ButtonSet.OK_CANCEL)
var CurrentCycleName = response.getResponseText()
// Assign datasets to variables
var indexdata = indexsheet.getDataRange().getValues();
// For each workbook in the index sheet, open it and copy the data to a cache
indexdata.forEach(function(row, r) {
try {
//open Entity specific workbook
var workbookid = indexsheet.getRange(r + 1, 7, 1, 1).getValues();
var Entityworkbook = SpreadsheetApp.openById(workbookid)
// Open workhseet
Entitysheet.getSheetByName(CurrentCycleName)
// Add PR Data to cache - stacking for all countrys
var PRDataCache = Entitysheet.getDataRange().push()
} catch {}
})
// Set the all values of the sheet at once
outputsheet.getRange(r + 1, 14).setValue('Issue Splitting Data')
Entitysheet.getRange(2, 1, PRDataCache.length || 1, 47).setValues(PRDataCache)
};
This is the index tab where we are getting the workbookid from to open each file
This is the output file, we are stacking all data from each country

I believe your goal is as follows.
You want to retrieve the Spreadsheet IDs from the column "G" of "Index" sheet.
You want to give the specific sheet name using a dialog.
You want to retrieve all values from the specification sheet in all Spreadsheets. In this case, you want to remove the header row.
You want to put the retrieved values on "Output" sheet.
In this case, how about the following sample script?
Sample script:
function myFunction() {
var ss = SpreadsheetApp.getActive();
var indexsheet = ss.getSheetByName("Index");
var outputsheet = ss.getSheetByName("Output");
var response = SpreadsheetApp.getUi().prompt('Current Cycle', 'Enter Cycle Name Exactly in YY-MMM-Cycle# format', SpreadsheetApp.getUi().ButtonSet.OK_CANCEL);
var CurrentCycleName = response.getResponseText();
var ids = indexsheet.getRange("G1:G" + indexsheet.getLastRow()).getValues();
var values = ids.reduce((ar, [id]) => {
try {
var [, ...values] = SpreadsheetApp.openById(id).getSheetByName(CurrentCycleName).getDataRange().getValues();
ar = [...ar, ...values];
} catch (e) {
console.log(`"${id}" was not found.`);
}
return ar;
}, []);
if (values.length == 0) return;
// If the number of columns is different in all Spreadsheets, please use the following script.
// var maxLen = Math.max(...values.map(r => r.length));
// values = values.map(r => r.length < maxLen ? [...r, ...Array(maxLen - r.length).fill("")] : r);
outputsheet.getRange(outputsheet.getLastRow() + 1, 1, values.length, values[1].length).setValues(values);
}
Note:
When the number of Spreadsheet IDs is large, the processing time might be over 6 minutes. I'm worried about this. At that time, how about separating the Spreadsheet IDs?
Reference:
reduce()

Need script to loop through stock ticker list, copy resulting output, and paste the output of multiple rows to separate sheet within workbook

My question is similar to Need script to loop through stock ticker list, copy resulting output, and paste output to separate sheet within workbook. However, instead of copying and pasting a single row (for each ticker) below the last non-empty row, I need to copy and paste 100 rows (for each ticker) below the last non-empty row.
I have three sheets named Tickers , Data, and Results within a single workbook.
If done manually, the job would be the following: Enter a stock symbol into cell A2 in the Data sheet. The sheet then retrieves the historical time series data from google finance, runs formulas and returns 100 rows and 7 columns of data (Ticker, Date, Open, High, Low, Close, Volume) into range A5:G254 within the same Data sheet. Copy the 100 rows of returned data and paste the data on the Results sheet below the last non-empty row. Then repeat the process with each ticker.
Request: Create a script to loop through a list of 50 stock symbols from the Ticker sheet (range B2:B51), paste 1 symbol at a time into cell A2 in the Data sheet, wait for google finance API to run, and then copy all the resulting rows generated in range A5:G254, and pasting those results into the Results sheet, below the data generated from previous ticker, until there is a "long format" table of time series data of each of the 50 tickers.
The issue with my current script is that it pastes only the first row of the wanted range A5:G254.
function getNewPrices() {
const ss = SpreadsheetApp.getActiveSpreadsheet();
const tickerSheet = ss.getSheetByName('Tickers');
const dataSheet = ss.getSheetByName('Data');
const resultsSheet = ss.getSheetByName('Results');
let data;
let myListOfTickers = tickerSheet.getRange('B2:B51').getValues().filter(String).sort();
myListOfTickers.forEach(ticker => {
dataSheet.getRange('A2').setValue(ticker[0]);
data = dataSheet.getRange('A5:G254').getValues()[0];
pasteData(resultsSheet, data);
});
};
function pasteData(resultsSheet,data){
let nextRow = resultsSheet.getLastRow() + 1;
data.forEach((datum,index) => {
resultsSheet.getRange(nextRow,index + 1,1,1).setValue(datum);
});
};

I'm not completely sure but I think this is what you want
function getNewPrices() {
const ss = SpreadsheetApp.getActive();
const tsh = ss.getSheetByName('Tickers');
let tvs = tsh.getRange('B2:B51').getValues().filter(String).sort();
const dsh = ss.getSheetByName('Data');
const rsh = ss.getSheetByName('Results');
dsh.getRange(2,1,tvs.length,tvs[0].length).setValues(tvs);
let dvs = dsh.getRange('A5:G254').getValues();
rsh.getRange(rsh.getLastRow() + 1, 1, dvs.length, dvs[0].length).setValues(dvs);
}
You can goto to Google Apps Script Reference and using the search box find any function that you don't understand. If it's a pure JavaScript function the go here

Problem solved. Here is the final code if anyone wants to use it later on. Thank you for the initial code Cooper.
function getNewPrices() {
const ss = SpreadsheetApp.getActive();
const tsh = ss.getSheetByName('Tickers');
let tvs = tsh.getRange('B2:B51').getValues().filter(String).sort();
const dsh = ss.getSheetByName('Data');
const rsh = ss.getSheetByName('Results');
tvs.forEach(ticker =>{
dsh.getRange('A2').setValue(ticker[0]);
let dvs = dsh.getRange('A5:G254').getValues();
rsh.getRange(rsh.getLastRow() + 1, 1, dvs.length, dvs[0].length).setValues(dvs);
});
}

Google Apps Script - Compare two sheets for changes by column names instead of hardcoded ranges

I have two sheets, "IMPORT" and "CASES".
In the "IMPORT" sheet, I am importing data from an external source that sometimes have more columns or existing columns are arranged each time differently.
In the "CASES" sheet, this is where I store a weekly snapshot of all last week's imported data, and I add my additional columns with more pieces of information such as comments, next steps etc.
I am looking for a way to compare both sheets without hardcoding any column ranges. I thought the most efficient way to do it is by looking up column header names in both sheets and then checking for changes in reference to the "Case Number" row. Please let me know if you can think of a better way.
I have already managed to write a code to look through headers and identify the Index number for a specific column name, "Case Number".
This column will always be present in both sheets, and it could serve as a reference point to the row that should be validated, but it could be a different row for each sheet at a time.
I will need the same time loop through all the column headers from the CASES sheet and check for updates from the IMPORT sheet.
I only need to check/loop for changes for few specific columns from the CASES sheet. Columns names such: Contact Name, Title, Priority, Status.
I am aiming to achieve 3 possible outcomes:
[ COMPLETED ] "Case Number" from the CASES sheet was NOT FOUND in the IMPORT sheet - that means the case was closed since last week.
Action: Highlight an entire row in the CASES sheet as grey (this will indicate the case is no longer open and should be removed from the list after confirmation).
"Case Number" from the IMPORT sheet was NOT FOUND in the CASES sheet - this means the case is new and needs to be added to the CASES sheet at the bottom.
Action: Copy the data from the IMPORT sheet to the CASES sheet and paste it in the correct columns at the bottom and highlight the entire row as green to indicate a new data entry.
For all non-existing columns in the CASES sheet that are in the IMPORT sheet, those should be skipped.
"Case Number" from the IMPORT sheet WAS FOUND in the CASES sheet - for the matching Case Number records, I need to validate if there were any changes in any CASES sheet columns since last week.
Action: If a change was found in any of the cells, update the cell with new data in CASES sheet and change the cell background colour to yellow to highlight the cell was updated. For cells without changes, skip.
I apologise for the lengthy problem statement.
I am new to JS and GAS, and I wrote it hoping that some JavaScript expert will understand my idea and advise maybe the easier way to complete my project.
Currently, I am stuck with finding a proper way to loop through Header Names then check cell value from the IMPORT sheet and comparing it with the CASES sheet based on the Case Name value/row.
OUTCOME 1 - Completed
OUTCOME 2 - In Progress
OUTCOME 3 - tbd...
I will continue to update this topic to show the latest progress on this project.
All the examples I found so far on the Internet were based on hardcoded ranges of cells and columns. I think my approach is interesting as it gives future-proof flexibility to the datasets.
Please let me know your thoughts or ideas for a more straightforward approach :)
Link to live sheet
UPDATED code:
// Create Top Menu
function onOpen() {
let ui = SpreadsheetApp.getUi();
ui.createMenu('>> REPORTS <<').
addItem('Highlight Closed Cases', 'closedCases').
addItem('Check for new Cases', 'addCases').addToUi();
}
// IN PROGRESS (Outcome 2) - Add and highlight new cases in CASES sheet
function addCases() {
let ss = SpreadsheetApp.getActiveSpreadsheet();
// Get column index number for Case Number
let activeImportCol = getColumnIndex("Case Number", "IMPORT");
let activeCasesCol = getColumnIndex("Case Number", "CASES");
let importHeaders = loadHeaderNames("IMPORT");
let casesHeaders = loadHeaderNames("CASES");
// Load Case Number columns values into array
let loadImportValues = getColumnValues("Case Number", "IMPORT");
let loadCasesValues = getColumnValues("Case Number", "CASES");
// Convert to 1D array
let newImportValues = loadImportValues.map(function (row) { return row[0]; });
let newCasesValues = loadCasesValues.map(function (row) { return row[0]; });
// Get number of columns
var numImportCol = ss.getSheetByName("IMPORT").getLastColumn();
// Loop through IMPORT sheet "Case Number" column to find new Case Numbers - execute OUTCOME 3 or 2
for (var line in newImportValues) {
var isMatched = newCasesValues.indexOf(newImportValues[line]);
if (isMatched !== -1) {
// "Case Number" from the IMPORT sheet WAS FOUND in the CASES sheet - EXECUTE OUTCOME 3
// ****************************************************************************************
// For the matching Case Number records, I need to validate if there were any changes in any CASES sheet columns since last week
// Action: If a change was found in any of the cells, update the cell with new data in CASES sheet
// and change the cell background colour to yellow to highlight the cell was updated. For cells without changes, skip.
} else {
// "Case Number" from the IMPORT sheet was NOT FOUND in the CASES sheet - EXECUTE OUTCOME 2
// ****************************************************************************************
// Copy the new data row from the IMPORT sheet to the CASES sheet and paste it in the correct columns
// at the bottom and highlight the entire row as green to indicate a new data entry.
// For all non-existing/not matching column names in the CASES sheet that are not in IMPORT sheet, those should be skipped.
}
}
}
// COMPLETED (Outcome 1) - Highlight entire row grey for missing values in CASES sheet
function closedCases() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
// Load all Casen Number columns values into array
var importValues = getColumnValues("Case Number", "IMPORT");
var casesValues = getColumnValues("Case Number", "CASES");
// Convert to 1D array
var newImportValues = importValues.map(function (row) { return row[0]; });
var newCasesValues = casesValues.map(function (row) { return row[0]; });
// Get column index number for Case Number
var activeCol = getColumnIndex("Case Number", "CASES");
// Get number of columns
var numCol = ss.getSheetByName("CASES").getLastColumn();
// Loop though CASES "Case Number" column and highlight closed cases (not found in IMPORT tab)
for (var line in newCasesValues) {
var isMatched = newImportValues.indexOf(newCasesValues[line]);
if (isMatched !== -1) {
// If found then...
ss.getSheetByName("CASES").getRange(+line + 2, 1, 1, numCol).setBackground(null);
} else {
// Higlight row with missing cases - grey
ss.getSheetByName("CASES").getRange(+line + 2, 1, 1, numCol).setBackground("#d9d9d9");
};
}
}
// Load column values
function getColumnValues(label, sheetName) {
var ss = SpreadsheetApp.getActiveSpreadsheet().getSheetByName(sheetName);
// Get column number for Case Number
var colIndex = getColumnIndex(label, sheetName);
// Get number of rows in Case Number
var numRows = ss.getLastRow() - 1;
// Load Case Number values into array
var colValues = ss.getRange(2, colIndex, numRows, 1).getValues();
return colValues;
}
// Load column header names
function loadHeaderNames(sheetName) {
var ss = SpreadsheetApp.getActiveSpreadsheet().getSheetByName(sheetName);
let HeaderArray = ss.getRange(1, 1, 1, ss.getLastColumn()).getValues()[0];
let colidx = {};
HeaderArray.forEach((h, i) => colidx[h] = i);
return HeaderArray;
}
// Get column name index value
function getColumnIndex(label, sheetName) {
var ss = SpreadsheetApp.getActiveSpreadsheet().getSheetByName(sheetName);
// Find last column
var lc = ss.getLastColumn();
// Load headers into array
var lookupRangeValues = ss.getRange(1, 1, 1, lc).getValues()[0];
// Search for label and return the column number
var index = lookupRangeValues.indexOf(label) + 1;
return index;
}

One way to make all this processing much easier is to reorder the columns so that they always fall in the same place, like this:
=arrayformula(
iferror(
vlookup(
hlookup("Case Number"; IMPORT!A1:G; row(IMPORT!A2:G); false);
{
hlookup("Case Number"; IMPORT!A1:G; row(IMPORT!A1:G); false) \
IMPORT!A1:G
};
match(IMPORT!A1:G1; CASES!A1:G1; 0) + 1;
false
)
)
)
The formula will reorder the columns in IMPORT so that the columns are in the same order as they are listed in CASES!A1:G1.
You can then use further formulas or script functions to work on the data, confident that a particular kind of data will always be in the same column. For instance, you can list closed cases with something like this:
=filter( 'CASES normalized'!A2:G; isna(match('CASES normalized'!C2:C; 'IMPORT normalized'!C2:C; 0)) )
...and open cases like this:
=filter( 'CASES normalized'!A2:G; match('CASES normalized'!C2:C; 'IMPORT normalized'!C2:C; 0) )
See your sample spreadsheet.

Move Specific Rows depending on Filtering Keywords within unknown amount of rows using Google Sheets Apps Scripts

I do SEO, and therefore I have a lot of keywords flowing around in different spreadsheets. I'd like a way to filter these into seperate sheets based on specific filters, but I can't for the life of me, figure out how to do this in Google Apps Script.
Criteria I set myself for this to work out:
A list of strings and their corresponding volumes are entered in column 1+2.
A list of filter-words are written in column 3.
The script has to create a new sheet for each of the filter words and move the strings + volumes into these different sheets if the string contains a filter word.
Example:
Filter words: Apple, Banana, Pineapple
String: "The Apple Was Big", Volume: "100"
The script would move the string and volume into the sheet called "Apple" on row 1
(Beware, I'm in no means experienced in coding)
I believe you can use the following structure:
for(let i = 0; i <= column3RowAmount; i++){ //Run as long as there are more filter words
create(column3Row[i]); //create a new sheet with the name of the filter word
for(let j = 0; j <= column1RowAmount; j++){ //Run as long as there are more keywords
if(column1Row[j].indexOf(column3Row[i]) >= 0){ //If the Row in column 1 contains the filter word
column1Row[j].moveToSheet(column3Row[i]); // Make sure not to move Column 3, but only 1+2
}
}
}
Example sheet: https://docs.google.com/spreadsheets/d/15YIMyGmmfZdy094gwuJNxFmTd8h7NOLnA8KevZrGtdU/edit?usp=sharing

Explanation:
Your goal is to create a sheet for every filter-word in column C. Then copy the data in columns A, B but only the rows that include the filter-word to the corresponding sheet.
For starters, you need to get the filter-word list. You can get the full range of column C and filter out the empty cells:
const sh_names = sh.getRange('C1:C').getValues().flat().filter(r=>r!='');
Similarly, you need to get the data in columns A and B:
const data = sh.getRange('A1:B'+sh.getLastRow()).getValues();
The next step is to iterate over sh_names and for every element / filter-word, check if a sheet with that name exists. If it does not exist, then create a sheet with that name, if it exists then skip the creation part:
if(!ss.getSheetByName(s)){
ss.insertSheet().setName(s);}
The next step is to filter data on the rows that include the filter-word:
let f_data = data.filter(r=>r[0].includes(s));
Finally, check if the length of the data is bigger than 0, otherwise there is not data to use and set the values of data to the corresponding sheet:
sheet.getRange(sheet.getLastRow()+1,1,f_data.length,f_data[0].length).setValues(f_data)
Solution
function myFunction() {
const ss = SpreadsheetApp.getActive();
const sh = ss.getSheetByName('Ark1');
const filter_sh = ss.getSheetByName('Filter');
const data = sh.getRange('A1:B'+sh.getLastRow()).getValues();
const sh_names = filter_sh.getRange('A1:A'+filter_sh.getLastRow()).getValues().flat();
sh_names.forEach(s=>{
if(!ss.getSheetByName(s)){
ss.insertSheet().setName(s);}
let sheet = ss.getSheetByName(s);
let f_data = data.filter(r=>r[0].includes(s));
if(f_data.length>0){
sheet.getRange(sheet.getLastRow()+1,1,f_data.length,f_data[0].length).setValues(f_data);}
});
}

This function will place all of your results into column 4 next to the appropriate word rather than creating a page for each word. So it runs much faster.
function stringswords() {
const ss=SpreadsheetApp.getActive();
const sh=ss.getSheetByName('Sheet1');
const sr=2;
const rgd=sh.getRange(sr,1,sh.getLastRow()-sr+1,2);
const data=rgd.getDisplayValues();
const rgw=sh.getRange(sr,3,sh.getLastRow()-sr+1,1);
const words=rgw.getDisplayValues().flat();
const wiObj={};
words.forEach(function(w,i){wiObj[w]=i});
const rgr=sh.getRange(sr,4,sh.getLastRow()-sr+1,1);
rgr.clearContent();
var results=rgr.getValues();
words.forEach(function(w,i,A){
data.forEach(function(r,j,D) {
if(data[j][0] && data[j][0].indexOf(w)!=-1) {
results[wiObj[w]][0]+=Utilities.formatString('String:%s Vol:%s\n',data[j][0],data[j][1]);
}
});
});
rgr.setValues(results);
}
Image of Data and output:

How to compare two sheets and delete/add any column with a distinct value in row 1? Google Script

I want to compare two sheets (based on header values in row 1) and delete any column with a unique value (without a match). For example, Assuming Sheet1, Row 1 data and Sheet 2, Row 1 are uniform, if a user adds/deletes a column within any sheet, I want to always match the number of columns in both sheets with their values
Screenshots of sheets headings.
IF both sheets looks like this
And a user adds a new Column N
Or delete column N
How can I ensure that both sheet matches by deleting the odd/distinct column in Sheet 1?
I have tried modifying this code below but I can't just get the unique one out. This code only look for headers with a defined value.
function deleteAloneColumns(){
var sheet = SpreadsheetApp.getActiveSheet();
var lastColumnPos = sheet.getLastColumn();
var headers = sheet.getRange( 1 ,1, 1, lastColumnPos ).getValues()[0];
for( var i = lastColumnPos ; i < 1; i--){
if( headers[i] === "alone" ) sheet.deleteColumn(i);
}
SpreadsheetApp.getUi().alert( 'Job done!' );
}
Any help to compare and delete the column with the unique value will be appreciated.

Problem
Balancing sheets based on header row values mismatch.
Solution
If I understood you correctly, you have a source sheet against which validation is run and two primary use cases: user adds a new column named differently than any other column (if you want to check that the column strictly matches the one in sheet1, it is easy to modify) in source sheet or deletes one that should be there.
const balanceSheets = (sourceShName = 'Sheet1',targetShName = 'Sheet2') => {
const ss = SpreadsheetApp.getActiveSpreadsheet();
const s1 = ss.getSheetByName(sourceShName);
const s2 = ss.getSheetByName(targetShName);
const s2lcol = s2.getLastColumn();
//keep all vals from source to reduce I/O
const s1DataVals = s1.getDataRange().getValues();
const s2Vals = s2.getRange(1, 1, 1, s2lcol).getValues();
const h1Vals = s1DataVals[0];
const h2Vals = s2Vals[0];
//assume s1 is source (validation) sheet
//assume s2 is target sheet that a user can edit
//case 1: target has value not present in source -> delete column in target
let colIdx = 0;
h2Vals.forEach(value => {
const isOK = h1Vals.some(val => val===value);
isOK ? colIdx++ : s2.deleteColumn(colIdx+1);
});
//case 2: target does not have values present in source -> append column from source
h1Vals.forEach((value,index) => {
const isOK = h2Vals.some(val => val===value);
!isOK && s2.insertColumnAfter(index);
const valuesToInsert = s1DataVals.map(row => [row[index]]);
const numRowsToInsert = valuesToInsert.length;
s2.getRange(1,index+1, numRowsToInsert,1).setValues(valuesToInsert);
});
};
Showcase
Here is a small demo of how it works as a macros:
Notes
Solving your problem with two forEach is suboptimal, but I kept number of I/O low (it can be lowered further by, for example, moving deleteColum out of the loop while only keeping track of column indices).
The script uses ES6 capabilities provided by V8, so please, be careful (although I would recommend migrating as soon as possible - even if you encounter bugs / inconsistencies , it is worth more than it costs.
UPD made script more flexible by moving sheet names to parameter list.
UPD2 after discussing the issue with deleteColumn() behaviour, the answer is updated to keep column pointer in bounds (for those curious about it - forEach kept incrementing the index, while deleteColumn reduced bounds for any given index).
Reference
insertColumnAfter() method reference