Develop Reference

Replacing Text with replacetext() and defining said replacement as heading

I have a Google spreadsheet and a Google document. The document is a report which gets filled by the spreadsheet. The spreadsheet is also defining what comes into the report. Therefore I have a script, which gathers a bunch of placeholders depending on values in the the document.
After all the placeholders have been inserted in the document (there are a couple of pages before that) it looks kind of like this:
{{header1.1}}
{{text1.1}}//this is already a couple lines of text
{{table1.1}}
{{table.dir}}
{{blob1.1}}
{{blob.dir}}
I already have a script, which inserts all the text parts and I have set up a script, which should be capable of writing the tables at the correct position. So far I can replace the {{header1.1}}, but if I try to define it as a heading it works, but everything after the header1.1 is also a heading
I've been at this problem for quite a while and didn't get and its always one step forward one step back. Also this is my first question after a couple of years just reading on stackoverflow. I'd appreciate if someone could help.
function myUeberschriftenboi() {
doc = DocumentApp.openById('someID');
console.log(doc.getName());
var body = doc.getBody();
//formate
const plain3style = {};
plain3style[DocumentApp.Attribute.HEADING] = DocumentApp.ParagraphHeading.HEADING3;
var lvl2array = [ "{{header1.1}}" , "{{header1.2}}" ];
var fill2array = [ "Energy" , "Energyflow" ]
var lvl2count = 1;
for( var j = 0 ; j < lvl2array.length ; j++)
{
var seek = body.findText(lvl2array[j]);
if( seek != null)
{
body.replaceText(lvl2array[j] , "1.1."+lvl2count+" "+fill2array[j]+"\n");
var seek2 = body.findText("1."+lvl2count+" "+fill2array[j]);
seek2.getElement().getParent().getChild().setAttributes(plain3style);
lvl2count++;
}}}

Grab data from website HTML table and transfer to Google Sheets using App-Script

Ok, I know there are similar questions out there to mine, but so far I have yet to find any answers that work for me. What I am trying to do is gather data from an entire HTML table on the web (https://www.sports-reference.com/cbb/schools/indiana/2022-gamelogs.html) and then parse it/transfer it to a range in my Google Sheet. The code below is probably the closest thing I've found so far because at least it doesn't error out, but it will only find one string or value, not the whole table. I've found other answers where they use xmlservice.parse, however that doesn't work for me, I believe because the HTML format has issues that it can't parse. Does anyone have an idea of how to edit what I have below, or a whole new idea that may work for this website?
function SAMPLE() {
const url="http://www.sports-reference.com/cbb/schools/indiana/2022-gamelogs.html#sgl-basic?"
// Get all the static HTML text of the website
const res = UrlFetchApp.fetch(url, {muteHttpExceptions: true}).getContentText();
// Find the index of the string of the parameter we are searching for
index = res.search("td class");
// create a substring to only get the right number values ignoring all the HTML tags and classes
sub = res.substring(index+92,index+102);
Logger.log(sub);
return sub;
}
I understand that I can use importHTML natively in a Google Sheet, and that's what I'm currently doing. However I am doing this for over 350 webpage tables, and iterating through each one to load it and then copy the value to another sheet. App Script bogs down quite a bit when it is repeatedly waiting on Sheets to load an importHTMl and then grab some data and do it all over again on another url. I apologize for any formatting issues in this post or things I've done wrong, this is my first time posting here.
Edit: ok, I've found a method that works, but it's still much slower than I would like, because it is using Drive API to create a document with the HTML data and then parse and create an array from there. The Drive.Files.Insert line is the most time consuming part. Anyone have an idea of how to make this quicker? It may not seem that slow to you right now, but when I need to do this 350 times, it adds up.
function parseTablesFromHTML() {
var html = UrlFetchApp.fetch("https://www.sports-reference.com/cbb/schools/indiana/2022-gamelogs.html");
var docId = Drive.Files.insert(
{ title: "temporalDocument", mimeType: MimeType.GOOGLE_DOCS },
html.getBlob()
).id;
var tables = DocumentApp.openById(docId)
.getBody()
.getTables();
var res = tables.map(function(table) {
var values = [];
for (var row = 0; row < table.getNumRows(); row++) {
var temp = [];
var cols = table.getRow(row);
for (var col = 0; col < cols.getNumCells(); col++) {
temp.push(cols.getCell(col).getText());
}
values.push(temp);
}
return values;
});
Drive.Files.remove(docId);
var range=SpreadsheetApp.getActive().getSheetByName("Test").getRange(3,6,res[0].length,res[0][0].length);
range.setValues(res[0]);
SpreadsheetApp.flush();
}

Solution by formula
Try
=importhtml(url,"table",1)
Other solution by script
function importTableHTML() {
var url = 'https://www.sports-reference.com/cbb/schools/indiana/2022-gamelogs.html'
var html = '<table' + UrlFetchApp.fetch(url, {muteHttpExceptions: true}).getContentText().replace(/(\r\n|\n|\r|\t| )/gm,"").match(/(?<=\<table).*(?=\<\/table)/g) + '</table>';
var trs = [...html.matchAll(/<tr[\s\S\w]+?<\/tr>/g)];
var data = [];
for (var i=0;i<trs.length;i++){
var tds = [...trs[i][0].matchAll(/<(td|th)[\s\S\w]+?<\/(td|th)>/g)];
var prov = [];
for (var j=0;j<tds.length;j++){
donnee=tds[j][0].match(/(?<=\>).*(?=\<\/)/g)[0];
prov.push(stripTags(donnee));
}
data.push(prov);
}
return(data);
}
function stripTags(body) {
var regex = /(<([^>]+)>)/ig;
return body.replace(regex,"");
}

Google sheets - Optimisation of function to create notes in a range (very slow)

I have a script to create Notes on cells based on their value, but the process is very slow and my sheet has 15000 rows. Is it possible to reduce the delay by optimizing the script ?
PS : I use spreadsheet with french parameters.
function InsertCellsNotes(){
var plage = SpreadsheetApp.getActiveSpreadsheet().getSelection().getActiveRange();
var Notes = plage.getValues();
var NB_lines = Notes.length;
for (var i=1; i<NB_lines+1; i++){ // ajouter +1 !
var myCell = plage.getCell(i, 1);
var cellValue = Notes[i-1];
if (cellValue == "#N/A" || ""){ }
else { myCell.setNote(cellValue); }
}
}
An example of the sheet : https://docs.google.com/spreadsheets/d/1lu7dEoyO2NDHV4phXeh8DAAkbBuQG5EQWwMA6SJDP1A/edit?usp=sharing

Explanation:
Two tricks:
Avoid unnecessary api calls when possible. You are iteratively using methods that interact with the spreadsheet file and that causes extreme delays. Read best practices.
When you use null as an argument for setNote, no note is set. We can take advantage of this and construct an array by using the map method. Namely, if the value is #N/A or blank "", assign null to the element, otherwise take the value of the cell:
var notes = rng.getValues().flat().map(v=>[v=="#N/A" || ""?null:v]);
This will allow you to get rid of the for loops but also create an array that can directly be used in the setNotes function: rng.setNotes(notes);
Solution - active range:
Select (with your mouse) a particular range and insert notes (depending on the condition):
function InsertCellsNotes(){
var rng = SpreadsheetApp.getActiveRange();
var notes = rng.getValues().flat().map(v=>[v=="#N/A" || ""?null:v]);
rng.setNotes(notes);
}
Solution - predefined range:
This is a more static approach. You define a particular sheet "Sheet1" and for all the cells in column B (until the last row with content of the sheet) you insert notes (depending on the condition):
function InsertCellsNotes(){
var plage = SpreadsheetApp.getActive().getSheetByName("Sheet1");
var rng = plage.getRange(1,2,plage.getLastRow(),1);
var notes = rng.getValues().flat().map(v=>[v=="#N/A" || ""?null:v]);
rng.setNotes(notes);
}

Long processing time likely due to getValue and cell inserts

I've just written my first google apps scripts, ported from VBA, which formats a column of customer order information (thanks to you all of your direction).
Description:
The code identifies state codes by their - prefix, then combines the following first name with a last name (if it exists). It then writes "Order complete" where the last name would have been. Finally, it inserts a necessary blank cell if there is no gap between the orders (see image below).
Problem:
The issue is processing time. It cannot handle longer columns of data. I am warned that
Method Range.getValue is heavily used by the script.
Existing Optimizations:
Per the responses to this question, I've tried to keep as many variables outside the loop as possible, and also improved my if statements. #MuhammadGelbana suggests calling the Range.getValue method just once and moving around with its value...but I don't understand how this would/could work.
Code:
function format() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var s = ss.getActiveSheet();
var lastRow = s.getRange("A:A").getLastRow();
var row, range1, cellValue, dash, offset1, offset2, offset3;
//loop through all cells in column A
for (row = 0; row < lastRow; row++) {
range1 = s.getRange(row + 1, 1);
//if cell substring is number, skip it
//because substring cannot process numbers
cellValue = range1.getValue();
if (typeof cellValue === 'number') {continue;};
dash = cellValue.substring(0, 1);
offset1 = range1.offset(1, 0).getValue();
offset2 = range1.offset(2, 0).getValue();
offset3 = range1.offset(3, 0).getValue();
//if -, then merge offset cells 1 and 2
//and enter "Order complete" in offset cell 2.
if (dash === "-") {
range1.offset(1, 0).setValue(offset1 + " " + offset2);
//Translate
range1.offset(2, 0).setValue("Order complete");
};
//The real slow part...
//if - and offset 3 is not blank, then INSERT CELL
if (dash === "-" && offset3) {
//select from three rows down to last
//move selection one more row down (down 4 rows total)
s.getRange(row + 1, 1, lastRow).offset(3, 0).moveTo(range1.offset(4, 0));
};
};
}
Formatting Update:
For guidance on formatting the output with font or background colors, check this follow-up question here. Hopefully you can benefit from the advice these pros gave me :)

Issue:
Usage of .getValue() and .setValue() in a loop resulting in increased processing time.
Documentation excerpts:
Minimize calls to services:
Anything you can accomplish within Google Apps Script itself will be much faster than making calls that need to fetch data from Google's servers or an external server, such as requests to Spreadsheets, Docs, Sites, Translate, UrlFetch, and so on.
Look ahead caching:
Google Apps Script already has some built-in optimization, such as using look-ahead caching to retrieve what a script is likely to get and write caching to save what is likely to be set.
Minimize "number" of read/writes:
You can write scripts to take maximum advantage of the built-in caching, by minimizing the number of reads and writes.
Avoid alternating read/write:
Alternating read and write commands is slow
Use arrays:
To speed up a script, read all data into an array with one command, perform any operations on the data in the array, and write the data out with one command.
Slow script example:
/**
* Really Slow script example
* Get values from A1:D2
* Set values to A3:D4
*/
function slowScriptLikeVBA(){
const ss = SpreadsheetApp.getActive();
const sh = ss.getActiveSheet();
//get A1:D2 and set it 2 rows down
for(var row = 1; row <= 2; row++){
for(var col = 1; col <= 4; col++){
var sourceCellRange = sh.getRange(row, col, 1, 1);
var targetCellRange = sh.getRange(row + 2, col, 1, 1);
var sourceCellValue = sourceCellRange.getValue();//1 read call per loop
targetCellRange.setValue(sourceCellValue);//1 write call per loop
}
}
}
Notice that two calls are made per loop(Spreadsheet ss, Sheet sh and range calls are excluded. Only including the expensive get/set value calls). There are two loops; 8 read calls and 8 write calls are made in this example for a simple copy paste of 2x4 array.
In addition, Notice that read and write calls alternated making "look-ahead" caching ineffective.
Total calls to services: 16
Time taken: ~5+ seconds
Fast script example:
/**
* Fast script example
* Get values from A1:D2
* Set values to A3:D4
*/
function fastScript(){
const ss = SpreadsheetApp.getActive();
const sh = ss.getActiveSheet();
//get A1:D2 and set it 2 rows down
var sourceRange = sh.getRange("A1:D2");
var targetRange = sh.getRange("A3:D4");
var sourceValues = sourceRange.getValues();//1 read call in total
//modify `sourceValues` if needed
//sourceValues looks like this two dimensional array:
//[//outer array containing rows array
// ["A1","B1","C1",D1], //row1(inner) array containing column element values
// ["A2","B2","C2",D2],
//]
//#see https://stackoverflow.com/questions/63720612
targetRange.setValues(sourceValues);//1 write call in total
}
Total calls to services: 2
Time taken: ~0.2 seconds
References:
Best practices
What does the range method getValues() return and setValues() accept?

Using methods like .getValue() and .moveTo() can be very expensive on execution time. An alternative approach is to use a batch operation where you get all the column values and iterate across the data reshaping as required before writing to the sheet in one call. When you run your script you may have noticed the following warning:
The script uses a method which is considered expensive. Each
invocation generates a time consuming call to a remote server. That
may have critical impact on the execution time of the script,
especially on large data. If performance is an issue for the script,
you should consider using another method, e.g. Range.getValues().
Using .getValues() and .setValues() your script can be rewritten as:
function format() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var s = ss.getActiveSheet();
var lastRow = s.getLastRow(); // more efficient way to get last row
var row;
var data = s.getRange("A:A").getValues(); // gets a [][] of all values in the column
var output = []; // we are going to build a [][] to output result
//loop through all cells in column A
for (row = 0; row < lastRow; row++) {
var cellValue = data[row][0];
var dash = false;
if (typeof cellValue === 'string') {
dash = cellValue.substring(0, 1);
} else { // if a number copy to our output array
output.push([cellValue]);
}
// if a dash
if (dash === "-") {
var name = (data[(row+1)][0]+" "+data[(row+2)][0]).trim(); // build name
output.push([cellValue]); // add row -state
output.push([name]); // add row name
output.push(["Order complete"]); // row order complete
output.push([""]); // add blank row
row++; // jump an extra row to speed things up
}
}
s.clear(); // clear all existing data on sheet
// if you need other data in sheet then could
// s.deleteColumn(1);
// s.insertColumns(1);
// set the values we've made in our output [][] array
s.getRange(1, 1, output.length).setValues(output);
}
Testing your script with 20 rows of data revealed it took 4.415 seconds to execute, the above code completes in 0.019 seconds

jQuery .prependTo() created table causing browser to "lag"

I have a problem I am not able to fix, quite difficult to explain but I will do my best.
Basically I have created a web application (in CodeIgniter) that takes some data of an array coming from a json encoding and adds rows to a table by using the jQuery .prependTo() method in the success function.
Every row of the table contains different elements from the database (coming from the json), depending from the value of the i counter.
The code is as the following (i cut the part about the <tr> and <td> content, it's just styling)
$.ajax({
type: "get",
async: false,
....
....
success: function(data) {
var i;
var item_cost = new Array();
var item_name = new Array();
var item_code = new Array();
var item_interno = new Array();
for(i = 0; i < data.length; i++) {
item_cost[i] = data[i].cost;
item_name[i] = data[i].name;
item_code[i] = data[i].code;
item_interno[i] = data[i].codiceinterno;
var newTr = // creates the <tr>
newTr.html('//creates the <td>')
newTr.prependTo("#myTable");
}
},
I am sorry if it is a bit unclear, if you need me to update the code because I missed something important let me know and I will do it, but this code alone should explain my problem.
The application works beautifully if in the database there are just a little number of rows (for example, i = 300). They are showed and rendered correctly and I don't get any browser slowing process. But when I work with i = 4000 rows, the browser starts acting slow just like the Javascript code is too heavy to render, and i get "lag" while trying to scroll down the html table, or inputting some values in the input boxes inside the table (to later update it by clicking a button). This is my problem: I am quite sure I'm doing something wrong that is loading up too much memory, as I tested this also on very strong computers. Even totally disabling my CSS won't do the trick.
Thanks for any help you can give me, it would be really appreciated.

The problem
You are using a lot of function call inside a loop. That take a lot of juice and the more item you have, the slower it is.
To solve that, we need to reduce the number of function calls.
My suggestion
Working with native JavaScript would save on performance here. So I suggestion you use string concatenation instead of DOM manipulation methods of jQuery.
Let rework you loop. Since you want your data in a descendant order, we need to reverse the loop :
for(i = data.length; i >= 0; i--)
Then simple string concatenation to build a tr. For that you need a var outside the loop:
var myHTML = ''; //That's the one!
var i;
var item_cost = new Array();
var item_name = new Array();
var item_code = new Array();
var item_interno = new Array();
And build the tr with +=. Although, using a single line would make it faster, but less readable :
for(i = data.length; i >= 0; i--) {
item_cost[i] = data[i].cost;
item_name[i] = data[i].name;
item_code[i] = data[i].code;
item_interno[i] = data[i].codiceinterno;
myHTML += '<tr>';
myHTML += '<td>'+yourData+'</td>';//add those data here
myHTML += '<td>'+yourData+'</td>';
myHTML += '<td>'+yourData+'</td>';
myHTML += '<td>'+yourData+'</td>';
myHTML += '<td>'+yourData+'</td>';
myHTML += '</tr>';
}
Then, prepend it to your table :
var myHTML = '';
var i;
var item_cost = new Array();
var item_name = new Array();
var item_code = new Array();
var item_interno = new Array();
for(i = data.length; i >= 0; i--) {
item_cost[i] = data[i].cost;
item_name[i] = data[i].name;
item_code[i] = data[i].code;
item_interno[i] = data[i].codiceinterno;
myHTML += '<tr>';
myHTML += '<td>'+yourData+'</td>';
myHTML += '<td>'+yourData+'</td>';
myHTML += '<td>'+yourData+'</td>';
myHTML += '<td>'+yourData+'</td>';
myHTML += '<td>'+yourData+'</td>';
myHTML += '</tr>';
}
$('#myTable').prepend(myHTML);
Limitation
From my test, the string length can be 2^28 but cannot be 2^29. That make a maximum length of approx. 268,435,456 (approx. might not be the best word here since it's between 268,435,456 and 536,870,912.
If you data character count is higher than that (but let be honnest, that would be a lot of data), you might have to split you string into 2 variables.

I know this is not a real answer to your question, but...
jQuery definitely causes the lag - no wonder. But - whether you choose jQuery to build the table or you just concatenate HTML - let's agree on that: 4000 rows is definitely a lot of data. Too much data? I would say: 200 already is. If we go thousands, the question pops up: why? Is there any application-related reason you really need to retrieve such a big number of records at once? Some of them will probably never be read.
Why not try an ajax-based lazy load approach? Loading 50-record portions every time would seem more robust IMO, and would definitely be a better UX.

My two cents: Why don't retrieve the rows composed from the server instead of making the client work to compound it? You could even cache it. The only thing would be that you'll have more traffic, but in the browser it'll go smoother
UPDATE
To achieve it, you can make your method to distinguish between ajax input or not, and show only $tbody, or the whole table according to it. It would be something like:
$tbody = $this->load->view( 'view_with_oly_tbody', $data, true );
if ( $this->input->is_ajax_request() ) {
return $tbody;
} else {
$data['tbody'] = $tbody;
$this->load->view('your_view_with_table_with_tbody_as_var', $data);
}
Note: Code above will vary according to your views, controller, if you have json in your AJAX or not, write the return in other part, etc. Code above is just for clarify my point with CI.
Ah! and you'll have to manage the answer in the client to append/prepend/substitute the body of the table.