I have written a Google Sheet application for myself that (among other things) has a schedule for consultants to work on projects. I have written a custom function that displays summary information from the calendar; for example:
Tubb, Ernest|Project 1
Tubb, Ernest|Project 2
Perkins, Carl|Project 1
I wrote the function because the mess of FILTER() functions I was using was incomprehensible; and the javascript required to do the same thing is relatively straightforward.
What I find however, is that when the underlying data changes, the cells written by my function are NOT recalculated the way they were with the FILTER() function. How can I program some sort of listener that makes my function's output be refreshed when the underlying data changes? Do I have to do this by hand with the onEdit() function?
And here is my function:
// Global Constants - these will change each quarter based on the calendar structure.
var CALENDAR_QUARTER_START = "F"
var CALENDAR_QUARTER_END = "CS"
// Generate a nice 2-column display of Consultant and Project(s) they are scheduled on,
// based on the consultant names and calendar codes in the Calendar tab.
// "billable" parameter specifies the return of only uppercase activities (true) or all activities (false)
function fortifyConsultantsAndProjects( billable ){
// Resolve arguments
var billable = arguments[0];
if (arguments[0] == undefined) billable = true;
var calendar = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Calendar");
var consultants = calendar.getRange( "A8:A" ).getValues();
var ret = [];
var row = 0;
for ( var i=0; i<consultants.length; i++ ){
var consultant = consultants[i].toString();
var projects = fortifyGetProjectsForConsultant( consultant, billable );
for (var j=0; j < projects.length; j++ ) {
ret.push( [] );
ret[row][0] = consultant;
ret[row][1] = projects[j];
row++;
}
}
return ret;
}
function fortifyGetProjectsForConsultant( consultant, billable ){
// Resolve arguments
var consultant = arguments[0];
if (arguments[0] == undefined) consultant = "Held, Doug";
var billable = arguments[1];
if (arguments[1] == undefined) billable = true;
// Get the range of consultants defined in Column A of Calendar tab.
var calendar = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Calendar");
var consultants = calendar.getRange( "A8:A" ).getValues();
// Identify which Calendar row contains the specified consultant's schedule.
var row;
for ( var i=0; i< consultants.length; i++){
//Browser.msgBox( "compare " + consultant + " and " + consultants[i] );
if ( consultants[i].toString() == consultant ){
row = i+8;
break;
}
}
// Obtain all the consultant's schedulings. Contains many duplicates, so sort.
var projects = calendar.getRange( CALENDAR_QUARTER_START + row + ":" + CALENDAR_QUARTER_END + row + "" ).getValues()[0].sort();
// Iterate through the sorted project codes, removing duplicates and blanks
var ret = [];
var row = 0;
var prev = "";
for ( var i = 0; i< projects.length; i++ ){
var temp = projects[i].toString();
if (temp != "" && temp != prev ){
// Resolve whether to return each row based on project billability (uppercase)
if ( temp.toUpperCase() == temp || billable == false ){
ret[row] = temp;
row++;
}
prev = temp;
}
}
return ret;
}
Well since nobody ever answered, I will explain how I solved the problem.
Instead of a zero-argument function parameter list, I receive all of the ranges (for instance the range Calendar!A8:A that you see hard coded here). Then any time data in the parameter ranges changes, the function is invoked again. This does occasionally cause some sort of runtime error in Google sheets - on update, I sometimes get "the document is not available" and an instruction to reload the sheet.
When a range is received as a parameter, that data is received byvalue as a 2 dimensional array of cell values -- the same as the array returned from Range.getValues().
Related
Essentially there are around ~200 in this department. We are looking to create squads of 4, where each squad must be from a different team, located in the same city, one must be a driver (and the other 3 therefore should not be drivers), and all four people on the team should have similar preferences. These criteria are captured via a Google Form so ultimately we will get the relevant data we need to build out the conditions. My only challenge currently is that I am not well-versed in Apps Script so will need high level guidance on how to approach it.
Currently, I am thinking of storing each column in their own respective arrays and then using a For loop and conditional statements within that to try to satisfy all the requirements. For example, the first conditional statement would be whether the person drives, and if they do then put them into this new array (which will be defined). Next, check the next response and if that person doesnt drive, and has same same preferences, is located in same city, and is on a different team, then include them into that array. Otherwise, move onto the next person and check for those requirements. Once the group hits 4 people, set the values into the Google Sheet somewhere and then search for the next driver in the list and build another group using same logic.
Alternative approach might be to first sort people into 2 groups (based on the two cities), then within each group sort folks based on driver and add in the other criterias as well.
Regardless of which approach I take in terms of algorithm, would first storing each column into an array make the most sense, then managing the array values using various conditional statements.
Here is what I have so far:
function matchUsers() {
//store each column into arrays for use later on
var responseSheet = SpreadsheetApp.getActive().getSheetByName("Responses");
var Alast = responseSheet.getRange("A2:A").getValues().filter(String).length;
Logger.log(Alast);
var emails = responseSheet.getRange("A2:A").getValues();
var userEmails = responseSheet.getRange(2,1, Alast,1).getValues();
var tmName = responseSheet.getRange(2,2, Alast,1).getValues();
var team = responseSheet.getRange(2,3, Alast,1).getValues();
var city = responseSheet.getRange(2,4, Alast,1).getValues();
var firstChoice = responseSheet.getRange(2,5, Alast,1).getValues();
var secChoice = responseSheet.getRange(2,6, Alast,1).getValues();
var thirdChoice = responseSheet.getRange(2,7, Alast,1).getValues();
var driveStatus = responseSheet.getRange(2,8, Alast,1).getValues();
var arrayResponses = [userEmails,tmName,team,city,firstChoice,secChoice,thirdChoice, driveStatus];
//clear sheets
SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Output").getRange(2, 1,20,50).clearContent();
SpreadsheetApp.getActiveSpreadsheet().getSheetByName("CitySort").getRange(2, 1,20,50).clearContent();
//prepare arrays
var edmTMs = [];
var calTMs = [];
var edmDrivers = [];
var calDrivers = [];
//loop through each row and sort team members by respective city and driving capability
for (let i = 0; i < Alast;i+=1){
if (arrayResponses[3][i] == "Edmonton" && arrayResponses[7][i] == "Yes"){
edmDrivers.push(arrayResponses[1][i]);
}else if (arrayResponses[3][i] == "Edmonton" && arrayResponses[7][i] == "No"){
edmTMs.push(arrayResponses[1][i]);
}else if (arrayResponses[3][i] == "Calgary" && arrayResponses[7][i] == "Yes"){
calDrivers.push(arrayResponses[1][i]);
}else if (arrayResponses[3][i] == "Calgary" && arrayResponses[7][i] == "No"){
calTMs.push(arrayResponses[1][i]);
}
}
Logger.log("\nEdmonton: "+edmTMs + "\nCalgary: "+calTMs );
var range = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("CitySort").getRange(2,1,edmTMs.length,1);
range.setValues(edmTMs);
var range = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("CitySort").getRange(2,2,calTMs.length,1);
range.setValues(calTMs);
var range = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("CitySort").getRange(2,3,edmDrivers.length,1);
range.setValues(edmDrivers);
var range = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("CitySort").getRange(2,4,calDrivers.length,1);
range.setValues(calDrivers);
}
Let's see if this fits your needs. I'm pretty sure there are more elegant and efficient solutions than this one, so I appreciate any suggestion.
I tested the script with 34 users, and I got 8 teams of 4 and 2 teams of 1:
Don't mind the repeated names, they are all different rows with different Cities, Teams and Choices. I added a letter to some of them to differentiate.
The script works as follows:
It gets each user row and compares the City, Teams and Choices with all the other 33. It also discards the users that are already in a team. The priority is City > Team > Choice 1 > Choice 2 > Choice 3.
It will also try to force teams of 4 members, so in case there is a team of 3, it will ignore the Teams to try to find the 4th, but it still will respect the Choices and Cities. You can disable this by setting the boolean ignoreTeams to false. In my test, only 3 groups have 1 repeated team.
The script will never put two users from different cities in the same group.
function main() {
var sprsheet = SpreadsheetApp.getActiveSpreadsheet();
var sheet_1 = sprsheet.getSheetByName("Members");
var sheet_2 = sprsheet.getSheetByName("Output");
var number = sheet_1.getLastRow() - 1;
var members = []; //Each row of Members will be stored here
getMembers(sheet_1, members, number);
orderMembers(sheet_2, members);
}
function getMembers(sheet_1, members, number){
for (var i = 0; i < number; i++){
var values = sheet_1.getRange(i+2, 1, 1, 8).getValues();
members.push(new Person(values[0][0],values[0][1],values[0][2],values[0][3],values[0][4],values[0][5],values[0][6],values[0][7]));
}
}
function orderMembers(sheet_2, members){
var col = 1;
var counter = 0; //Number of Persons in each group column [0-3]
loop1:
for (var i = 0; i < members.length; i++){
var loopbreaker = 0; //Avoids an endless loop if ignoreTeams = true
var ignoreTeams = false;
var row = 2;
if (!members[i].assigned || i == 0){
var teams = []; //Array of teams of each group
sheet_2.getRange(row, col).setValue(members[i].name);
members[i].assigned = true;
teams.push(members[i].team);
loop2:
for (var loop = 0; loop < 9; loop++){
var value = order_params(i, loop, members, teams, ignoreTeams);
if (value != null){
sheet_2.getRange(row + 1, col).setValue(value); //Writes in sheet Output
counter++;
row++;
}
//If the group is smaller than 4 and the loop is over, it will ignore the teams and start again in order to complete the group.
if (counter < 3 && loop == 8 && loopbreaker < 1){
loop = 0;
ignoreTeams = true; //Set to false to never ignore teams
loopbreaker++;//The loopbreaker avoids an endless loop
} else if (counter == 3 || loop == 8){
col++;
counter = 0;
break loop2;
}
}
}
}
}
function order_params(i, loop, members, teams, ignoreTeams){
for (var x = 0; x < members.length; x++){
if (!members[x].assigned && members[x].email != members[i].email && members[x].city == members[i].city && (!checkTeams(members[x].team, teams) || ignoreTeams)) {
if (members[x].choice1 == members[i].choice1 && loop < 3) {
members[x].assigned = true;
teams.push(members[x].team);
return members[x].name
} else if (members[x].choice2 == members[i].choice1 && loop >= 3 && loop < 6) {
members[x].assigned = true;
teams.push(members[x].team);
return members[x].name
} else if (members[x].choice3 == members[i].choice1 && loop >= 6 && loop < 9) {
members[x].assigned = true;
teams.push(members[x].team);
return members[x].name
}
}
}
}
//Checks for duplicate teams in the group
function checkTeams(member_t, teams){
for (var i = 0; i < teams.length; i++){
if (member_t == teams[i]){
return true;
}
}
}
//Converts the data from the Sheet into Person objects.
function Person(email, name, team, city, choice1, choice2, choice3, driver){
this.email = email;
this.name = name;
this.team = team;
this.city = city;
this.choice1 = choice1;
this.choice2 = choice2;
this.choice3 = choice3;
this.driver = driver;
this.assigned = false; //Assigned = true means the Person is already in a group
}
As you see I'm not using the field Driver, so you can make a condition in case you see too many groups without drivers, or change the members manually. If this is a serious problem, I will modify the code.
Object property value differentiates from the value I'm assigning to it,even though none other changes are made to the property which I checked with implementation of Object. watch I found at this link Watch for object properties changes in JavaScript.
From sofascore I'm getting an array with weeks (objects with start and end time) and I want to add home and away standings to each week as they were at the end of that week.I'm getting matches and their start time and result also from sofascore.
My sorting algorithm which sorts the table and pushes teams to their index in standings.home or standings.away array is working as expected.
I've worked on this for a couple hours now and a couple of hours last night and I think that I've already tried every possible solution that I can think of.
const https = require('https');
function get(url) {
return new Promise( resolve => {
https.get(url , res => {
res.setEncoding("utf8");
let body = "";
res.on("data", data => {body += data;});
res.on("end", () => {resolve(body)});
});
});
}
function standingsByWeek(tournamentID,seasonID) {
const baseURL = 'https://www.sofascore.com/';
var league,weeks,matches,teams,teamsIDs,standings,currentWeekIndex = 0,table = {},possiblePoints = [3,1,0];
get( baseURL + 'u-tournament/' + tournamentID + '/season/' + seasonID + '/json').then(function(league) { //get data for league
league = JSON.parse(league);
weeks = league.events.weeks;
teams = league.teams;
for (let i = 0; i < teams.length; i++) { //fill up table with teams
table[teams[i].id] = { id : teams[i].id, home : [0,0,0,0,0,0,teams[i].id], away : [0,0,0,0,0,0,teams[i].id]} ; // wins,draws,losses,score,conceded,points
};
teamsIDs = Object.keys(table); //extract keys (teamsIDs) from table
return get( baseURL + 'u-tournament/'+tournamentID+'/season/'+seasonID+'/matches/week/'+weeks[0].weekStartDate+'/'+weeks[weeks.length-1].weekEndDate)// get data for matches
}).then(function(matches) {
matches = JSON.parse(matches).weekMatches.tournaments[0].events; // assign matches to matches
for (let i = 0; i < matches.length; i++) { //update the table for every finished match
with (matches[i]){
if(status.code != 100){continue} // continue if match not finished
var outcome = Math.sign(homeScore.normaltime - awayScore.normaltime); // get outcome 1 = win , 0 = draw , -1 lose (for home team)
table[homeTeam.id].home[Math.abs(outcome - 1)] ++; // incrementing number at index for outcome
table[awayTeam.id].away[outcome + 1] ++; // -||-
table[homeTeam.id].home[3] += homeScore.normaltime; // adding scored goals in home team's home table
table[homeTeam.id].home[4] += awayScore.normaltime; // adding conceded goals in home team's home table
table[awayTeam.id].away[3] += awayScore.normaltime; // adding scored goals in away team's away table
table[awayTeam.id].away[4] += homeScore.normaltime; // adding conceded goals in away team's away table
table[homeTeam.id].home[5] += possiblePoints[Math.abs(outcome - 1)]; // adding earned points to home team's home table
table[awayTeam.id].away[5] += possiblePoints[Math.abs(outcome + 1)]; // adding earned points to away team's away table
}
// if this is the last match of the season or current week ended ,sort the table and push standings to that week and increase current week index
if ( (i + 1) == matches.length || weeks[currentWeekIndex].weekEndDate < matches[i + 1].startTimestamp ) {
standings = {home : [], away:[] };
for (let j = 0; j < 2; j++) { // loop through both keys in standings
var HoA = Object.keys(standings)[j]; //home or away
k : for (let k = 0; k < teamsIDs.length; k++) { // loop through all teams in the table
for (let l = 0; l <= standings[HoA].length; l++) { // loop through all teams added to standings
var EoL = Boolean(l == standings[HoA].length) // empty or last
var tableTeam = table[teamsIDs[k]][HoA] ;
var standingsTeam = EoL ? null : standings[HoA][l]; //standings team or null if standings is empty
var pointDiff = l == standings[HoA].length ? null : Math.sign(tableTeam[5] - standingsTeam[5]) ; // point difference 1 = table team has more, 0 = equal # of points, -1= standings team has more points
// standings empty(or last) or more points or equal points and better goal difference
if ( EoL || (pointDiff == 1) || !pointDiff && (tableTeam[3] - tableTeam[4]) > (standingsTeam[3] - standingsTeam[4] ) ) {
standings[HoA].splice(l,0,table[teamsIDs[k]][HoA]); // put team from table to index at which team which it was compared to was
continue k // go to another team
}
}
}
}
// USING THIS INSTEAD OF LINE BELOW IT YOU CAN SEE WHAT VALUE IS BEING ASSIGNED TO STANDINGS PROPERTY OF WEEK OBJECT
//Object.defineProperty(weeks[currentWeekIndex],'setter',{set: function (value) {this.standings = value;console.log(currentWeekIndex,value);}})
// weeks[currentWeekIndex].setter = Object.assing({},standings);
weeks[currentWeekIndex].standings = Object.assign({},standings);// assigning standings clone to standing property of current week
currentWeekIndex++;// increase week index
}
}
console.log(weeks[0].standings);
}
)
}
standingsByWeek(17,13380); // premier league , season 17/18
I expect object property to have the same value as the assigned value if none other changes to property are made.
EDIT
link to tidied and runnable version of the code https://repl.it/#mwittig/objectPropertiesChange
(thanks to Marcus)
I am trying speedup the processing/calculation in Google spreadsheet by putting my bonus setting into global array using PropertiesServices instead of loading the bonus % from the Setting sheet whenever user update their Daily sheet.
The method I used seem not working as I expected, the FOR loop is unable to look for "ALL". Hope someone able to give some advice.
I suspected that JSON structure is different from my 2D array, but I don't know how to solve it, I am new to javascripts.
Setting sheet only containing 4 columns: Game, StartDate, EndDate, Bonus %
p/s: This is a simplified scripts.
function onOpen(event) {
var bonusSheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Setting");
var bonusCompany = bonusSheet.getRange("A46:D503").getValues();
PropertiesService.getDocumentProperties().setProperty("bonusCompany", JSON.stringify(bonusCompany));
}
function onEdit(event) {
var colGame = "H";
var colBonus = "K";
var numColBonus = 11;
var nPoint;
var sheet = event.source.getActiveSheet();
var row = sheet.getActiveCell().getRow();
var col = sheet.getActiveCell().getColumn();
var cGame = sheet.getRange(colGame+row).getValue();
var dDate = new Date();
for(var k=0, kLen=bonusCompany.length; k<kLen; k++)
{
if((bonusCompany[k][0] == cGame || bonusCompany[k][0] == "ALL") && dDate.valueOf() >= bonusCompany[k][1] && dDate.valueOf() <= bonusCompany[k][2] ){
sheet.getRange(event.range.rowStart,numColBonus,event.range.rowEnd-event.range.rowStart+1,1).setValue(nPoint*bonusCompany[i][4]/100);
return;}
}
sheet.getRange(event.range.rowStart,numColBonus,event.range.rowEnd-event.range.rowStart+1,1).setValue(0);
}
If you want to use PropertiesService.getDocumentProperties(), how about following sample script?
Your script is almost correct. When the data is set by PropertiesService.getDocumentProperties().setProperty(), the data can be retrieved by PropertiesService.getDocumentProperties().getProperty(). The detail information is https://developers.google.com/apps-script/reference/properties/properties-service.
And when the array data is set by setProperty(), the array data is stored as strings by JSON.stringify(). So it is necessary to convert from the string data to the array data. Fortunately, it has already known that the values retrieved by getValues() from spreadsheet is 2 dimensional array. So the string data can be converted to an array data using the regular expression. The sample script is as follows. I added 2 lines to your script. Please check them.
Script :
function onOpen(event) {
var bonusSheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Setting");
var bonusCompany = bonusSheet.getRange("A46:D503").getValues();
PropertiesService.getDocumentProperties().setProperty("bonusCompany", JSON.stringify(bonusCompany));
}
function onEdit(event) {
var colGame = "H";
var colBonus = "K";
var numColBonus = 11;
var nPoint;
var sheet = event.source.getActiveSheet();
var row = sheet.getActiveCell().getRow();
var col = sheet.getActiveCell().getColumn();
var cGame = sheet.getRange(colGame+row).getValue();
var dDate = new Date();
var d = PropertiesService.getDocumentProperties().getProperty("bonusCompany").split(/^\[|\]$/)[1].split(/\[(.*?)\]/); // I added this line.
var bonusCompany = [[(e=="\"\"" ? "" : e) for each (e in f)] for each (f in [d[j].split(",") for (j in d) if(j%2==1)])]; // I added this line.
for(var k=0, kLen=bonusCompany.length; k<kLen; k++) {
if((bonusCompany[k][0] == cGame || bonusCompany[k][0] == "ALL") && dDate.valueOf() >= bonusCompany[k][1] && dDate.valueOf() <= bonusCompany[k][2] ){
sheet.getRange(event.range.rowStart,numColBonus,event.range.rowEnd-event.range.rowStart+1,1).setValue(nPoint*bonusCompany[i][4]/100);
return;
}
}
sheet.getRange(event.range.rowStart,numColBonus,event.range.rowEnd-event.range.rowStart+1,1).setValue(0);
}
I don't know data of your spreadsheet. So if this modification doesn't work, can I ask you about the data of your spreadsheet?
I am using the below script to delete duplicate rows from the google spreadsheet. The script was working good but as the data in the spreadsheet is being added daily, now the script is throwing "Exceeded maximum execution time" error. As I am new to scripting I don't understand what is my problem.
Could someone help me in solving this problem of mine.
function Deleteduplicates() {
var SpreadSheetKey = "My key";
var sheetD = SpreadsheetApp.openById(SpreadSheetKey).getSheetByName("Daily");
var sheetW = SpreadsheetApp.openById(SpreadSheetKey).getSheetByName("Weekly");
var dataD = sheetD.getDataRange().getValues();
var dataW = sheetW.getDataRange().getValues();
//Daily
var newDataD = new Array();
for(i in dataD){
var row = dataD[i];
var duplicate = false;
for(j in newDataD){
if(row.join() == newDataD[j].join()){
duplicate = true;
}
}
if(!duplicate){
newDataD.push(row);
}
}
//weekly
var newDataW = new Array();
for(i in dataW){
var row = dataW[i];
var duplicate = false;
for(j in newDataW){
if(row.join() == newDataW[j].join()){
duplicate = true;
}
}
if(!duplicate){
newDataW.push(row);
}
}
sheetD.clearContents();
sheetW.clearContents();
sheetD.getRange(1, 1, newDataD.length, newDataD[0].length).setValues(newDataD);
sheetW.getRange(1, 1, newDataW.length, newDataW[0].length).setValues(newDataW);
}
Conceptually, this should be quite a bit faster. I have not tried it on a large data set. The first version will leave the rows sorted as they were originally. The second version will be faster but will leave the rows sorted according to the columns from first to last on first text.
function Deleteduplicates() {
var SpreadSheetKey = "My key";
var ss = SpreadsheetApp.openById(SpreadSheetKey);
var sheetD = ss.getSheetByName("Daily");
var sheetW = ss.getSheetByName("Weekly");
var sheets = [sheetD, sheetW];
var toSs = {};
for(s in sheets) {
var data = sheets[s].getDataRange().getValues();
for(i in data){
// EDIT: remove commas from join("") for blank test
data[i].unshift(data[i].join(""),(1000000 + i).toString());
}
data.sort();
// remove blank rows -- Edit
var blank = 0;
while(data[blank][0].trim().length == 0) {blank++};
if(blank > 0) data.splice(0, blank);
// end Edit
var len = data.length - 1;
for(var x = len; x > 0; x-- ) {
if(data[x][0] == data[x-1][0]) {
data.splice(x, 1);
};
};
for(i in data) {
data[i].splice( 0, 1);
};
data.sort();
for(i in data) {
data[i].splice(0, 1);
};
toSs[sheets[s].getSheetName()] = data;
};
for(s in sheets) {
var data = toSs[sheets[s].getSheetName()];
sheets[s].clearContents();
sheets[s].getRange(1, 1, data.length, data[0].length).setValues(data);
}
}
Faster leaving rows sorted by join() created to test for duplicates
function Deleteduplicates() {
var SpreadSheetKey = "My key";
var ss = SpreadsheetApp.openById(SpreadSheetKey);
var sheetD = ss.getSheetByName("Daily");
var sheetW = ss.getSheetByName("Weekly");
var sheets = [sheetD, sheetW];
var toSs = {};
for(s in sheets) {
var data = sheets[s].getDataRange().getValues();
for(i in data){
// EDIT: remove commas from join("") for blank test
data[i].unshift(data[i].join(""));
}
data.sort();
// remove blank rows -- Edit
var blank = 0;
while(data[blank][0].trim().length == 0) {blank++};
if(blank > 0) data.splice(0, blank);
// end Edit
var len = data.length - 1;
for(var x = len; x > 0; x-- ) {
if(data[x][0] == data[x-1][0]) {
data.splice(x, 1);
};
};
for(i in data) {
data[i].splice( 0, 1);
};
toSs[sheets[s].getSheetName()] = data;
};
for(s in sheets) {
var data = toSs[sheets[s].getSheetName()];
sheets[s].clearContents();
sheets[s].getRange(1, 1, data.length, data[0].length).setValues(data);
}
}
Edited per Henrique's comment.
Edited 5/8: Remove blank rows(2 edited areas marked)
There is no problem with your script. It is just exceeding the "maximum execution time" allowed for any script (which is currently 6 minutes).
To workaround this problem you'll have to split your problem into "less than 6 minutes" parts.
For example, in your code you're clearing duplicates from 2 sheets. Trying creating two functions, one for each, and run them separately.
Also, there could be some performance enhancements that could make the script run under 6 minutes. For example, I'm not sure joining each row is the best way (performance-wise) to do an array comparison.
Creating a new array to re-set the data might not be optimal either, I'd probably go with a map verification, which is constant-time, instead of O(n^2) double array checking you're doing.
Bottom line, this is a limitation you have to live with in Apps Script. And any solution anyone proposes is just a workaround, that will also eventually fail if your data gets overly big.
I have a spreadsheet of surveys, in which I need to see how particular users have varied over time. As such, I need to disregard all rows with unique values in a particular column. The data looks like this:
Response Date Response_ID Account_ID Q.1
10/20/2011 12:03:43 PM 23655956 1168161 8
10/20/2011 03:52:57 PM 23660161 1168152 0
10/21/2011 10:55:54 AM 23672903 1166121 7
10/23/2011 04:28:16 PM 23694471 1144756 9
10/25/2011 06:30:52 AM 23732674 1167449 7
10/25/2011 07:52:28 AM 23734597 1087618 5
I've found a way to do so in Excel VBA:
Sub Del_Unique()
Application.ScreenUpdating = False
Columns("B:B").Insert Shift:=xlToRight
Columns("A:A").Copy Destination:=Columns("B:B")
i = Application.CountIf(Range("A:A"), "<>") + 50
If i > 65536 Then i = 65536
Do
If Application.CountIf(Range("B:B"), Range("A" & i)) = 1 Then
Rows(i).Delete
End If
i = i - 1
Loop Until i = 0
Columns("B:B").Delete
Application.ScreenUpdating = True
End Sub
I'd like to do it in Google Spreadsheets with a script that won't have to be changed. Closest I can get is retrieving all duplicate user ids from the range, but can't associate that with the row. That code follows:
function findDuplicatesInSelection() {
var activeRange = SpreadsheetApp.getActiveRange();
var values = activeRange.getValues();
// values that appear at least once
var once = {};
// values that appear at least twice
var twice = {};
// values that appear at least twice, stored in a pretty fashion!
var final = [];
for (var i = 0; i < values.length; i++) {
var inner = values[i];
for (var j = 0; j < inner.length; j++) {
var cell = inner[j];
if (cell == "") continue;
if (once.hasOwnProperty(cell)) {
if (!twice.hasOwnProperty(cell)) {
final.push(cell);
}
twice[cell] = 1;
} else {
once[cell] = 1;
}
}
}
if (final.length == 0) {
Browser.msgBox("No duplicates found");
} else {
Browser.msgBox("Duplicates are: " + final);
}
}
This is maybe not very efficient, but I think it's what you want:
var ar=[1,3,3,5,6,8,6,6];
console.log("Before:");
display(ar);//1 3 3 5 6 8 6 6
var index=[];
var ar2=[];
for(var a=0;a<ar.length;a++)
{
var duplicate=false;
for(var b=0;b<ar.length;b++)
{
if(ar[a]==ar[b]&&a!=b)
{
duplicate=true;
}
}
if(!duplicate)
{
index.push(a);
}
}
for(var a=0;a<index.length;a++)
{
ar[index[a]]=null;
}
for(var a=0;a<ar.length;a++)
{
if(ar[a]!=null)ar2.push(ar[a]);
}
console.log("After:");
display(ar2);//3 3 6 6 6
function display(x)
{
for(var a=0;a<x.length;a++)console.log(x[a]);
}
The fiddle : http://jsfiddle.net/mageek/6AGQ4/
And a shorter version that is as a function :
var ar=[1,3,3,5,6,8,6,6];
function removeUnique(x)
{
var index=[];
var ar2=[];
for(var a=0;a<ar.length;a++)
{
var duplicate=0;
for(var b=0;b<ar.length;b++)if(ar[a]==ar[b]&&a!=b)duplicate=1;
if(!duplicate)index.push(a);
}
for(var a=0;a<index.length;a++)ar[index[a]]=null;
for(var a=0;a<ar.length;a++)if(ar[a]!=null)ar2.push(ar[a]);
return x;
}
ar=removeUnique(ar);
The fiddle : http://jsfiddle.net/mageek/6AGQ4/2
I'd suggest going for something simple.
Create a short script that flags duplicates
Write the formula directly into the cell "=flagDuplicate(C2,C$2:C$10)"
Copy the forumla down the column
Use Spreadsheet's built in QUERY formula to pull the information you need
"=QUERY(A1:E10; "SELECT * WHERE E = TRUE"; 1)"
Here is a simple function to flag duplicates
function flagDuplicate(value, array) {
var duplicateCounter = 0;
for (var i=0; i<array.length; i++){
if (array[i] == value){ // I avoid === in Spreadsheet functions
duplicateCounter++;
}
}
if (duplicateCounter > 1){
return true;
}else{
return false;
}
}
Too many functions on a large table can slow things down. If it becomes a problem, you can always copy and "paste values only" - that will retain the information but remove the functions.
Best of luck.
Note: When I tested this I noticed that can take a while before the spreadsheet recognizes the new custom function (gives error like can't find function FLAGDUPLICATE)
You could also do it using arrays to handle the whole sheet at once :
function removeUnique(){
var col = 2 ; // choose the column you want to check for unique elements
var sh = SpreadsheetApp.getActiveSheet();
var ss = SpreadsheetApp.getActiveSpreadsheet();
var data=ss.getDataRange().getValues();// get all data
data.sort(function(x,y){
// var xp = Number(x[col]);// use these to sort on numeric values
// var yp = Number(y[col]);
var xp = x[col];// use these for non-numeric values
var yp = y[col];
Logger.log(xp+' '+yp); // just to check the sort is OK
return xp == yp ? 0 : xp < yp ? -1 : 1;// sort on column col numeric ascending
});
var cc=0;
var newdata = new Array();
for(nn=0;nn<data.length-1;++nn){
if(data[nn+1][col]==data[nn][col]||cc>0){
newdata.push(data[nn]);
++cc;
if(cc>1){cc=0}}
}
ss.getDataRange().clearContent(); // clear the sheet
sh.getRange(1,1,newdata.length,newdata[0].length).setValues(newdata);// paste new values sorted and without unique elements
}
EDIT : here is the version that keeps all duplicates (the working one)
function removeUnique(){
var col = 2 ; // choose the column you want to check for unique elements
var sh = SpreadsheetApp.getActiveSheet();
var ss = SpreadsheetApp.getActiveSpreadsheet();
var data=ss.getDataRange().getValues();// get all data
data.sort(function(x,y){
// var xp = Number(x[col]);// use these to sort on numeric values
// var yp = Number(y[col]);
var xp = x[col];// use these for non-numeric values
var yp = y[col];
Logger.log(xp+' '+yp); // just to check the sort is OK
return xp == yp ? 0 : xp < yp ? -1 : 1;// sort on column col numeric ascending
});
var newdata = new Array();
for(nn=0;nn<data.length-1;++nn){
if(data[nn+1][col]==data[nn][col]){
newdata.push(data[nn]);
}
}
if(data[nn-1][col]==data[nn][col]){newdata.push(data[nn])}
ss.getDataRange().clearContent(); // clear the sheet
sh.getRange(1,1,newdata.length,newdata[0].length).setValues(newdata);// paste new values sorted and without unique elements
}