Develop Reference

Pivot Table Created From CSV File

I'm dealing with an issue where the formatting on a CSV file that we're importing needs to be"pivoted" to match the formatting required for the program we are using to process the import.
Currently we are importing the file which comes with the following format:
Account
Department
Jan2022
Feb2022
Mar2022
12345
Sales
$456
$876
$345
98765
HR
$765
$345
$344
We need the format to hold the time periods in one column which would make each account be repeated per time period. For example:
Account
Department
Period
Amount
12345
Sales
Jan2022
$456
12345
Sales
Feb2022
$876
12345
Sales
Mar2022
$345
We are importing this CSV using JavaScript however its basic JS as the program does not support JQuery or any other JS library. Once we import the table into our staging area using JS, we can use SQL to modify the data as well, so this could be solved with either JS or SQL.
We are using a CSV to Array function to read the CSV file for importing into staging:
function CSVToArray(strData, strDelimiter) {
// Check to see if the delimiter is defined. If not, then default to comma.
strDelimiter = strDelimiter || ",";
// Create a regular expression to parse the CSV values.
var objPattern = new RegExp(
// Delimiters.
"(\\" +
strDelimiter +
"|\\r?\\n|\\r|^)" +
// Quoted fields.
'(?:"([^"]*(?:""[^"]*)*)"|' +
// Standard fields.
'([^"\\' +
strDelimiter +
"\\r\\n]*))",
"gi"
);
// Create an array to hold our data. Give the array a default empty first row.
var arrData = [[]];
// Create an array to hold our individual pattern matching groups.
var arrMatches = null;
// Keep looping over the regular expression matches until we can no longer find a match.
while ((arrMatches = objPattern.exec(strData))) {
// Get the delimiter that was found.
var strMatchedDelimiter = arrMatches[1];
// Check to see if the given delimiter has a length (is not the start of string) and if it matches
// field delimiter. If id does not, then we know that this delimiter is a row delimiter.
if (strMatchedDelimiter.length && strMatchedDelimiter !== strDelimiter) {
// Since we have reached a new row of data, add an empty row to our data array.
arrData.push([]);
}
//Now that we have our delimiter out of the way, let's check to see which kind of value we captured (quoted or unquoted).
var strMatchedValue;
if (arrMatches[2]) {
// We found a quoted value. When we capture this value, unescape any double quotes.
strMatchedValue = arrMatches[2]
.replace(new RegExp('""', "g"), '"')
.replace('"', "");
} else {
// We found a non-quoted value.
strMatchedValue = arrMatches[3];
}
// Now that we have our value string, let's add it to the data array.
arrData[arrData.length - 1].push(strMatchedValue);
}
// Return the parsed data.
return arrData;
}

UNPIVOT should work for you:
/* sample data */
with t as
(select '12345' account,
'Sales' department,
'$456' jan2022,
'$876' feb2022,
'$345' mar2022
from dual
union all
select '98765' account,
'HR' department,
'$765' jan2022,
'$345' feb2022,
'$344' mar2022
from dual)
select *
from t
unpivot include nulls(amount for period in(jan2022 as 'jan2022',
feb2022 as 'feb2022',
mar2022 as 'mar2022'));
If you have dynamic columns you gonna have bad time with this aproach - you have to generate "unpivot in clause" (that part with jan2022 as 'jan2022') on your own.

Comparing '==' in an IF statement is not working Google Sheets script

The objective is to compare ColA in the Orders sheet with ColF in the Ordered Items sheet, if they match grab the email from ColB.
The script outputs no errors, but it doesn't match the highlighted cells in either sheet.
(note: the items are automatically added by an app, so the formatting of the cells are default and need to keep it that way as I'm using the last 6 digits as the order reff eg; 49.263Z)
Orders sheet
Ordered Items sheet
function getEmailFromOrderedItemToOrders(){
var orders = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Orders');
var lr = getLastRow(orders, "A1:G");
Logger.log(lr); //LastRow index
//Get last 'OrderID (Paid at)' value from 'Orders' sheet
var orderIdCol = 1;
var orderId = orders.getRange(lr, orderIdCol).getValue();
Logger.log(orderId); //LastRow 'orderId' value
//Match 'orderId' to 'orderId' in 'Ordered Items' and return col 1
var items = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Ordered Items');
var itemsData = items.getDataRange().getValues();
Logger.log(itemsData[0][1]); //'Purchase Email' col
Logger.log(itemsData[0][5]); //'Paid at' col
for(var i = 0; i<itemsData.length;i++){
if(itemsData[i][5] == orderId){ //Issue here: not comparing values as a match
var email = itemsData[i][1];
Logger.log(email); //Does not print
return i+1;
}
}
}

In javascript when you compare two new Date, you will be getting the value as false.
Reason is when you have two dates, its basically two different objects. So One object is not equal to another object when you use == or ===. So the simple and better way is converting the date to a number using .getTime() that will return how many milliseconds passed from that date and 00:00:00 of Jan 1, 1970.
Sample code snippet
console.log(new Date() === new Date())
console.log(new Date().getTime() === new Date().getTime())

This can be simply solved by using a Sheets Formula which. What you would do is:
=INDEX( 'Ordered Items!B:B', MATCH(A2, 'Ordered Items!:F:F,0) )
The formula basically says:
Return the Email Column
Find the Index (# of the row) where the value of A2 is in column Ordered Items F:F
See here for a tutorial on it: https://www.youtube.com/watch?v=9sLWDjAEuyc

Find missing quarters in time series

I am trying to see how could I fill the missing financial quarters of a time series like this in Javascript:
["2012-Q2","2012-Q4","2013-Q4","2014-Q1","2014-Q2","2014-Q3",
"2014-Q4","2015-Q1","2015-Q2","2015-Q3","2015-Q4","2016-Q1",
"2016-Q2","2016-Q3","2016-Q4","2017-Q1","2017-Q2","2017-Q3",
"2017-Q4","2018-Q1"]
I would like somehow to get a time series with the missing elements i.e. for each year I should see 4 "dates".
I don't mind ignoring the first quarter before the first element "2012-Q2" and the last 3 quarters after the last element "2018-Q1".
I know moment.js has functions like quarter() or fquarter() (via a plugin), but I am looking for something closer to the other way around. I already have the quarters (as date-strings), and I have to parse them as date objects.
I need to fill the "quarter holes" in between those input string values.
In my case I probably need to parse first those date-strings in that custom format to make them something moment could understand, but I am a bit lost. In here https://momentjs.com/docs/#/parsing/string-format/ a potential format could involve Y for years and Q for quarters, but I am not sure how to escape the literal Q inside every input date-string of that array?
Also assuming I could somehow parse all those date strings into moment objects, then I am not sure how that could help in filling the holes?
I can not find a pure javascript solution involving date types.
Another approach could be to parse those date-strings and get the year and the quarter number using substring and then manually filling the holes checking year/quarter pairs, is there anything simpler than this?

Assuming that you want a full list of quarters between the first one of your input array until the last one, you can:
parse with moment the first and the last element of your array, using moment(String, String) with 'YYYY[-Q]Q' as format parameter, see Escaping charaters section of the docs.
loop from start to end using isSameOrBefore (or other query functions) adding 1 quarter on each iteration (add(1, 'Q'))
Here a live sample:
var quarters = ["2012-Q2","2012-Q4","2013-Q4","2014-Q1","2014-Q2","2014-Q3",
"2014-Q4","2015-Q1","2015-Q2","2015-Q3","2015-Q4","2016-Q1",
"2016-Q2","2016-Q3","2016-Q4","2017-Q1","2017-Q2","2017-Q3",
"2017-Q4","2018-Q1"];
var format = 'YYYY[-Q]Q';
var start = moment(quarters[0], format);
var end = moment(quarters[quarters.length-1], format);
var results = [];
while( start.isSameOrBefore(end) ){
results.push(start.format(format));
start.add(1, 'Q');
}
console.log(results);
<script src="https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.20.1/moment.min.js"></script>

Why not just make a function that returns the quarters from a particular range?
function getQuarters(startYear, endYear){
var times = [];
for(var i = startYear; i <= endYear; i++){
times.push(i + "-Q1");
times.push(i + "-Q2");
times.push(i + "-Q3");
times.push(i + "-Q4");
}
return times;
}
Calling:
getQuarters(2017,2017);
Returns:
["2017-Q1", "2017-Q2", "2017-Q3", "2017-Q4"]

Assuming you want to get an array with the missing values, you could take a start quarter and cehck against the given data for either pushing the quartal or incremet the index of the array.
function incQ(time) {
time[1]++;
if (time[1] === 5) {
time[0]++;
time[1] = 1;
}
}
var quarters = ["2012-Q2", "2012-Q4", "2013-Q4", "2014-Q1", "2014-Q2", "2014-Q3", "2014-Q4", "2015-Q1", "2015-Q2", "2015-Q3", "2015-Q4", "2016-Q1", "2016-Q2", "2016-Q3", "2016-Q4", "2017-Q1", "2017-Q2", "2017-Q3", "2017-Q4", "2018-Q1"],
actual = quarters[0].split('-Q'),
missing = [],
i = 0;
while (i < quarters.length) {
if (actual.join('-Q') !== quarters[i]) {
missing.push(actual.join('-Q'));
} else {
i++;
}
incQ(actual);
}
console.log(missing);

A string of form nnnn-nnnn is displayed in a spreadsheet as a date or otherwise incorrectly

I have a script I have been using in my test environment to programmically create a tracking number by parsing the year from timestamp and padding the response index.
function setTrackingNumber(ss, lastRowInx, createDateColumn) //This block generates and stores a tracking number in Column AU on the backend
{
var padTrackNo = "" + lastRowInx;
var trackSize = 4;
var trackingNumberColumn = createDateColumn-3; //trackingNumberColumn is currently in AU (Column 47) Calculating using it's relative position to createDateColumn Position
if (ss.getRange(lastRowInx, trackingNumberColumn).getValue() == "") // so that subsequent edits to Google Form don't overwrite original tracking number
{
if (padTrackNo > trackSize)
{
var padTrackNo = pad(padTrackNo, trackSize);
}
else {} //do nothing
var shortYear = setShortYear(ss, lastRowInx, createDateColumn);
var trackingNumber = shortYear + "-" + padTrackNo;
var createTrackingNumber = ss.getRange(lastRowInx, trackingNumberColumn);
createTrackingNumber.setValue(trackingNumber);
}
else {} //should do nothing
return;
}//This is the end of the setTrackingNumber function
function setShortYear(ss, lastRowInx, createDateColumn)
{
var newCreateDate = ss.getRange(lastRowInx,createDateColumn).getValue();
var strDate = "" + newCreateDate;
var splitDate = strDate.split(" ");
var trimYear = splitDate[3];
var shortYear = trimYear;
return shortYear;
}//This is the end of the shortYear function
function pad(padTrackNo, trackSize)
{
while (padTrackNo.length < trackSize)
{
padTrackNo = "0"+padTrackNo;
}
return padTrackNo;
}//This is the end of pad function
That gets me test result which is as expected ex. 2016-0005. However when we added it to another production sheet it seemed to work with test data and then production data showed up like a date 3/1/2016. production result - first cell.
I thought it must just be formatting the string as a date because of the numbers so I tried formatted the column as plain text but that just changed the date to a plain text version of the date.
I thought this might be similar to needing to specify the format like I did in this question Appending initial timestamp from Google Form to end of record in order to permanently store create date onFormSubmit at #SandyGood 's suggestion so I tried setting the number format as [0000-0000] by changing
createTrackingNumber.setValue(trackingNumber);
to
createTrackingNumber.setValue(trackingNumber).setNumberFormat("0000-0000");
which resulted in the [production result - second cell] which again doesn't match the expected result.
Oddly, some submissions seem to work just fine like [production result - third cell]. Over the past 3 days and approximately 10 records it has been fine, then hinky, then fine, they hinky, then fine again. I am not really sure what else to try to debug this odd behaviour.
Note: I had to parse the date as a string as I was having trouble getting it to parse the date correctly from the create date which is taken from initial timestamp.

To my understanding, "2016-0005" is not a number but a string, so the cell containing it should be formatted as plain text. With a script, this can be done by
range.setNumberFormat('#STRING#')
(source), and this must be done before you set the value to the cell. Like this:
createTrackingNumber.setNumberFormat('#STRING#').setValue(trackingNumber);

Finding duplicates within strings

I have been handed a project at work where I need to find duplicate pairings from multiple rows within a dataset. While the data set is much larger, the main portion revolves around the date of a training, the location of a training, and the names of the trainers. So every row of data has a date, a location, and then a comma separated list of names:
Date Location Names
1/13/2014 Seattle A, B, D
1/16/2014 Dallas C, D, E
1/20/2014 New York A, D
1/23/2014 Dallas C, E
1/27/2014 Seattle B, D
1/30/2014 Houston C, A, F
2/3/2014 Washington DC D, A, F
2/6/2014 Phoenix B, E
2/10/2014 Seattle C, B
2/13/2014 Miami A, B, E
2/17/2014 Miami C, D
2/20/2014 New York B, E, F
2/24/2014 Houston A, B, F
My goal is to be able to find rows with similar pairings of names. One example would be to know that A & B were in paired in Seattle on 1/13, Miami on 2/13, and Houston on 2/24, even though the third name is different in each occurrence. So instead of just simply finding duplicates among the entire string of names, I would also like to find pairings among partial segments of the “Names” column.
Is this possible to do within Excel or would I need to use a programming language to accomplish the task?
While I can manually do this, it represents a lot of time that could be used towards other things. If there was a way that I could automate this it would make this portion of my task a lot simpler.
Thank you in advance for any assistance or advice on a way forward.

You can do it with VBA. The solution below assumes
Your data is on the active sheet in columns A:C
You results will be output in columns E:G
The output will be a list sorted by pairs, and then by dates, so you can easily see where pairs repeated.
The routine assumes no more than three trainers at a time, but could be modified add more possible combinations.
Cities with just a single trainer will be ignored.
The routine uses a Class module to gather the information, and two Collections to process the data. It also makes use of the feature that collections will not allow addition of two items with the same key.
Class Module
Rename the Class Module: cPairs
Option Explicit
Private pTrainer1 As String
Private pTrainer2 As String
Private pCity As String
Private pDT As Date
Public Property Get Trainer1() As String
Trainer1 = pTrainer1
End Property
Public Property Let Trainer1(Value As String)
pTrainer1 = Value
End Property
Public Property Get Trainer2() As String
Trainer2 = pTrainer2
End Property
Public Property Let Trainer2(Value As String)
pTrainer2 = Value
End Property
Public Property Get City() As String
City = pCity
End Property
Public Property Let City(Value As String)
pCity = Value
End Property
Public Property Get DT() As Date
DT = pDT
End Property
Public Property Let DT(Value As Date)
pDT = Value
End Property
Regular Module
Option Explicit
Option Compare Text
Public cP As cPairs, colP As Collection
Public colCityPairs As Collection
Public vSrc As Variant
Public vRes() As Variant
Public rRes As Range
Public I As Long, J As Long
Public V As Variant
Public sKey As String
Sub FindPairs()
vSrc = Range("A1", Cells(Rows.Count, "C").End(xlUp))
Set colP = New Collection
Set colCityPairs = New Collection
'Collect Pairs
For I = 2 To UBound(vSrc)
V = Split(Replace(vSrc(I, 3), " ", ""), ",")
If UBound(V) >= 1 Then
'sort the pairs
SingleBubbleSort V
Select Case UBound(V)
Case 1
AddPairs V(0), V(1)
Case 2
AddPairs V(0), V(1)
AddPairs V(0), V(2)
AddPairs V(1), V(2)
End Select
End If
Next I
ReDim vRes(0 To colCityPairs.Count, 1 To 3)
vRes(0, 1) = "Date"
vRes(0, 2) = "Location"
vRes(0, 3) = "Pairs"
For I = 1 To colCityPairs.Count
With colCityPairs(I)
vRes(I, 1) = .DT
vRes(I, 2) = .City
vRes(I, 3) = .Trainer1 & ", " & .Trainer2
End With
Next I
Set rRes = Range("E1").Resize(UBound(vRes, 1) + 1, UBound(vRes, 2))
With rRes
.EntireColumn.Clear
.Value = vRes
With .Rows(1)
.HorizontalAlignment = xlCenter
.Font.Bold = True
End With
.Sort key1:=.Columns(3), order1:=xlAscending, key2:=.Columns(1), order2:=xlAscending, _
Header:=xlYes
.EntireColumn.AutoFit
V = VBA.Array(vbYellow, vbGreen)
J = 0
For I = 2 To rRes.Rows.Count
If rRes(I, 3) = rRes(I - 1, 3) Then
.Rows(I).Interior.Color = .Rows(I - 1).Interior.Color
Else
J = J + 1
.Rows(I).Interior.Color = V(J Mod 2)
End If
Next I
End With
End Sub
Sub AddPairs(T1, T2)
Set cP = New cPairs
With cP
.Trainer1 = T1
.Trainer2 = T2
.City = vSrc(I, 2)
.DT = vSrc(I, 1)
sKey = .Trainer1 & "|" & .Trainer2
On Error Resume Next
colP.Add cP, sKey
If Err.Number = 457 Then
Err.Clear
colCityPairs.Add colP(sKey), sKey & "|" & colP(sKey).DT & "|" & colP(sKey).City
colCityPairs.Add cP, sKey & "|" & .DT & "|" & .City
Else
If Err.Number <> 0 Then Stop
End If
On Error GoTo 0
End With
End Sub
Sub SingleBubbleSort(TempArray As Variant)
'copied directly from support.microsoft.com
Dim Temp As Variant
Dim I As Integer
Dim NoExchanges As Integer
' Loop until no more "exchanges" are made.
Do
NoExchanges = True
' Loop through each element in the array.
For I = LBound(TempArray) To UBound(TempArray) - 1
' If the element is greater than the element
' following it, exchange the two elements.
If TempArray(I) > TempArray(I + 1) Then
NoExchanges = False
Temp = TempArray(I)
TempArray(I) = TempArray(I + 1)
TempArray(I + 1) = Temp
End If
Next I
Loop While Not (NoExchanges)
End Sub

Ok. I got bored and did this whole thing in Python code. I assume you are familiar with the language; however, you should be able to get the following piece of code to work on any computer with Python installed.
I have made a few assumptions. For instance, I have used your example input as definite input.
A few things which will mess up the program:
Not entering with case sensitivity. Beware of capital letters etc.
Having a inputfile which has the following row: "Date Location Names". Just remove and keep straight facts in the file. I got lazy and do not bother adjusting this.
A ton of other small stuff. Just do what the program asks you to do and dont enter funky input.
About program:
Revolves around using a dictionary with person names as keys. The values in the dictionary is a set with tuples containing the places they've been during what date. By then comparing these sets and getting the intersection, we can find the answer.
Kinda messy since I took this as Python practice. Have not coded in Python for a while and I got a thrill out of doing it all without utilizing objects. Just follow the "instructions" and keep the inputfile, which stores all information, in the same folder as the piece of code are running.
As a side note, you might want to check that the program yields correct output.
If you have any questions, feel free to contact me.
def readWord(line, stringIndex):
word = ""
while(line[stringIndex] != " "):
word += line[stringIndex]
stringIndex += 1
return word, stringIndex
def removeSpacing(line, stringIndex):
while(line[stringIndex] == " "):
stringIndex += 1
return stringIndex
def readPeople(line, stringIndex):
lineSize = len(line)
people = []
while(stringIndex < lineSize):
people.append(line[stringIndex])
stringIndex += 3
return people
def readLine(travels, line):
stringIndex = 0
date, stringIndex = readWord(line, stringIndex)
stringIndex = removeSpacing(line, stringIndex)
location, stringIndex = readWord(line, stringIndex)
stringIndex = removeSpacing(line, stringIndex)
people = readPeople(line, stringIndex)
for person in people:
if(person not in travels.keys()):
travels[person] = set()
travels[person].add((date, location))
return travels
def main():
f = open(input("Enter filename (must be in same folder as this program code. For instance, name could be: testDocument.txt\n\n"))
travels = dict()
for line in f:
travels = readLine(travels, line)
print("\n\n\n\n PROGRAM RUNNING \n \n")
while(True):
persons = []
userInput = "empty"
while(userInput):
userInput = input("Enter person name (Type Enter to finish typing names): ")
if(userInput):
persons.append(userInput)
output = travels[persons[0]]
for person in persons[1:]:
output = output.intersection(travels[person])
print("")
for hit in output:
print(hit)
print("\nFINISHED WITH ONE RUN. STARTING NEW ONE\n")