Develop Reference

Fast-CSV modify before loading CSV to MySQL

I am trying to load a CSV file to my MYSQL database, however before I do so I need to modify it slightly. The CSV file is Pipe delimitated (|)
I have a column in the CSV file called Party:Identification. This column has results such as "a:hello, b:hi c:151 ......" This can go on infinitely.
I only need to get the value for c. I have come up with a method that works for this, however I am stuck on how to modify the value before the file is inserted into the database.
I tried replacing all the ":" in the headers with "" and then using .transform to modify the values, however this doesn't appear to change the values in the column, only the header. Code is attached below.
csv.parseFile(req.file.path, {
headers: headers => headers.map(function (header) {
const newHeaders = header.replaceAll(" ", "").replaceAll(":", "")
console.log(newHeaders)
return newHeaders
}),
delimiter: '|'
})
.transform(function(data) {
console.log(data)
PartyIdentification: getPartyID(data.partyIdentification)
})
.on("error", (err) => console.error(err))
.on("finish", function () {
query("LOAD DATA LOCAL INFILE '" +
file +
"' INTO TABLE table " +
" FIELDS TERMINATED BY '|'" +
" LINES TERMINATED BY '\n'" +
" IGNORE 1 ROWS;").then(r =>
console.log(file)
)
})
function getPartyID(str) {
if (str === undefined) return ""
const split = str.split(",")
const value = split.find(val => {
return val.includes("c")
})
if(value === undefined) return ""
return (value.split(":")[1].trim())
}

You can use a regex to parse the value of c:123 in a string:
function getPartyID(str) {
if (str === undefined) return "";
const m = str.match(/\bc:([^ ]*)/);
return m ? m[1] : null;
}
[
"a:hello, b:hi c:151 d:foo",
"a:hello, b:no_c",
].forEach(str => {
console.log(str, '==>', getPartyID(str));
});
Output:
a:hello, b:hi c:151 d:foo ==> 151
a:hello, b:no_c ==> null
Explanation of regex:
\b -- word boundary
c: -- literal text
([^ ]*) -- capture group 1 with value, up to and excluding space
UPDATE 1: Based on additional question on how to insert modified data into MySQL, here is a solution that does not use INFILE, but instead loads the file into memory (here simulated with const input), modifies the data as needed, and constructs a SQL statement that inserts all the data. IMPORTANT: You likely want to add escapes against SQL injections.
const input = `Name|Party:Identification|Date
Foo|a:hello, b:hi c:151 d:foo|2022-01-11
Bar|a:hola, b:hey c:99 d:bar|2022-01-12
Nix|a:nix, b:ni d:nix|2022-01-13`;
const partIdFieldName = 'Party:Identification';
function getPartyID(str) {
if (str === undefined) return "";
const m = str.match(/\bc:([^ ]*)/);
return m ? m[1] : 0;
}
let partIdIdx = 0;
let data = input.split(/[\r\n]+/).map((row, idx) => {
let cells = row.split('|');
if(idx === 0) {
partIdIdx = cells.indexOf(partIdFieldName);
} else {
cells[partIdIdx] = getPartyID(cells[partIdIdx]);
}
return cells;
});
//console.log('data', '==>', data);
let sql = 'INSERT INTO tbl_name\n' +
' (' + data[0].map(val => '"' + val + '"').join(',') + ')\n' +
'VALUES\n' +
data.slice(1).map(row => {
return ' (' + row.map(val => /^[\d+\.]+$/.test(val)
? val
: '"' + val + '"'
).join(',') + ')'
}).join('\n') + ';';
console.log(sql);
Output:
INSERT INTO tbl_name
("Name","Party:Identification","Date")
VALUES
("Foo",151,"2022-01-11")
("Bar",99,"2022-01-12")
("Nix",0,"2022-01-13");

Don't bother fixing the csv file before loading, simply toss the unwanted columns as you LOAD it.
This, for example, will load only the 3rd column:
LOAD DATA ...
(#a, #b, c_col, #d, #e, ...)
That is, capture the unwanted columns into #variables that you will then ignore.
If you need to remove the c: before storing into the table, then
LOAD DATA ...
(#a, #b, #c, #d, #e, ...)
SET c_c0l = mid(#c, 3)
(or whatever expression will work. See also SUBSTRING_INDEX in case it would work better.)
LOAD DATA is plenty fast, even in this wasteful mode. And a lot less coding on your part.

Removing "," at the beginning of a line CSV

i want to convert a .csv file and write a new one. However I am not able to remove the first , i am kinda stuck here and it is driving me crazy.
This is my code:
var extractedtasks = tasks.slice(0, 3)
var extractedtasksformated = extractedtasks.toString().replace(/,$/g, "\n")
let csvformat = "EMAIL,PASSWORD,MAILBOX"
fs.writeFileSync(tasklocation[0], csvformat + "\n" + extractedtasksformated.replace(/,^/g,
""))
console.log(chalk.green("Successfully updated the CSV file"))
That's the output i am getting in the newly generated file
EMAIL,PASSWORD,MAILBOX
example1#gmail.com,Password123,example#gmail.com:password
,example2#gmail.com,Password123,example#gmail.com:password
,example3#gmail.com,Password123,example#gmail.com:password
Output extractedtasks:
[
'example1#gmail.com,Password123,example#gmail.com:password\r',
'example2#gmail.com,Password123,example#gmail.com:password\r',
'example3#gmail.com,Password123,example#gmail.com:password\r'
]
Output extractedtasksformated:
,example3#gmail.com,Password123,example#gmail.com:passwordxample#gmail.com:password

Because extractedtasks is an array, instead of converting it to a string you should just join it with the expected separator:
extractedtasks = [
'example1#gmail.com,Password123,example#gmail.com:password\r',
'example2#gmail.com,Password123,example#gmail.com:password\r',
'example3#gmail.com,Password123,example#gmail.com:password\r'
]
extractedtasksJoined = extractedtasks.join("\n")
// "example1#gmail.com,Password123,example#gmail.com:password\r\nexample2#gmail.com..."
// depending on the target line separator, you should also probably
// remove the "\r"
extractedtasksJoined = extractedtasksJoined.replace("\r", "")
// finally
fs.writeFileSync(tasklocation[0], csvformat + "\n" + extractedtasksJoined + "\n")

Getting Uncaught TypeError: Cannot use 'in' operator to search for 'length' in

I want to add extracted genre from IMDB to my Categories automatically. I made JSON code to do this for me.
$('input[name=Checkbx]').click(function() {
var coc = $('input[name=Checkbx2]').get(0).value;
// Send Request
$.getJSON("http://www.omdbapi.com/?i=" + coc + "&plot=short&r=json", function(data) {
var valDir = "";
var valWri = "";
var valAct = "";
$.each(data, function(key, val) {
$('input[name=' +key+ ']').val(val);
if(key == "Director"){
valDir+= " "+val+",";
}
if(key == "Genre"){
var genr = "";
$.each( data.Genre, function( key, val ) {
genr += "" + val + ", ";
genr1 = val;
$('input[name=newcategory]').val( genr1 );
$('#category-add-submit').trigger('click');
$('#category-add-submit').prop("disabled", false);
$('input[name=newcategory]').val("");
});
$('input[name=' +key+ ']').val( genr );
}
if(key == "Actors"){
valAct+= " "+val+",";
}
if(key == "Year"){
$('#new-tag-<?php echo get_option('year'); ?>').val(val);
}
if(key == "Country"){
$('#new-tag-movie-country').val(val);
}
});
$('#new-tag-<?php echo get_option("director"); ?>').val(valDir);
$('#new-tag-<?php echo get_option("actor"); ?>').val(valAct);
});
});
For the record: I'm using omdbapi.com API for getting data from IMDB and my movie ID for test is tt0111161 and the JSON output is:
{"Title":"The Shawshank Redemption","Year":"1994","Rated":"R","Released":"14 Oct 1994","Runtime":"142 min","Genre":"Crime, Drama","Director":"Frank Darabont","Writer":"Stephen King (short story \"Rita Hayworth and Shawshank Redemption\"), Frank Darabont (screenplay)","Actors":"Tim Robbins, Morgan Freeman, Bob Gunton, William Sadler","Plot":"Two imprisoned men bond over a number of years, finding solace and eventual redemption through acts of common decency.","Language":"English","Country":"USA","Awards":"Nominated for 7 Oscars. Another 14 wins & 20 nominations.","Poster":"http://ia.media-imdb.com/images/M/MV5BODU4MjU4NjIwNl5BMl5BanBnXkFtZTgwMDU2MjEyMDE#._V1_SX300.jpg","Metascore":"80","imdbRating":"9.3","imdbVotes":"1,610,411","imdbID":"tt0111161","Type":"movie","Response":"True"}
Everything works fine. I mean Actors, Director and Casts are added automatically. My problem is with Genres.
I've got this error:
Uncaught TypeError: Cannot use 'in' operator to search for 'length' in Crime, Drama
My JQuery version is 2.2.1. How can i fix this?

The "Genre" field in your JSON looks like this:
"Genre":"Crime, Drama"
That is a single string containing two words separated by a comma. It is not an array containing two strings. You can't iterate over a single string using $.each.
You need to split the string into an array of strings first using String.split():
$.each( data.Genre.split(/\s*,\s*/), function( key, val ) {
This uses a regular expression to split the Genre string at each comma (including possible white space). The result is an array ["Crime", "Drama"] which $.each can iterate over.

Make comma separated list from different paths

I have list of paths as follows in multiple lines.
abc/xyz/../a1.txt
abc/xyz/../a2.txt
abc/xyz/../a3.txt
abc1/xyz/../a4.txt
abc1/xyz/../a5.txt
abc1/xyz/../a6.txt
i want to remove all paths and extract only file names as follows
"a1.txt","a2.txt","a3.txt" and
"a4.txt","a5.txt","a6.txt"
pls help me.

Let's say the list of paths is stored in one string:
var paths= "abc/xyz/../a1.txt"
+ "\n" + "abc/xyz/../a2.txt"
+ "\n" + "abc/xyz/../a3.txt"
+ "\n" + "abc1/xyz/../a4.txt"
+ "\n" + "abc1/xyz/../a5.txt"
+ "\n" + "abc1/xyz/../a6.txt";
You just need to split it by lines and reduce each entry to file name (e.g. by using regex):
var fileNamePattern = /[^\/]+$/;
var fileNames = paths.split("\n").map(function (entry) {
return fileNamePattern.exec(entry)[0];
}
For more information about how to use regex look at this link and here you can find how to use the map method.
Update
Here you can find an example

You could use regular expressions to convert a multi-line string with paths into an object that will have an array of file names per folder, where the folder name is used as the object's property name:
// Test data, with some border cases:
var paths = " abc/xyz/../a1.txt\n" +
"abc/xyz/../a2.txt\r\n" +
" \n" +
" no-folder.txt \n" +
"abc/xyz/../a3.txt \n" +
" abc1/xyz/../a4.txt\n" +
"abc1/xyz/../a5.txt\n" +
"abc1/xyz/../a6.txt\n" +
" ";
var filesPerFolder = paths.split(/\r\n|\n/).reduce(function (filesPerFolder, path) {
path = path.trim();
if (path.length) {
var folder = path.replace(/[^\/]*$/, '');
(filesPerFolder[folder] = filesPerFolder[folder] || []).
push(path.substr(folder.length));
}
return filesPerFolder;
}, {});
filesPerFolder will be
{
"abc/xyz/../": [
"a1.txt",
"a2.txt",
"a3.txt"
],
"": [
"no-folder.txt"
],
"abc1/xyz/../": [
"a4.txt",
"a5.txt",
"a6.txt"
]
}
The above code:
recognises LF or CRLF line breaks
skips white-space around paths
skips empty lines
can handle lines that have file names without folder prefix
Here is a fiddle where you can type the list as input, press the Go button and see the Json encoded filesPerFolder.
There are plenty of ways to work with that object. For instance, you could print out the contents as follows:
Object.keys(filesPerFolder).forEach(function(folder) {
console.log('Contents of folder "' + folder + '": '
+ JSON.stringify(filesPerFolder[folder]));
});
This will output (with the example data above):
Contents of folder "abc/xyz/../": ["a1.txt","a2.txt","a3.txt"]
Contents of folder "": ["no-folder.txt"]
Contents of folder "abc1/xyz/../": ["a4.txt","a5.txt","a6.txt"]
Or, if you want to print each of the files with prefixed folder (which would be a clean version of the input):
Object.keys(filesPerFolder).forEach(function(folder) {
filesPerFolder[folder].forEach(function(file) {
console.log(folder + file);
});
});
Output:
abc/xyz/../a1.txt
abc/xyz/../a2.txt
abc/xyz/../a3.txt
no-folder.txt
abc1/xyz/../a4.txt
abc1/xyz/../a5.txt
abc1/xyz/../a6.txt

Assuming this is string, first store different paths inside an array using split for newline in the string:
arr = str.split(/\n/);
Then loop over array and use pop to store file name in another array(say file_names):
file_names.push(arr[i].split('/').pop())

Example code from Javascript textbook giving unexpected error

Preface - not relevant to problem
First off I am a noob to Javascript and am in the process of learning it. I have read a lot of Eloquent Javascript but towards the end found it to be too advanced for my level. I was recommended to start reading Professional Javascript for Web Developers by Nicholas Zakas.
Problem
Now I'm attempting to just try and run ObjectTypeExample04.htm which I've included here:
function displayInfo(args) {
var output = “”;
if (typeof args.name == “string”){
output += “Name: “ + args.name + “\n”;
}
if (typeof args.age == “number”) {
output += “Age: “ + args.age + “\n”;
}
alert(output);
}
displayInfo({
name: “Nicholas”,
age: 29
});
displayInfo({
name: “Greg”
});
From what I can understand, it'll just print the object defined in the call to displayInfo and it'll do this through appending the output variable inside that function.
I don't get why I'm getting this error
Exception: SyntaxError: illegal character
when running it in a JavaScript code runner (don't know the technical name). But basically I tried using JSFiddle to no luck and I don't understand what's wrong because I copied and pasted it out of a textbook.

Replace the “ ” symbols with standard " quotes and try it again.

You are using LEFT DOUBLE QUOTATION MARK (“) and RIGHT DOUBLE QUOTATION MARK (”) where you should be using QUOTATION MARK (") (or APOSTROPHE (')).
This is typically caused by writing code using a word processor which automatically replaces straight quotes with typographic quotes instead of a regular text editor.

Replace the "smart quotes" by regular double quotes:
function displayInfo(args) {
var output = "";
if (typeof args.name == "string"){
output += "Name: " + args.name + "\n";
}
if (typeof args.age == "number") {
output += "Age: " + args.age + "\n";
}
alert(output);
}
displayInfo({
name: "Nicholas",
age: 29
});
displayInfo({
name: "Greg"
});