Develop Reference

split csv where fields contain commas

Thanks to the answer posted here: Put data from a csv file into an array (Javascript) I have been able to split a csv file in the manner I need to but I have come across a problem with the output. Nothing to do with the code as it works just as I wanted it to. The issue relates to the source data.
The csv files are provided to me as is so I did not realise that they had fields that contained commas. This means the split doesn't work as needed.
This is the code I am using:
$.get("export.csv", function process(dataString) {
var lines = dataString
.split(/\n/)
.map(function(lineStr) {
return lineStr.split(",");
});
var keys = lines[0];
var objects = lines
.slice(1)
.map(function(arr) {
return arr.reduce(function(obj, val, i) {
obj[keys[i]] = val;
return obj;
}, {});
});
console.log(objects);
})
This gives me the output in this format:
{
"PROBLEM_NUMBER": "ticket_number",
"CALLER_NAME": "\"surname",
"PRIORITY": " forename\"",
"CALL_TIME": "4",
"CALL_DETAILS": "date",
"RESOLVER": "group",
"RESTORING_GROUP": "\"surname",
"RESOLVING_GROUP": " forename\"",
"RESTORATION_TIME": "group",
"RAG_STATUS": "group",
"CALL_STATUS": "date",
"CALL_TYPE": "RED",
"RESTORATION_CODE": "Closed",
"SUBMITTER_GROUP": "Problem",
"ASSIGNEE_GROUP": "resolution",
"ASSIGNEE_NAME": "group",
"RESOLVED_DATE_TIME": "group",
"RESTORED_DATE_TIME": "",
"TIME_WITH_TEAM": "date",
"MONTH/YEAR\r": "date",
"undefined": "Jan-21\r"
}
As you can see, the final field is "undefined" due to 2 fields containing a comma and splitting incorrectly.
I know I need to use regex to modify the split correctly however I don't understand or know where to put it. Is anyone able to assist me please?
Thanks

I managed to solve the problem so thought I would post here for anyone else that may come across a similar issue:
To resolve it I declared the regex string as a variable and then called that in the .split() instruction.
The regex string I used was - /("[^"]+"|[^,]+)*,/g
My code now looks like this:
$.get("export.csv", function process(dataString) {
var regex = /("[^"]+"|[^,]+)*,/g;
var lines = dataString
.split(/\n/)
.map(function(lineStr) {
return lineStr.split(regex);
});
var keys = lines[0];
var objects = lines
.slice(1)
.map(function(arr) {
return arr.reduce(function(obj, val, i) {
obj[keys[i]] = val;
return obj;
}, {});
});
console.log(objects);
})
This gives me the correct output I need and maps all values accordingly

How can I merge individual object values?

I have the following problem:
I want to read out my table names from a SQL database and then make a comparison as to whether it already exists. I know there is the formula IF EXISTS ... but IF doesn't work .. So here's my previous variant:
First i extracted everything before the filename.csv (C:\Users\Frederic\Desktop\Drag&Drop...) and then the ".csv". Do not be surprised why 51;) The filename is so long
var filename = filePath.slice(51);
var richtigername = filename.replace(".csv","").toString();
console.log(richtigername)
here the result in the console:
for example: fxbewertung
As a second step I let me show the file names:
connection.query('Show tables from datein', function(err, datein) {
let string = JSON.stringify(datein);
let json = JSON.parse(string);
console.log(json)
here the result in the console:
[ { Tables_in_datein: 'fxbewertung' },
{ Tables_in_datein: 'kontoauszug' },
{ Tables_in_datein: 'lsreport' } ]
Furthermore, I extracted the values (name of the SQL tables):
for (var k = 0; k < json.length; k++) {
var werte = Object.values(json[k])
console.log(werte)
};
here the result in the console:
[ 'fxbewertung' ]
[ 'kontoauszug' ]
[ 'lsreport' ]
Now I don't know how i can take the comparison that for example for the file fxbewertung exist a database ('fxbewertung').
My consideration is to somehow browse the individual arrays .. or merge and then browse. At the end should come out true or false
P.S .: it may well be complicated yet but I'm not a real programmer or something;)
Best regards
Frederic

You can use some() method to check if a table exists for that filename.
var tableExists = tables.some(item => item['Tables_in_datein'] === filename);
Live Example:
var tables = [{
Tables_in_datein: 'fxbewertung'
},
{
Tables_in_datein: 'kontoauszug'
},
{
Tables_in_datein: 'lsreport'
}
];
var filename = 'fxbewertung';
var tableExists = tables.some(item => item['Tables_in_datein'] === filename);
if (!tableExists) {
// Table not found for filename.
} else {
// Table found. Do something.
}

Assuming you finished executing your query and stored the data as following:
const queryResult = [ { Tables_in_datein: 'fxbewertung' },
{ Tables_in_datein: 'kontoauszug' },
{ Tables_in_datein: 'lsreport' } ]
You'll then need to map this array to extract the values and store them in a single array like so:
const values = queryResult.map(e=>e[Object.keys(e)[0]]) // ["fxbewertung", "kontoauszug", "lsreport"]
Since you're looking for a true/false result by giving a file name, you'll need to use indexOf to achieve that.
const valueExists = filename => values.indexOf(filename) !== -1
After that execute valueExists with the file name you're looking for:
valueExists("kontoauszug"); // true
valueExists("potato"); // false
Hope this helps!

An efficient solution could be to use Array.prototype.find(). Where it would return from the moment it finds a truthy value and would not iterate till the end (unless the match exists at the end).
Demo Code:
const tablesArr = [
{
Tables_in_datein: "fxbewertung"
},
{
Tables_in_datein: "kontoauszug"
},
{
Tables_in_datein: "lsreport"
}
];
const tabletoFind = "fxbewertung";
const tableFound = tablesArr.find(item => item["Tables_in_datein"] === tabletoFind) ? true: false;
console.log(tableFound);
if(tableFound){
//*yes table found*/
}else{
///*nope table not found*/
}

How can I reformat this simple JSON so it doesn't catch "Circular structure to JSON" exception?

Introduction
I'm learning JavaScript on my own and JSON its something along the path. I'm working on a JavaScript WebScraper and I want, for now, load my results in JSON format.
I know I can use data base, server-client stuff, etc to work with data. But I want to take this approach as learning JSON and how to parse/create/format it's my main goal for today.
Explaining variables
As you may have guessed the data stored in the fore mentioned variables comes from an html file. So an example of the content in:
users[] -> "Egypt"
GDP[] -> "<td> $2,971</td>"
Regions[] -> "<td> Egypt </td>"
Align[] -> "<td> Eastern Bloc </td>"
Code
let countries = [];
for(let i = 0; i < users.length; i++)
{
countries.push( {
'country' : [{
'name' : users[i],
'GDP' : GDP[i],
'Region' : regions[i],
'Align' : align[i]
}]})
};
let obj_data = JSON.stringify(countries, null, 2);
fs.writeFileSync('countryballs.json', obj_data);
Code explanation
I have previously loaded into arrays (users, GDP, regionsm align) those store the data (String format) I had extracted from a website.
My idea was to then "dump" it into an object with which the stringify() function format would format it into JSON.
I have tested it without the loop (static data just for testing) and it works.
Type of error
let obj_data = JSON.stringify(countries, null, 2);
^
TypeError: Converting circular structure to JSON
--> starting at object with constructor 'Node'
| property 'children' -> object with constructor 'Array'
| index 0 -> object with constructor 'Node'
--- property 'parent' closes the circle
What I want from this question
I want to know what makes this JSON format "Circular" and how to make this code work for my goals.
Notes
I am working with Node.js and Visual Studio Code
EDIT
This is further explanation for those who were interested and thought it was not a good question.
Test code that works
let countries;
console.log(users.length)
for(let i = 0; i < users.length; i++)
{
countries = {
country : [
{
"name" : 'CountryTest'
}
]
}
};
let obj_data = JSON.stringify(countries, null, 2);
fs.writeFileSync('countryballs.json', obj_data);
});
Notice in comparison to the previous code, right now I am inputing "manually" the name of the country object.
This way absolutely works as you can see below:
Now, if I change 'CountryTest' to into a users[i] where I store country names (Forget about why countries are tagged users, it is out of the scope of this question)
It shows me the previous circular error.
A "Partial Solution" for this was to add +"" which, as I said, partially solved the problem as now there is not "Circular Error"
Example:
for(let i = 0; i < users.length; i++)
{
countries = {
country : [
{
"name" : users[i]+''
}
]
}
};
Resulting in:
Another bug, which I do not know why is that only shows 1 country when there are 32 in the array users[]
This makes me think that the answers provided are not correct so far.
Desired JSON format
{
"countries": {
"country": [
{
"name": "",
"GDP" : "",
"Region" : "",
"Align" : ""
},
{
"name": "",
"GDP" : "",
"Region" : "",
"Align" : ""
},
{
"name": "",
"GDP" : "",
"Region" : "",
"Align" : ""
}
]}
}

Circular structure error occurs when you have a property of the object which is the object itself directly (a -> a) or indirectly (a -> b -> a).
To avoid the error message, tell JSON.stringify what to do when it encounters a circular reference. For example, if you have a person pointing to another person ("parent"), which may (or may not) point to the original person, do the following:
JSON.stringify( that.person, function( key, value) {
if( key == 'parent') { return value.id;}
else {return value;}
})
The second parameter to stringify is a filter function. Here it simply converts the referred object to its ID, but you are free to do whatever you like to break the circular reference.
You can test the above code with the following:
function Person( params) {
this.id = params['id'];
this.name = params['name'];
this.father = null;
this.fingers = [];
// etc.
}
var me = new Person({ id: 1, name: 'Luke'});
var him = new Person( { id:2, name: 'Darth Vader'});
me.father = him;
JSON.stringify(me); // so far so good
him.father = me; // time travel assumed :-)
JSON.stringify(me); // "TypeError: Converting circular structure to JSON"
// But this should do the job:
JSON.stringify(me, function( key, value) {
if(key == 'father') {
return value.id;
} else {
return value;
};
})
The answer is from StackOverflow question,
Stringify (convert to JSON) a JavaScript object with circular reference

From your output, it looks as though users is a list of DOM nodes. Rather than referring to these directly (where there are all sort of possible cyclical structures), if you just want their text, instead of using users directly, try something like
country : [
{
"name" : users[i].textContent // maybe also followed by `.trim()
}
]
Or you could do this up front to your whole list:
const usersText = [...users].map(node => node.textContent)
and then use usersText in place of users as you build your object.
If GDP, regions and align are also references to your HTML, then you might have to do the same with them.

EUREKA!
As some of you have mentioned above, let me tell you it is not a problem of circularity, at first..., in the JSON design. It is an error of the data itself.
When I scraped the data it came in html format i.e <td>whatever</td>, I did not care about that as I could simply take it away later. I was way too focused in having the JSON well formatted and learning.
As #VLAZ and #Scott Sauyezt mentioned above, it could be that some of the data, if it is not well formatted into string, it might be referring to itself somehow as so I started to work on that.
Lets have a look at this assumption...
To extract the data I used the cheerio.js which gives you a kind of jquery thing to parse html.
To extract the name of the country I used:
nullTest = ($('table').eq(2).find('tr').eq(i).find('td').find('a').last());
//"Partial solution" for the OutOfIndex nulls
if (nullTest != null)
{
users.push(nullTest);
}
(nullTest helps me avoid nulls, I will implement some RegEx when everything works to polish the code a bit)
This "query" would output me something like:
whatEverIsInHereIfThereIsAny
or else.
to get rid off this html thing just add .html() at the end of the "jquery" such as:
($('table').eq(2).find('tr').eq(i).find('td').find('a').last().html());
That way you are now working with String and avoiding any error and thus solves this question.

Get first word of string inside array - from return REST

I try get the sessionid before REST function, but in the case if I does not convert toString(); show only numbers (21 22 2e ...).
See this image:
1º:
Obs.: Before using split.
!!xxxxxxx.xxxxx.xxxxxxx.rest.schema.xxxxResp {error: null, sessionID: qdaxxxxxxxxxxxxxj}
My code:
var Client = require('./lib/node-rest-client').Client;
var client = new Client();
var dataLogin = {
data: { "userName":"xxxxxxxx","password":"xxxxxxxx","platform":"xxxxx" },
headers: { "Content-Type": "application/json" }
};
client.registerMethod("postMethod", "xxxxxxxxxxx/login", "POST");
client.methods.postMethod(dataLogin, function (data, response) {
// parsed response body as js object
// console.log(data); all return, image 1
// raw response
if(Buffer.isBuffer(data)){
data = data.toString('utf8'); // if i does not convert to string, return numbers, see image 1..
console.log(data); //all inside image 2, and i want just value from sessionid
var output = data;
var res = output.split(" "); // using split
res = res[4].split("}", 1);
}
console.log(res); //image 3
});
I tested with JSON.parse and JSON.stringify and it did not work, show just 'undefined' for all. After convert toString();, And since I've turned the values ​​into string, I thought of using split to get only the value of sessionid.
And when I used split, all transform to array and the return is from console.log(data), see image 2:
2º:
Obs.: After use split and convert to array automatically.
And the return after use split is with the conditions inside my code:
3º:
And the return after use split is with the conditions inside my code:
[ 'bkkRQxxxxxxxxxxxxx' ]
And I want just:
bkkRQxxxxxxxxxxxxx
I would like to know how to solve this after all these temptations, but if you have another way of getting the sessionid, I'd be happy to know.
Thanks advance!

After converting the Buffer to a string, remove anything attached to the front with using data.substr(data.indexOf('{')), then JSON.parse() the rest. Then you can just use the object to get the sessionID.
if(Buffer.isBuffer(data)){
data = data.toString('utf8');
data = data.substr(data.indexOf('{'));
obj = JSON.parse(data);
console.log(obj.sessionID);
}
EDIT:
The issue you are having with JSON.parse() is because what is being returned is not actually JSON. The JSON spec requires the properties to be quoted ("). See this article
If the string looked like this, it would work: {"error": null, "sessionID": qdaxxxxxxxxxxxxxj}
Because the json is not really json, you can use a regular expression to get the info you want. This should get it for you.
re = /(sessionID: )([^,}]*)/g;
match = re.exec(data);
console.log(match[2]);
EDIT 2: After fully reading the article that I linked above (oops haha), this is a more preferable way to deal with unquoted JSON.
var crappyJSON = '{ somePropertyWithoutQuotes: "theValue!" }';
var fixedJSON = crappyJSON.replace(/(['"])?([a-zA-Z0-9_]+)(['"])?:/g, '"$2": ');
var aNiceObject = JSON.parse(fixedJSON);

JSON check if no data is available

I am using this code to get data from json.
$.each(data.id, function(index, value){
output += '<li>'+value.title+'</li>';
}
$('#listview').html(output).listview('refresh');
Sometimes there are no records to display.
How can I know if there are no records and display an alert example: alert('No records'); ?

I am assuming that the data.id is an array, since you used$.each on it:
if (data.id.length == 0) {
// no data
}

In general use this code in javascript to check if an array is not empty :
if (myArray.length) { // javascript shortcut for data.id.length>0
alert('no elems found');
}
In your specific case the following code should works :
if (data.id.length) {
$.each(data.id, function(index, value){
output += '<li>'+value.title+'</li>';
});
$('#listview').html(output).listview('refresh');
} else {
alert('no records found ');
}
Hope this helps.

The most general way to check if there is data in object is to use the
length
method.
But you should be very careful. For example if you data is in the following format:
SourceData =[
{
ID:1,
Title:'Test1',
},
{
ID:2,
Title:'Test2',
},
{
ID:3,
Title:'Test3',
},
]
And you do the following check:
if(SourceData.ID.length>0)
and because of some reason(error on server,request timeout,etc) the "SourceData" is not defined, an JavaScript error is going to be thrown.
That's why I am using combination of conditions:
if(typeof SourceData==='undefined'|typeof SourceData.ID==='undefined'|SourceData.ID.length==0)
Then if there is something wrong with the "SoruceData" (it is undefined) a message "No Data" will be shown. Or when the "SourceData" is initialized, but for example someone have changed the structure of the return data from the server (it is no more "ID", it is now "TitleID") no error will be generated.
Note, that I am using "|" instead of "||". This mean, if one of the conditions is "true", the right of him condiations, will not be executed.