MongoImport csv combine/concat various columns to one array for import - javascript

I have another interesting case which I have never faced before, so I'm asking help from SO community and also share my experience with it.
The case || What we have:
A csv file (exported from other SQL DB) with such structure
(headers):
ID,SpellID,Reagent[0],Reagent[1..6]Reagent[7],ReagentCount[0],ReagentCount[1..6],ReagentCount[7]
You could also check a full -csv data file here, at my
dropbox
My gist from Github, which helps you to understand how MongoImport works.
What we need:
I'd like to receive such structure(schema) to import it into MongoDB collection:
ID(Number),SpellID(Number),Reagent(Array),ReagentCount(Array)
6,898,[878],[1]
with ID, SpellID, and two arrays, in first we store all Reagent IDs, like [0,1,2,3,4,5,6,7] from all Reagent[n] columns, and in the second array we have the array with the same length that represent quantity of ReagentIDs, from all ReagentCount[n]
OR
A transposed objects with such structure (schema):
ID(Number),SpellID(Number),ReagentID(Number),Quantity/Count(Number)
80,2675,1,2
80,2675,134,15
80,2675,14,45
As you may see, the difference between the first example and this one, that every document in the collection represents each ReagentID and it's quantity to SpellID. So if one Spell_ID have N different reagents it will be N documents in the collection, cause we all know, that there can't be more then 7 unique Reagent_ID belonging to one Spell_ID according to our -csv file.
I am working on this problem right now, with the help of node js and npm i csv (or any other modules for parsing csv files). Just to make my csv file available for importing to my DB via mongoose. I'll be very thankful for all those, who could provide any relevant contribution to this case. But anyway, I will solve this problem eventually and share my solution in this question.
As for the first variant I guess there should be one-time script for MongoImport that could concat all columns from Reagent[n] & ReagentCount[n] to two separate arrays like I mentioned above, via -fields but unfortunately I don't know it, and there are no examples on SO or official Mongo docs relevant to it. So if you have enough experience with MongoImport feel free to share it.


Finally I solve my problem as I want it to, but without using mongoimport
I used npm i csv and write function for parsing my csv file. In short:
async function FuncName (path) {
try {
let eva = fs.readFileSync(path,'utf8');
csv.parse(eva, async function(err, data) {
//console.log(data[0]); we receive headers, if they exist
for (let i = 1; i < data.length; i++) { //we start from 1, because 0 is headers, if we don't have it, then we start from 0
console.log(data[i][34]); //where i is row number and j(34) is a header address
}
});
} catch (err) {
console.log(err);
}
}
It loops over csv file and shows data in array that allows you to operate with them as you want it to.

Related

PostgreSQL: INSERT Error while reading data from CSV File

I am inserting values from a CSV file into a postgresql table, the code used to work fine earlier, but now that I'm on my local machine, it refuses despite so many different attempts.
const query =
"INSERT INTO questions VALUES (DEFAULT,$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12)";
questionData.forEach((row) => {
questions.push(
db.query(query, row).catch((err) => {
console.log(err);
})
);
});
This is my insertion logic, the questionData just holds every row of the CSV file, and questions is the array of promises which I Promise.all() in the end.
The Error I get is in this link
I am going crazy trying to fix this, I have changed absolutely nothing in the backend, my CSV files have only 12 rows which are the one's I'm trying to insert.
Edit:
What is 5+3,qwerty,mcq,chemistry,cat,cat,easy,2001,FALSE,nah,{8},"{7,2,8,3}"
What is 5+4,qwerty,mcq,maths,cat,cat,easy,2002,FALSE,nah,{9},"{7,9,5,3}"
What is 5+5,qwerty,mcq,physics,cat,cat,easy,2003,FALSE,nah,{10},"{7,2,10,3}"
What is 5+6,qwerty,mcq,chemistry,cat,cat,easy,2004,FALSE,nah,{11},"{11,2,5,3}"
What is 5+7,qwerty,mcq,maths,cat,cat,easy,2005,FALSE,nah,{12},"{7,2,12,3}"
What is 5+8,qwerty,mcq,physics,cat,cat,easy,2006,FALSE,nah,{13},"{13,2,5,3}"
What is 5+9,qwerty,mcq,chemistry,cat,cat,easy,2007,FALSE,nah,{14},"{7,14,5,3}"
What is 5+10,qwerty,mcq,maths,cat,cat,easy,2008,FALSE,nah,{15},"{7,2,15,3}"
This is my CSV

The error states that you're trying to push more values than the columns in your table.
i see your insert statement has a DEFAULT first parameter... what is that about?
If your target table has only 12 columns then you should be inserting the following
INSERT INTO questions VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12)

Pentaho/Kettle - Javascript or java that gets file names older than a specified date

Please excuse the rookie question as I'm not a programmer :)
We're using Pentaho 8
I'm looking for a way to have Javascript or Java read a directory and return the file names of any files that are older than a date that will be provided by a Pentaho parameter.
Here is what I currently have using a Modified Java Script Value step that only lists the directory contents:
var _getAllFilesFromFolder = function(dir) {
var filesystem = require("fs");
var results = [];
filesystem.readdirSync(dir).forEach(function(file) {
file = dir+'\'+file;
var stat = filesystem.statSync(file);
if (stat && stat.isDirectory()) {
results = results.concat(_getAllFilesFromFolder(file))
} else results.push(file);
});
return results;
};
Is Javascript/Java the right way to do this?

There's a step called "Get file names". You just need to provide the path you want to poll. It also allows doing so recursively, only showing filenames that match a given filter, and in the filters tab allow you to show only folders, only files, or both.

nsousa's answer would be the easiest, then after you get your file list you can use a filter rows step on the lastmodifiedtime returned from the Get file names. 2 -steps, 3 if you want to format the date/time returned to something easier to sort/filter through. This is the approach I use and its is faster then the transformations can keep up with generally.

How to load multiple files with Queue.js and D3.js?

Situation
I am trying to load multiple xml files (located on server) without the need to declare the name of the files hard coded. For this I am trying to use the d3.queue library https://github.com/d3/d3-queue.
I have implemented the xml to force layout to my own needs (https://bl.ocks.org/mbostock/1080941), but there is one crucial flaw namely I need to manually type in the name of the xml file that I want to load...
Reproduce
Given (adjusted example from http://learnjsdata.com/read_data.html) :
queue()
.defer(d3.xml, "/mappings/Customer.hbm.xml")
.defer(d3.xml, "/mappings/Actor.hbm.xml")
.await(analyze);
function analyze(error, Customer, Actor) {
if(error) { console.log(error); }
// do stuff with Customer data, do stuff with Actor data
}
And given my implementation of the processing of an xml:
d3.xml("mappings/Customer.hbm.xml","application/xml", function(error,xml){
if (error) throw error;
// do stuff with the data retrieved from Customer.hbm.xml
});
Question
How do I combine above two snippets in such a way that I dont have to write the locations of the xml hard coded and pass all the parameters to the analyze function? Any nudge in the right direction would be much appreciated.
In psuedocode I have tried to code something like the following (but I cant get it to work):
function to get all names of the xmls from the mappings folder (probably with node.js fs.readdir or fs.readdirSync methods, but I am unsure of how that would work exactly)
for each xml .defer(d3.xml, nameofxml)
pass all the found names as parameters to the analyze function
In Java I would have chosen to do this with a var...args but I dont know how to do it in JS.

There's really two parts to this question:
How do I get a list of server-side files to client-side JavaScript?
Short answer is you don't without having a server-side api that can return that list. Depending on what backend you are using, you write a method that returns a JSON array of the files in your target directory. You call this first, get the response and then process them all with queue:
d3.json('/get/list/of/xml/files', function(error, fileArray){
var q = d3.queue();
fileArray.forEach(function(d){
q = q.defer(d3.xml, d);
});
q.await(analyze);
});
How do a process a variable number of arguments in JavaScript?
This is actually very well supported in JavaScript.
function analyze(error) {
if(error) { console.log(error); }
// skip 0 it's error variable
for (i = 1; i < arguments.length; i++) {
var xml = arguments[i];
...
}
}

MongoDb bulk insert limit issue

Im new with mongo and node. I was trying to upload a csv into the mongodb.
Steps include:
Reading the csv.
Converting it into JSON.
Pushing it to the mongodb.
I used 'csvtojson' module to convert csv to json and pushed it using code :
MongoClient.connect('mongodb://127.0.0.1/test', function (err, db) { //connect to mongodb
var collection = db.collection('qr');
collection.insert(jsonObj.csvRows, function (err, result) {
console.log(JSON.stringify(result));
console.log(JSON.stringify(err));
});
console.log("successfully connected to the database");
//db.close();
});
This code is working fine with csv upto size 4mb; more than that its not working.
I tried to console the error
console.log(JSON.stringify(err));
it returned {}
Note: Mine is 32 bit system.
Is it because there a document limit of 4mb for 32-bit systems?
I'm in a scenario where I can't restrict the size and no.of attributes in the csv file (ie., the code will be handling various kinds of csv files). So how to handle that? I there any modules available?

If you are not having a problem on the parsing the csv into JSON, which presumably you are not, then perhaps just restrict the list size being passed to insert.
As I can see the .csvRows element is an array, so rather than send all of the elements at once, slice it up and batch the elements in the call to insert. It seems likely that the number of elements is the cause of the problem rather than the size. Splitting the array up into a few inserts rather than 1 should help.
Experiment with 500, then 1000 and so on until you find a happy medium.
Sort of coding it:
var batchSize = 500;
for (var i=0; i<jsonObj.csvRows.length; i += batchSize) {
var docs = jsonObj.csvRows.slice(i, i+(batchSize -1));
db.collection.insert( docs, function(err, result) {
// Also don't JSON covert a *string*
console.log(err);
// Whatever
}
}
And doing it in chunks like this.

You can make those data as an array of elements , and then simply use the MongoDB insert function, passing this array to the insert function

How go I get csv data into netsuite?

I've got an update to my question.
What I really wanted to know was this:
How do I get csv data into netsuite?
Well, it seems I use the csv import tool to create a mapping and use this call to import the csv nlapiSubmitCSVImport(nlobjCSVImport).
Now my question is: How do I iterate through the object?!
That gets me half way - I get the csv data but I can't seem to find out how I iterate through it in order to manipulate the date. This is, of course, the whole point of a scheduled script.
This is really driving me mad.
#Robert H
I can think of a million reasons why you'd want to import data from a CSV. Billing, for instance. Various reports on data any company keeps and I wouldn't want to keep this in the file cabinet nor would I really want to keep the file at all. I just want the data. I want to manipulate it and I want to enter it.

Solution Steps:
To upload a CSV file we have to use a Suitelet script.
(Note: file - This field type is available only for Suitelets and will appear on the main tab of the Suitelet page. Setting the field type to file adds a file upload widget to the page.)
var fileField = form.addField('custpage_file', 'file', 'Select CSV File');
var id = nlapiSubmitFile(file);
Let's prepare to call a Restlet script and pass the file id to it.
var recordObj = new Object();
recordObj.fileId = fileId;
// Format input for Restlets for the JSON content type
var recordText = JSON.stringify(recordObj);//stringifying JSON
// Setting up the URL of the Restlet
var url = 'https://rest.na1.netsuite.com/app/site/hosting/restlet.nl?script=108&deploy=1';
// Setting up the headers for passing the credentials
var headers = new Array();
headers['Content-Type'] = 'application/json';
headers['Authorization'] = 'NLAuth nlauth_email=amit.kumar2#mindfiresolutions.com, nlauth_signature=*password*, nlauth_account=TSTDRV****, nlauth_role=3';
(Note: nlapiCreateCSVImport: This API is only supported for bundle installation scripts, scheduled scripts, and RESTlets)
Let's call the Restlet using nlapiRequestURL:
// Calling Restlet
var output = nlapiRequestURL(url, recordText, headers, null, "POST");
Create a mapping using Import CSV records available at Setup > Import/Export > Import CSV records.
Inside the Restlet script Fetch the file id from the Restlet parameter. Use nlapiCreateCSVImport() API and set its mapping with mapping id created in step 3. Set the CSV file using the setPrimaryFile() function.
var primaryFile = nlapiLoadFile(datain.fileId);
var job = nlapiCreateCSVImport();
job.setMapping(mappingFileId); // Set the mapping
// Set File
job.setPrimaryFile(primaryFile.getValue()); // Fetches the content of the file and sets it.
Submit using nlapiSubmitCSVImport().
nlapiSubmitCSVImport(job); // We are done
There is another way we can get around this although neither preferable nor would I suggest. (As it consumes a lot of API's if you have a large number of records in your CSV file.)
Let's say that we don't want to use the nlapiCreateCSVImport API, so let's continue from the step 4.
Just fetch the file Id as we did earlier, load the file, and get its contents.
var fileContent = primaryFile.getValue();
Split the lines of the file, then subsequently split the words and store the values into separate arrays.
var splitLine = fileContent.split("\n"); // Splitting the file on the basis of lines.
for (var lines = 1,count=0; lines < splitLine.length; lines++)
{
var words = (splitLine[lines]).split(","); // words stores all the words on a line
for (var word = 0; word < words.length; word++)
{
nlapiLogExecution("DEBUG", "Words:",words[word]);
}
}
Note: Make sure you don't have an additional blank line in your CSV file.
Finally create the record and set field values from the array that we created above.
var myRec = nlapiCreateRecord('cashsale'); // Here you create the record of your choice
myRec.setFieldValue('entity', arrCustomerId[i]); // For example, arrCustomerId is an array of customer ID.
var submitRec = nlapiSubmitRecord(myRec); // and we are done

fellow NetSuite user here, I've been using SuiteScripts for a while now but never saw nlobjCSVImport object nor nlapiSubmitCSVImport .. I looked in the documentation, it shows, but there is no page describing the details, care to share where you got the doc from?
With the doc for the CSVImport object I might be able to provide some more help.
P.S. I tried posting this message as a comment but the "Add comment" link didn't show up for some reason. Still new to SOF

CSV to JSON:
convert csv file to json object datatable
https://code.google.com/p/jquery-csv/
If you know the structure of the CSV file, just do a for loop and map the fields to the corresponding nlapiSetValue.
Should be pretty straightforward.

Categories

Resources