I tried to add a CSV file into aerospike using nodejs with the put() command. It is showing all records but storing only last record. I need to store the entire CSV file into aerospike using client node js.
client.put(key, rec, function (error) {
if (error) {
console.log('error: %s ', error.message)
}
else {
console.log('Record %d written to database successfully.',count)
}
How to store a CSV file in aerospike using client nide js?
This is a repeated action if I understand it correctly. The problem is that since key remains unchanged, put will override it, the last overrides the penultimate, the penultimate its previous, etc. You will either need to have multiple different keys and use those or inside the loop concatenate your texts and use client.put after the loop, but with the concatenated string.
Another option for loading CSV files would be to use the Aerospike Data Loader tool: https://github.com/aerospike/aerospike-loader.
Related
I am reviewing a nodeJS program someone wrote to merge objects from two files and write the data to a mongodb. I am struggling to wrap my head around how this is working - although I ran it and it works perfectly.
It lives here: https://github.com/muhammad-asad-26/Introduction-to-NodeJS-Module3-Lab
To start, there are two JSON files, each containing an array of 1,000 objects which were 'split apart' and are really meant to be combined records. The goal is to merge the 1st object of both files together, and then both 2nd objects ...both 1,000th objects in each file, and insert into a db.
Here are the excerpts that give you context:
const customerData = require('./data/m3-customer-data.json')
const customerAddresses = require('./data/m3-customer-address-data.json')
mongodb.MongoClient.connect(url, (error, client) => {
customerData.forEach((element, index) => {
element = Object.assign(element, customerAddresses[index])
//I removed some logic which decides how many records to push to the DB at once
var tasks = [] //this array of functions is for use with async, not relevant
tasks.push((callback) => {
db.collection('customers').insertMany(customerData.slice(index, recordsToCopy), (error, results) => {
callback(error)
});
});
})
})
As far as I can tell,
element = Object.assign(element, customerAddresses[index])
is modifying the current element during each iteration - IE the JSON object in the source file
to back this up,
db.collection('customers').insertMany(customerData.slice(index, recordsToCopy)
further seems to confirm that when writing the completed merged data to the database the author is reading out of that original customerData file - which makes sense only if the completed merged data is living there.
Since the source files are unchanged, the two things that are confusing me are, in order of importance:
1)Where does the merged data live before being written to the db? The customerData file is unchanged at the end of runtime.
2)What's it called when you access a JSON file using array syntax? I had no idea you could read files without the functionality of the fs module or similar. The author read files using only require('filename'). I would like to read more about that.
Thank you for your help!
Question 1:
The merged data lives in the customerData variable before it's sent to the database. It exists only in memory at the time insertMany is called, and is passed in as a parameter. There is no reason for anything on the file system to be overwritten -- in fact it would be inefficient to modify that .json file every time you called the database -- storing that information is the job of the database, not a file within your application. If you wanted to overwrite the file, it would be easy enough -- just add something like fs.writeFile('./data/m3-customer-data.json', JSON.stringify(customerData), 'utf8', console.log('overwritten')); after the insertMany. Be sure to include const fs = require('fs');. To make it clearer what is happening, try writing the value of customerData.length to the file instead.
Question 2:
Look at the docs on require() in Node. All it's doing is parsing the data in the JSON file.
There's no magic here. A static json file is parsed to an array using require and stored in memory as the customerData variable. Its values are manipulated and sent to another computer elsewhere where it can be stored. As the code was originally written, the only purpose that json file serves is to be read.
When you launch a rethinkdb instance, it will automatically create a database called 'test'. When you run multiple instances and cluster them using rethinkdb proxy this leads to the issue:
Database name conflict: test is the name of more than one database
If you try to delete the databases i.e using
r.dbDrop('test').run(conn, function(result) {
console.log(result) // Will result in error
});
This will produce the following error:
ReqlOpFailedError: Database 'test' is ambiguous; there are multiple databases with that name in r.dbDrop("test")
So how to prevent RethinkDB from creating the 'test' database automatically? Or how to remove a database if you run into name conflict?
If you use rethinkdb create to create a data directory, no test database will be created.
For example, if rethinkdb_data doesn't exist, this will create it with no test database:
rethinkdb create
rethinkdb
This will do the same, but uses -d to specify the location of the data directory:
rethinkdb create -d /var/lib/rethinkdb/data
rethinkdb -d /var/lib/rethinkdb/data
After digging for a while I didn't find any good options to prevent RethinkDB from trying to create the default test database on startup. The problem above only occurs when the cluster nodes use different data directories. Otherwise they will not try to create extra test databases as the other nodes will simply recognise it already exists (created by the first node that launched).
I ended up solving this in my backend software by enumerating all database named test during startup from the db_config table in rethinkdb database.
Example:
// Read from the database 'rethinkdb' that holds information about other databases
r.db("rethinkdb")
// Table db_config contains database id, name information
.table("db_config")
// Filter by name 'test'
.filter({name: "test"})
.run(conn, function(err, cursor) {
// Get results from cursor by id and rename
cursor.each(function(err, result){
r.db("rethinkdb")
.get(result.id)
.update({name: "other_name"})
.run(conn, function(err, result) {
console.log(result.replaced);
});
});
});
I'm trying to push a new object into an external Javascript document but I am having problems pulling in the JSON file to write to. I have an external, local file called check.json.
How do I call the external json file correctly in node?
var newData = JSON.parse(check.json);
newData.check.push({
cheap: $el.text()
});
check.json = JSON.stringify(newData);
You can use the Filesystem object to read and write to files. Specifically, the readFile and writeFile methods.
For example, to read:
fs.readFile('/path/to/my/json.json', function (err, data) {
if (err) throw err;
var newData = JSON.parse(data);
});
That said, flat files are not a good format for storing data and being accessed like a database. You can run into race conditions and lose data. You would be better off with a real database that will take case of that sort of thing for you.
SQLite gives you a simple file (but puts lots of protection in around it). If you really wanted JSON as the storage format then you could look at something like couchDB.
This is how I append data to a file:
self.fs.appendFile(targetFilePath, new Buffer(sourceData, 'base64'), function(err) { ... });
How can I remove data from a file that have been appended? This is required if the stream I receive earlier is interrupted and parts have been written.
Is it possible to undo the last append command?
No, you can't undo anything.
You can truncate the file to a known point (length...):
self.fs.truncateSync(targetFilePath, somePoint);
Or you could originally not write what you're unsure of, just aggregate to memory or a temporary file until you're certain.
I am using dropbox-js to access a user's dropbox files. Please note that I am very new to this...
What I need to do, is to access existing files using dropbox-js. As I understand it, the pullChanges method should be used for this. This is how I currently have it:
client.pullChanges(null, function(error, results) {
if (error) {
return showError(error);
}
alert("Dropbox contains: " + results.shouldPullAgain); //just to test that something happens
});
I'm not sure if this is the correct way to make use of pullChanges, and then I need to know how to use results to actually get the file/folder information??