Develop Reference

Read/Write and store data internelly in a Local App (no server) with JavaScript

So I am making a local App using Javascript , React and Electron and I want it to be able to work just fine without internet.
I can't use 'localStorage' because the data might get deleted if the user deletes the cache.
I tried reading/writing using differant Modules, none of them worked mostly because of CROS. Using XMLHTTPrequests and Ajax doesn't work either and am running out of time.
When I use them on the test server, they return the index.html for the mainpage (They can at least access that ... and still they can't read the data) but when I try it on the build I get CORS the error.
My Idea for now is to enable CORS on my webpage since I have no worries about security : The App will run ONLY offline so there is no danger.
But After many hours...I didn't find a solution to do it on the client side.
If anyone has an idea or suggestion I would be grateful.
I tried : fs,FileReader,FileSaver, $.ajax,XMLHTTPrequests
//using $ajax
var test = $.ajax({
crossDomain:true,
type: 'GET',
url:'../data/DefaultCategorie.txt',
contentType: 'text/plain',
success: function(data){
console.log(data);
},
error: function(){
alert('failed');
},
})
//using fs
fs.readFile('../data/DefaultCategorie.txt', 'utf8', (err, data) => {
if (err) {
console.log("Failed");
throw err
}
console.log(data);
fs.close(data, (err) => {
if (err) throw err;
});
});

This article covers the 3 most common ways to store user data: How to store user data in Electron
The Electron API for appDatadoes what you want. It is very easy to use.
From the above article:
const userDataPath = (electron.app || electron.remote.app).getPath('userData');
this.path = path.join(userDataPath, opts.configName + '.json')
this.data = parseDataFile(this.path, opts.defaults);
function parseDataFile(filePath, defaults) {
try {
return JSON.parse(fs.readFileSync(filePath));
} catch(error) {
// if there was some kind of error, return the passed in defaults instead.
return defaults;
}
}
Docs
app.getPath(name)
name String
Returns String - A path to a special directory or file associated with
name. On failure, an Error is thrown.
You can request the following paths by the name:
appData - Per-user application data directory, which by default points to:
%APPDATA% on Windows
$XDG_CONFIG_HOME or ~/.config on Linux
~/Library/Application Support on macOS
userData - The directory for storing your app's configuration files,
which by default it is the appData directory appended with your app's
name.

How to use the bottleneck npm module

I just found out about this bottleneck npm module to limit the no of requests per second. I understood the bottleneck() constructor, but cannot understand the submit and schedule() methods, probably because I am a beginner in node and don't know about promise.
Anyway, I couldn't find any examples about using bottleneck from google.
A bottleneck example in basic nodejs and express could help a lot.
Here is the npm package: bottleneck npm module

I'd recommend learning about promises first, and look into request-promise. Here is how you could use it with promises to get info from a simple weather service:
var rp = require("request-promise");
var Bottleneck = require("bottleneck");
// Restrict us to one request per second
var limiter = new Bottleneck(1, 1000);
var locations = ["London","Paris","Rome","New York","Cairo"];
// fire off requests for all locations
Promise.all(locations.map(function (location) {
// set up our request
var options = {
uri: 'https://weatherwebsite.com?location=' + location,
json: true
};
// run the api call. If we weren't using bottleneck, this line would have just been
// return rp(options)
// .then(function (response) {...
//
return limiter.schedule(rp,options)
.then(function (response) {
console.log('Weather data is', response);
})
.catch(function (err) {
// API call failed...
});
});

ajax get to 3rd party server within express middleware results in 500 error

I have a requirement to access multiple devices over IP from an HTML5 web app.
My approach to working around the cross-domain impossibility of doing this all on the client side has been to "cook" requests from the client inside of express' middleware. A route receives a get or post from the client, and then performs a get or post to the 3rd party device identified by the payload from the client.
I'm using the code to get info from a device. It works just fine when I run it directly from file inside of a client I made for testing purposes. Running directly from file avoids the CORS difficulty because the client is also the server I guess.
When I run the same code from within an express route, I get a 500 error.
Am I trying to do something impossible? I'm only about a week into node, express etc so hopefully it's something dumb and easy to solve. I'm taking fact that I haven't been able to find any other questions quite like this as an indication that there's a proper way to achieve what I need.
// post to find a camera
router.post('/find', function(req, res) {
var url = 'http://' + req.body.addr + '/cgi-bin/aw_cam?cmd=QID&res=1';
console.log(url);
$.ajax({
type: 'GET',
url: url,
dataType: 'html',
success: function (result) {
console.log('success: ' + result);
res.send(result);
},
error: function (xhr, textStatus, err) {
console.log('error: ' + textStatus);
}
});
});
Here's what logged to the server console:
http://192.168.0.10/cgi-bin/aw_cam?cmd=QID&res=1
POST /cameras/find 500 126.593 ms - 1656
Thanks in advance!

ok I found how to do this. The trick is to use Node's built-in http messaging capabilities. I found a good article on how to do this here
The code below does exactly what I wanted from within my custom route middleware. I guess I just learned that I can only use AJAX how I wanted on the client side.
This lets me abstract the hairier details of the device control protocol into the server, leaving my client apps to use JSON/AJAX model to interact with them. Success!
var http = require('http');
// post to find a camera
router.post('/find', function(req, res) {
var url = 'http://' + req.body.addr + '/cgi-bin/aw_cam?cmd=QID&res=1';
console.log(url);
http.get(url, (response) => {
console.log(`Got response: ${response.statusCode}`);
var body = '';
response.on ('data', function(d) {
body += d;
});
response.on ('end', function () {
console.log('received: ' + body);
var reply = {};
if (body.indexOf('OID:') == 0) {
reply.msg = body.slice(4);
reply.ok = true;
} else {
reply.msg = body;
reply.ok = false;
}
res.send(reply);
});
// consume response body
response.resume();
}).on('error', (e) => {
console.log(`Got error: ${e.message}`);
});
});

Node.js Scraping ASU Course

I'm pretty new to Node.js, so apologies in advance if I don't know what I'm talking about.
I'm trying to scrape some courses off ASU's course catalog (https://webapp4.asu.edu/catalog/) and have made numerous attempts using Zombie, Node.IO, and the HTTPS api. In both cases I've run into a redirect loop.
I'm wondering if it's because I'm not setting my headers properly?
Below is a sample code of what I used (not Zombie/Node.IO):
var https = require('https');
var option = {
host: 'webapp4.asu.edu',
path: '/catalog',
method: 'GET',
headers: {
'set-cookie': 'onlineCampusSelection=C'
}
};
var req = https.request(options, function(res) {
console.log("statusCode: ", res.statusCode);
console.log("headers: ", res.headers);
res.on('data', function(d) {
process.stdout.write(d);
});
});
Just to clarify, I'm not having trouble with scraping with Node.js in general. More specifically, however, is ASU's course catalog that is giving me trouble.
Appreciate any ideas you guys could give me, thanks!
Update: My request successfully went through if I create a cookie with a JSESSIONID I got from Chrome/FF. Is there a way for me to request/create a JSESSIONID?

It looks like the server sets the JSESSIONID cookie and then redirects away, so you need to tell node.js not to follow redirects if you want to grab the cookie. I don't know how to do this with the http or https packages, but there is another package you can get via npm: request, which lets you do it. Here's a sample that should get you started:
var request = require("request");
var options = {
url: "https://webapp4.asu.edu/catalog/",
followredirect: false,
}
request.get(options, function(error, response, body) {
console.log(response.headers['set-cookie']);
});
Output should look something like this:
[ 'JSESSIONID=B43CC3BB09FFCDE07AE6B3B702717431.catalog1; Path=/catalog; Secure' ]

Id highly recommend using jsDOM in conjunction with jQuery(for node). I've used it many many times for scaping as it makes it super easy.
heres the example from jsdom's readme:
// Count all of the links from the nodejs build page
var jsdom = require("jsdom");
jsdom.env("http://nodejs.org/dist/", [
'http://code.jquery.com/jquery-1.5.min.js'
],
function(errors, window) {
console.log("there have been", window.$("a").length, "nodejs releases!");
});
Hope that helps, jsdom has made it real easy to hack together scraping experiments (for me at least).

Connect to Cloudant CouchDB with Node.js?

I am trying to connect to my CouchDB database on Cloudant using Node.js.
This worked on the shell:
curl https://weng:password#weng.cloudant.com/my_app/_all_docs
But this node.js code didn't work:
var couchdb = http.createClient(443, 'weng:password#weng.cloudant.com', true);
var request = couchdb.request('GET', '/my_app/_all_docs', {
'Host': 'weng.cloudant.com'
});
request.end();
request.on('response', function (response) {
response.on('data', function (data) {
util.print(data);
});
});
It gave me this data back:
{"error":"unauthorized","reason":"_reader access is required for this request"}
How do I do to list all my databases with Node.js?

The built-in Node.js http client is pretty low level, it doesn't support HTTP Basic auth out of the box. The second argument to http.createClient is just a hostname. It doesn't expect credentials in there.
You have two options:
1. Construct the HTTP Basic Authorization header yourself
var Base64 = require('Base64');
var couchdb = http.createClient(443, 'weng.cloudant.com', true);
var request = couchdb.request('GET', '/my_app/_all_docs', {
'Host': 'weng.cloudant.com',
'Authorization': 'Basic ' + Base64.encode('weng:password')
});
request.end();
request.on('response', function (response) {
response.on('data', function (data) {
util.print(data);
});
});
You will need a Base64 lib such as one for node written in C, or a pure-JS one (e.g. the one that CouchDB Futon uses).
2. Use a more high-level Node.js HTTP client
A more featureful HTTP client, like Restler, will make it much easier to do the request above, including credentials:
var restler = require('restler');
restler.get('https://weng.cloudant.com:443/my_app/_all_docs', {
username: 'weng',
password: 'password'
}).on('complete', function (data) {
util.print(data);
});

There are lots of CouchDB modules for Node.js.
node-couch - a CouchDB connector
node-couchdb - A full API implementation
node-couchdb-min - Light-weight client with low level of abstraction and connection pooling.
cradle - a high-level, caching, CouchDB client

Just wanted to add
nano - minimalistic couchdb driver for node.js
to the list. It is written by Nuno Job, CCO of nodejitsu, and actively maintained.

This answer is looking a bit dated. Here is an updated answer that I verified using the following Cloudant Supported NPM Node Client library that works.
https://www.npmjs.com/package/cloudant#getting-started
And to answer his question on how to list his databases use the following code.
//Specify your Cloudant Database Connection URL. For Bluemix format is: https://username:password#xxxxxxxxx-bluemix.cloudant.com
dbCredentials_url = "https://username:password#xxxxxxxxx-bluemix.cloudant.com"; // Set this to your own account
// Initialize the library with my account.
// Load the Cloudant library.
cloudant = require('cloudant')(dbCredentials_url);
// List the Cloudant databases
cloudant.db.list(function(err, allDbs) {
console.log('All my databases: %s', allDbs.join(', ')) });