I just found out about this bottleneck npm module to limit the no of requests per second. I understood the bottleneck() constructor, but cannot understand the submit and schedule() methods, probably because I am a beginner in node and don't know about promise.
Anyway, I couldn't find any examples about using bottleneck from google.
A bottleneck example in basic nodejs and express could help a lot.
Here is the npm package: bottleneck npm module
I'd recommend learning about promises first, and look into request-promise. Here is how you could use it with promises to get info from a simple weather service:
var rp = require("request-promise");
var Bottleneck = require("bottleneck");
// Restrict us to one request per second
var limiter = new Bottleneck(1, 1000);
var locations = ["London","Paris","Rome","New York","Cairo"];
// fire off requests for all locations
Promise.all(locations.map(function (location) {
// set up our request
var options = {
uri: 'https://weatherwebsite.com?location=' + location,
json: true
};
// run the api call. If we weren't using bottleneck, this line would have just been
// return rp(options)
// .then(function (response) {...
//
return limiter.schedule(rp,options)
.then(function (response) {
console.log('Weather data is', response);
})
.catch(function (err) {
// API call failed...
});
});
Related
I want to make a GET request to a web site using the Amazon Profile API. I am trying to do what is described in the last code chunk in this article: https://developer.amazon.com/blogs/post/Tx3CX1ETRZZ2NPC/Alexa-Account-Linking-5-Steps-to-Seamlessly-Link-Your-Alexa-Skill-with-Login-wit (very end of article) and it just does not happen. My callback function never seems to get called.
I have added the required context.succeed(), actually the latest version of that, and am still not getting results. I know the url is good, as I can take it and copy/paste into a browser and it returns an expected result.
Here is a SO answer on using the appropriate context function calls within the callback, which I have tried. Why is this HTTP request not working on AWS Lambda?
I am not using a VPC.
What am I doing wrong? I feel like a moron, as I have been researching this and trying solutions for 2 days. I log the full URL, and when I copy/paste that out of the log file, and put it in a browser window, I do get a valid result. Thanks for your help.
Here is the code:
function getUserProfileInfo(token, context) {
console.log("IN getUserProfileInfo");
var request = require('request');
var amznProfileURL = 'https://api.amazon.com/user/profile?access_token=';
amznProfileURL += token;
console.log("calling it");
console.log(amznProfileURL);
console.log("called it");
request(amznProfileURL, function(error, response, body) {
if (!error && response.statusCode == 200) {
var profile = JSON.parse(body);
console.log("IN getUserProfileInfo success");
console.log(profile);
context.callbackWaitsForEmptyEventLoop = false;
callback(null, 'Success message');
} else {
console.log("in getUserProfileInfo fail");
console.log(error);
context.callbackWaitsForEmptyEventLoop = false;
callback('Fail object', 'Failed result');
}
});
console.log("OUT getUserProfileInfo");
}
This is the logging output I get in CloudWatch:
2017-03-08T22:20:53.671Z 7e393297-044d-11e7-9422-39f5f7f812f6 IN getUserProfileInfo
2017-03-08T22:20:53.728Z 7e393297-044d-11e7-9422-39f5f7f812f6 OUT getUserProfileInfo
Problem might be that you are using var request = require('request'); which is external dependency and will require you to make a packaged lambda deployment for it to work. See this answer for relevant information.
AWS Node JS with Request
Another way is that you can use NodeJS module such as var http= require('http'); which is builtin module to make the requests. This way you can just make plain lambda script deployment.
Reference
http://docs.aws.amazon.com/lambda/latest/dg/nodejs-create-deployment-pkg.html
I use the following lib to connect to the cloud controller
https://github.com/prosociallearnEU/cf-nodejs-client
const endpoint = "https://api.mycompany.com/";
const username = "myuser";
const password = "mypass";
const CloudController = new (require("cf-client")).CloudController(endpoint);
const UsersUAA = new (require("cf-client")).UsersUAA;
const Apps = new (require("cf-client")).Apps(endpoint);
CloudController.getInfo().then((result) => {
UsersUAA.setEndPoint(result.authorization_endpoint);
return UsersUAA.login(username, password);
}).then((result) => {
Apps.setToken(result);
return Apps.getApps();
}).then((result) => {
console.log(result);
}).catch((reason) => {
console.error("Error: " + reason);
});
I try to run it against our API and its not working and Im not getting no error message in the console, what it can be ?
where does the space/org is handled here ? since when I connect from the cli it ask me to which space/org I want to connect...
Im able to login via the CLI, just from the code I cant, any idea what is missing here?
The issue it when I run it I dont get any error that can help to understand what is the root cause
I cloned the original git repository and modified some methods to support proxy. Please note that I modified just some methods to get the sample code working, but a complete refactor of the package is needed.
Basically what you have to do is to add a proxy parameter before calling the request method (this is done throughout the package, so several modifications are needed), for example this is for one of the methods in the Organization.js file:
getSummary (guid) {
const url = `${this.API_URL}/v2/organizations/${guid}/summary`;
const proxy = `${this.API_PROXY}`;
const options = {
method: "GET",
url: url,
proxy: proxy,
headers: {
Authorization: `${this.UAA_TOKEN.token_type} ${this.UAA_TOKEN.access_token}`
}
};
return this.REST.request(options, this.HttpStatus.OK, true);
}
You can find my changes in the git repository below:
https://github.com/adasilva70/cf-nodejs-client.git
I have also created a new sample below. This sample lists all organizations for a user, gets the first organization returned and lists its spaces. You can modify the code to provide a similar functionality that cf login provides (allow you to select an organization then a space).
const endpoint = "https://api.mycompany.com/";
const username = "youruser";
const password = "yourpassword";
const proxy = "http://proxy.mycompany.com:8080";
const CloudController = new (require("cf-nodejs-client")).CloudController(endpoint, proxy);
const UsersUAA = new (require("cf-nodejs-client")).UsersUAA;
const Apps = new (require("cf-nodejs-client")).Apps(endpoint, proxy);
const Orgs = new (require("cf-nodejs-client")).Organizations(endpoint, proxy);
CloudController.getInfo().then((result) => {
console.log(result);
UsersUAA.setEndPoint(result.authorization_endpoint, proxy);
return UsersUAA.login(username, password);
}).then((result) => {
//Apps.setToken(result);
//return Apps.getApps();
Orgs.setToken(result);
return Orgs.getOrganizations();
}).then((result) => {
console.log(result);
org_guid = result.resources[1].metadata.guid;
return Orgs.getSummary(org_guid);
}).then((result) => {
console.log(result);
}).catch((reason) => {
console.error("Error: " + reason);
});
I have done just minor tests to make sure the sample works, so use carefully. Also, the changes will only work for a case where proxy is needed now.
The first thing that strikes me on the library's github site is the warning:
Note: This package is not ready for a production App yet.
It also seems that the project is not being maintained as there are a number of tickets ooened that are quite a few months old that don't have a response.
Anyway, to figure out why the library is not working and producing no error message, I would check out the library source code and add some console logging statements, probably starting with the HttpUtils. For example:
requestWithDefaults(options, function (error, response, body) {
console.log("requestWithDefaults error: ", error)
console.log("requestWithDefaults response: ", response)
console.log("requestWithDefaults body: ", body)
...
}
Alternatively, you could try debugging the code by adding breakpoints to the requestWithDefaults and other key places in the library, using the nodejs debugger.
You could also try debugging the network calls similar to this how to monitor the network on node.js similar to chrome/firefox developer tools?
To understand how to use the library, I would take a look into the tests folder and look for a test that is similar to your use case. There are a reasonable amount if tests that look useful in the test/lib/model/cloudcontroller folder.
As for the question about spaces, I have found an example where you can pass in a space guid to return apps for that space guid.
CloudFoundrySpaces.getSpaceApps(space_guid, filter).then( ... )
I'm assuming the call you are using App.getApps() will return Apps for all spaces/organizations.
I have my test.login.js:
it('calls login when there\'s a username present', () => {
React.findDOMNode(LoginElement.refs.username).value = 'foo';
TestUtils.Simulate.submit(form);
expect(LoginElement.state.errored).toEqual(false);
});
By submitting the form, it calls a login method:
login() {
let typedUsername = React.findDOMNode(this.refs.username).value;
if (!typedUsername) {
return this.setState({
errored: true
});
}
// we don't actually send the request from here, but set the username on the AuthModel and call the `login` method below
AuthModel.set('username', typedUsername);
AuthModel.login();
},
So I'm trying to test the functionality of Login.jsx, not AuthModel.js, however by calling AuthModel.login(), it sends a message over a WebSocket. However, the issue is that in my actual app, I don't load anything until the WebSocket has connected (I fire an event to then render the React app), however in my Jasmine test, I don't wait for this event, so I receive:
ERROR: null, DOMException{stack: 'Error: Failed to execute 'send' on 'WebSocket': Still in CONNECTING state.
And my test fails, which, it shouldn't fail because it's encapsulated functionality does what I want it to. It just errors further up the dependency tree.
What is my best approach for either working around this, or to mitigate the WebSocket trying to connect in my test env? (I'm extremely new to testing, so these concepts are very alien to me right now)
I won't pretend to know a lot about this, but can't you dependency inject AuthModel so how and then mock it in your tests? Sorry this isn't a complete answer it's just what my first instinct would be.
If you need a library to assist this, angular/di (from angular2) is pretty great.
You could mock / stub the server request using Sinon JS. - http://sinonjs.org/
If you just want to know that Auth.login makes a request to the server, use sinon.stub (http://sinonjs.org/docs/#stubs), e.g.
var stub = sinon.stub($, 'ajax');
//Do something that calls $.ajax
//Check stub was called and arguments of first call:
console.log(stub.called)
console.log(stub.args[0])
stub.restore();
If your code requires a response, use sinon's fake server (http://sinonjs.org/docs/#server):
var server = sinon.fakeServer.create(),
myResults = [1, 2, 3];
//Set-up server response with correct selected tags
server.respondWith('POST', url, [200, {
'Content-Type': 'application/json'
},
JSON.stringify({
response: myResults
})
]);
//Do something which posts to the server...
sendToServer('abc').done(function(results) {
console.log('checking for results ', results);
})
server.restore();
You can get a lot more complicated with the server responses - using functions, etc. to handle multiple request types, e.g.
function initServer(respondOk) {
var server = sinon.fakeServer.create();
server.respondWith('POST', /.*\/endpoint\/.*/, function(request) {
var header = { 'Content-Type': 'application/json' };
if(!respondOk) {
var response = JSON.stringify([{
'error_code': '500',
'message': 'Internal server error'
}]);
request.respond(500, header, response);
} else {
var code = 200,
resources = JSON.parse(request.requestBody),
result = JSON.stringify({ customer: resources });
request.respond(code, header, result);
}
});
return server;
});
I'm attempting to utilize Google's authorization services this guide.
I'm having trouble trading the code in for a token from the server.
var token_request='?code='+code+
'&client_id='+client_id+
'&client_secret='+client_secret+
'&redirect_uri='+redirect_uri+
'&grant_type=authorization_code';
options = {
host: "accounts.google.com",
path: '/o/oauth2/token'+token_request,
method: "POST"
}
var tokenRequest = https.request(options, function(res){
var resp = "";
res.on('data', function(data){
resp+= data;
})
res.on('end', function(){
console.log(resp);
})
res.on('error', function(err){
console.log("\033[;33mIt's an Error.\033[0;39m");
console.log(err);
})
}).end();
I would say from this site that you should use 'method: "GET"' instead of 'method: "POST"' since your values are in the query string.
EDIT:
According to the comments, I would say that you have to rework your code in order for it to work properly.
To be honest I am trying to do the same thing with difficulty. Not withstanding that is it worth trying googleapis.
You need to use npm to install the google apis
npm install googleapis
see https://npmjs.org/package/googleapis
for the documentation
I got awesome client side tests that I run with Yeoman. Yeoman compiles my CoffeeScript, opens up the test page in a server, visit it with PhantomJS and pass all the tests results to the command line. The process is pretty hacky, the test results are passed via alert() messages to the Phantom process which creates a temporary file and fills it with the messages as JSON. Yeoman (well, Grunt) loops over the temporary file, parses the tests and displays them in the command line.
The reason I explained the process is that I want to add a few things to it. I got server side tests as well. They use mocha and supertest to check the API endpoints and a Redis client to make sure the database state is as expected. But I want to merge those two test suites!
I don't want to write client side mock response for the server calls. I don't want to send the server mock data. Somewhere along the way I'll change the server or the client and the test will not fail. I want to do a real integration testing. So, whenever a test finishes in the client side I want a hook to run a relevant test on the server side (checking db state, session state, moving to a different test page).
Are there any solutions to this? Or, altenatively, where do I start hacking on Yeoman / Grunt / grunt-mocha to make this work?
I think the Phantom Handlers in grunt-mocha is a good place to start:
// Handle methods passed from PhantomJS, including Mocha hooks.
var phantomHandlers = {
// Mocha hooks.
suiteStart: function(name) {
unfinished[name] = true;
currentModule = name;
},
suiteDone: function(name, failed, passed, total) {
delete unfinished[name];
},
testStart: function(name) {
currentTest = (currentModule ? currentModule + ' - ' : '') + name;
verbose.write(currentTest + '...');
},
testFail: function(name, result) {
result.testName = currentTest;
failedAssertions.push(result);
},
testDone: function(title, state) {
// Log errors if necessary, otherwise success.
if (state == 'failed') {
// list assertions
if (option('verbose')) {
log.error();
logFailedAssertions();
} else {
log.write('F'.red);
}
} else {
verbose.ok().or.write('.');
}
},
done: function(failed, passed, total, duration) {
var nDuration = parseFloat(duration) || 0;
status.failed += failed;
status.passed += passed;
status.total += total;
status.duration += Math.round(nDuration*100)/100;
// Print assertion errors here, if verbose mode is disabled.
if (!option('verbose')) {
if (failed > 0) {
log.writeln();
logFailedAssertions();
} else {
log.ok();
}
}
},
// Error handlers.
done_fail: function(url) {
verbose.write('Running PhantomJS...').or.write('...');
log.error();
grunt.warn('PhantomJS unable to load "' + url + '" URI.', 90);
},
done_timeout: function() {
log.writeln();
grunt.warn('PhantomJS timed out, possibly due to a missing Mocha run() call.', 90);
},
// console.log pass-through.
// console: console.log.bind(console),
// Debugging messages.
debug: log.debug.bind(log, 'phantomjs')
};
Thanks! There will be a bounty on this.
I don't know about Yeoman - I haven't tried it yet - but I got the rest of the puzzle running. I believe you will figure out the rest.
Why Doing Integration Tests?
In your question you were talking about the situation when you have both client-side tests and server-side tests running with mocks. I assume that for some reason you can't get both test sets running with the same mocks. Otherwise, if you changed the mocks on client-side your server-side tests would fail because they would get the broken mock data.
What you need are the integration tests so when you run some client-side code in your headless browser your server-side code would also run. Moreover, simply running your server-side and client-side code is not enough, you also want to be able to put assertions on both sides, don't you?
Integration Tests with Node and PhantomJS
Most of the examples of integration tests that I found online either use Selenium or Zombie.js. The former is a big Java-based framework to drive real browsers while the later is a simple wrapper around jsdom. I assume you're hesitant to use either of those and would prefer PhantomJS. The tricky part, of course, is to get that running from your Node app. And I got just that.
There are two node modules to drive PhantomJS:
phantom
node-phantom
Unfortunately, both projects seem abandoned by their authors and other community members fork them and adapt to their needs. That means that both projects got forked numerous times and all forks are barely running. The API is almost non-existent. I got my tests running with one of the phantom forks (Thank you, Seb Vincent). Here's a simple app:
'use strict';
var express = require('express');
var app = express();
app.APP = {}; // we'll use it to check the state of the server in our tests
app.configure(function () {
app.use(express.static(__dirname + '/public'));
});
app.get('/user/:name', function (req, res) {
var data = app.APP.data = {
name: req.params.name,
secret: req.query.secret
};
res.send(data);
});
module.exports = app;
app.listen(3000);
})();
It listens for request to /user and returns path parameter name and query parameter secret. Here's the page where I call the server:
window.APP = {};
(function () {
'use strict';
var name = 'Alex', secret ='Secret';
var xhr = new XMLHttpRequest();
xhr.open('get', '/user/' + name + '?secret=' + secret);
xhr.onload = function (e) {
APP.result = JSON.parse(xhr.responseText);
};
xhr.send();
})();
And here's a simple test:
describe('Simple user lookup', function () {
'use strict';
var browser, server;
before(function (done) {
// get our browser and server up and running
phantom.create(function (ph) {
ph.createPage(function (tab) {
browser = tab;
server = require('../app');
server.listen(3000, function () {
done();
});
});
});
});
it('should return data back', function (done) {
browser.open('http://localhost:3000/app.html', function (status) {
setTimeout(function () {
browser.evaluate(function inBrowser() {
// this will be executed on a client-side
return window.APP.result;
}, function fromBrowser(result) {
// server-side asserts
expect(server.APP.data.name).to.equal('Alex');
expect(server.APP.data.secret).to.equal('Secret');
// client-side asserts
expect(result.name).to.equal('Alex');
expect(result.secret).to.equal('Secret');
done();
});
}, 1000); // give time for xhr to run
});
});
});
As you can see I have to poll the server inside the timeout. That's because all the phantom bindings are incomplete and too limiting. As you can see I'm able to check both client state and server state in a single test.
Run your tests with Mocha: mocha -t 2s You'll probably need to increase the default timeout setting for more evolved tests to run.
So, as you can see the whole thing is doable. Here's the repo with complete example.