I'm using this module https://www.npmjs.com/package/limiter to limit http requests to a site using something like this:
let running = true;
const limiter = new RateLimiter(
1,
'second'
);
function removeTokens(limiter, tokens){
const self = this;
return new Promise((resolve) => {
if (!running) throw new Error('Terminated prematurely');
limiter.removeTokens(tokens, resolve)
});
}
Array.apply(null, {length: 1000}).map(Number.call, Number).map(number=>{
return removeTokens(limiter,1).then(()=>{
request('http://test.com/'+number);
});
});
setTimeout(()=>{running=false},1000);
The problem I have is that I'm not able to clear underlying timeouts that the module uses so the requests keep getting called. I have tried to fork the repository and create my own solutions but they have all failed. I can supply my attempted solutions if needed. Is there any simple way that I can clear the timeouts (without building my own queueing system) that I'm missing?
Related
I have a Selenium webdriverIO V5 framework. The issue I am facing here is, the below code works fine on Mac OS, but it does not work correctly on the Windows OS. In the Windows OS it gets stuck with an infinite loop issue.
The below code functionality is: Merge yaml files (which contains locators) and return the value of the locator by passing the key:
const glob = require('glob');
const yamlMerge = require('yaml-merge');
const sleep = require('system-sleep');
let xpath;
class Page {
getElements(elementId) {
function objectCollector() {
glob('tests/wdio/locators/*.yml', function (er, files) {
if (er) throw er;
xpath = yamlMerge.mergeFiles(files);
});
do {
sleep(10);
} while (xpath === undefined);
return xpath;
}
objectCollector();
return xpath[elementId];
}
}
module.exports = new Page();
Since you are waiting on the results of a callback, I would recommend returning a new Promise from your getElements function and resolve() the value you receive inside the callback. Then when you call getElements, you will need to resolve that Promise or use the await notation. The function will stop at that point and wait until the Promise resolves, but the event loop will still continue. See some documentation for more information.
I'll write an example below of what your code might look like using a Promise, but when you call getElements, you will need to put the keyword await before it. If you want to avoid that, you could resolve the Promise from objectCollector while you're in getElements and remove the async keyword from its definition, but you really should not get in the way of asynchronous JavaScript. Also, you can probably shorten the code a bit because objectCollector looks like an unnecessary function in this example:
const glob = require('glob')
const yamlMerge = require('yaml-merge')
const sleep = require('system-sleep')
let xpath
class Page {
function async getElements(elementId) {
function objectCollector() {
return new Promise((resolve,reject) => {
glob('tests/wdio/locators/*.yml', function (er, files) {
if (er) reject(er)
resolve(yamlMerge.mergeFiles(files))
})
})
}
let xpath = await objectCollector()
return xpath[elementId]
}
}
module.exports = new Page();
I want to make a request and cache it, in a functional style.
const req = (uri) =>
(console.log(`requesting: ${uri}`), Promise.resolve({ status: 200 }));
const cache = (fn) => (...args) =>
fn(...args).then((result) => { console.log('caching:', result) });
const cachedReq = cache(req);
cachedReq('example.com/foo');
Two questions:
Is this code idiomatic?
How can I supply logic to generate the cache key from the result, while maintaining separation of concerns? For example, I might use req to retrieve different kinds of resource which need different logic to generate the key to be used in the cache. How should I supply this key-generation logic to the cache function?
Edit:
In reality, the URI should be the key (thanks to #epascarello). I chose a poor example. But I'd like to ask about the more general case, where logic needs to be supplied "down composition", while maintaining decent separation of concerns.
You almost close to achieve your goal, you are in the right direction, with composition concept. maybe this code can help you to make your goal come true.
Let's simulate your req function like so:
var req = (uri) => {
console.log("inside req", uri);
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve({ status: 200 });
}, 3000);
});
}
then you have the cacheFunc version as:
var withCache = (promiseFunc) => {
const cache = {};
return (...args) => {
// suppose first param is uri
var uri = args[0];
return new Promise((resolve, reject) => {
if (cache.hasOwnProperty(uri)) {
return resolve(cache[uri]);
}
promiseFunc(...args).then((data) => {
cache[uri] = data;
resolve(data);
}).catch(reject);
});
}
}
as you can see, you need to create and cache object into the first function, so this is a a little similar to Currying in JS, so you need to wrap your req (that is a promise) wrapped into another promise from the cache version, so before execute the req function, you need to verify if some response exists into cache with the same uri key, if it is, so resolve inmmediatly the promise, else execute the req function, once you receive the response cache the response and resolve the cache promise version.
So you can use it like so:
var cacheReq = withCache(req);
cacheReq('https://anywhere.com').then(console.log.bind(null, 'response')).catch(console.log.bind(null, 'error response'));
you will notice that in the first time you promise wait until 3 seconds to resolve the req, in the second call the promise will resolve the promise ASAP because of cache, if you try with another uri it will wait 3 seconds again and will cache the response to use it the next time.
Hope it can help you.
You can use a combination of a Map and the Request constructor:
// I'll be using ramda for object equality, but any
// deepEquals checker should work.
const R = window.R;
const genRequest = ((cache, eqComparator) => {
return (url, fetchOpts={}) => {
const key = {url, fetchOpts};
const alreadyHave = [...cache.keys].find(x => eqComparator(x, key));
if (alreadyHave) return cache.get(alreadyHave);
const req = new Request(url, fetchOpts);
cache.set(key, req);
return req;
};
})(new Map(), R.equals);
const req = genRequest('http://www.google.com');
fetch(req)
.then(...)
.catch(...);
Some nice properties fall out of this:
Each request is constructed only once but can be repeatedly fetched.
No side-effects until you fetch: creating the request and fetching it are separate.
...thus, concerns are about as separated as they can be.
You could re-jigger parameter application to easily support custom equality comparisons using the same cache.
You can use the same strategy to cache the results of a fetch, separately from caching the requests.
I've learned node.js and javascript lately. I loved node.js a lot, but I am working on a project coded in node.js, mongodb, cordova etc. I notice that I needed to use Promise Object in the code a lot.
I create a module in the project to query the db and bring results. In every exported function I need to declare a local function, then use promise, for example:
I have the following local functions in the Module:
var Initialize = function() {
return new Promise(function(resolve, reject) {
try {
MongoClient.connect("db_url_conn", function(err, database) {
if (err) return console.log(err)
db = database;
return resolve(db);
})
} catch (e) {
console.error(e);
}
});
};
then in every exported function in the module I needed to use:
mongoOperation.prototype.getLength = function() {
Initialize(function(db) {
return db;
}).then(function(db) {
getSize(db).then(function(length) {
console.log(length);
});
});
}
The Question is:
Is that normal according to the nature of node.js and JavaScript nature to use promise a lot?
Do I have any other choices to fulfill that?
Since MongoClient.connect() already returns a promise, you can simplify your code:
var Initialize = function() {
return MongoClient.connect("db_url_conn");
};
...
Initialize().then(function(db) { ... });
However, this will create a new client each time you call Initialize, where you should be reusing the client for better performance and to leverage the built-in connection pool:
var client = MongoClient.connect("db_url_conn");
var Initialize = function() { return client };
I may have miss something about Angular's promises but I was wondering something : are promises asynchronous ? I'm not sure if 'asynchronous' is the right word but let me explain myself.
In my code I use promises to do a really big process (read and write hundreds of big files) while I display a loading bar to watch the progress of the process. I've noticed that even if my code is in a promise, it seems to not really be asynchronous and freeze the display (that I assume is manage by the main thread).
For example in the code bellow that you can find in this Plnkr, I'm wondering how to let the progress bar move while the big process is done. I understand why it's freezing when I call it in the main thread but not when I'm using Angular's promises.
var app = angular.module('plunker', []);
app.controller('MainCtrl', function($scope, $q) {
function hugeProcess () {
var i = 0;
var start = new Date().getTime();
while (i++ < 100000) {
console.log(i);
}
var end = new Date().getTime();
var time = end - start;
$scope.processTime = 'Done in ' + time + 'ms';
}
$scope.onClickStartHugeProcess = function () {
console.log('onClickStartHugeProcess');
hugeProcess();
};
$scope.onClickStartHugeProcessWithPromise = function () {
console.log('onClickStartHugeProcessWithPromise');
$q.when()
.then(function () {
return hugeProcess();
});
};
});
The issue in your code is that your hugeProcess function never yields. So yes, it's called asynchronously (then callbacks are always called asynchronously in a Promises/A+ promise implementation), but that doesn't change what hugeProcess is doing when it gets called, which is hogging the main UI thread such that nothing else can happen while it's running. There's only one main UI thread, and all of your JavaScript runs on that one main UI thread except web workers.
To make hugeProcess not do that, you have to break it up and have it call itself after a brief delay, via setTimeout (or perhaps something built into Angular).
As Joe Clay points out, this code doesn't make a lot of sense:
$q.when()
.then(function () {
return hugeProcess();
});
That's effectively:
setTimeout(hugeProcess, 0);
...since $q.when() with no arguments returns a resolved promise, and adding a then callback to a resolved promise just results in your callback being called as soon as possible (but asynchronously; e.g., then returns before the callback is called).
So, I've discover Web Workers and here is a first version of my code using them.
app.controller('MainCtrl', function($scope, $q) {
function hugeProcess () {
var i = 0;
var start = new Date().getTime();
while (i++ < 100000) {
console.log(i);
}
var end = new Date().getTime();
var time = end - start;
postMessage(time);
}
var blob = new Blob(["onmessage = " + hugeProcess.toString()]);
// Obtain a blob URL reference to our worker 'file'.
var blobURL = window.URL.createObjectURL(blob);
var worker = new Worker(blobURL);
worker.onmessage = function (message) {
$scope.processTime = 'Done in ' + message.data + 'ms';
$scope.$apply()
};
$scope.onClickStartHugeProcessWithPromise = function () {
console.debug('onClickStartHugeProcessWithPromise');
$q(function () {
worker.postMessage(''); // Start the worker.
});
};
});
I don't think I'm using right but it does what I want ... I've found the package ng-webworker for Angular that seems to mix promises and web workers so that's exactly what I'm looking for.
Thank you all for your help.
Web Worker is right solution. I had similar problem and developed angular plugin ng-vkThread to simplify such kind of tasks.
Basic usage is:
/* function to execute in a thread */
function foo(n, m){
return n + m;
}
/* create an object, which you pass to vkThread as an argument*/
var param = {
fn: foo // <-- function to execute
args: [1, 2] // <-- arguments for this function
};
/* run thread */
vkThread.exec(param).then(
function (data) {
console.log(data); // <-- thread returns 3
},
function(err) {
alert(err); // <-- thread returns error message
}
);
Live demo
--Vadim
How can I change things around in my db connection call so that I can do db.collection():
// Create a Mongo connection
Job.prototype.getDb = function() {
if (!this.db)
this.db = Mongo.connectAsync(this.options.connection);
return this.db;
};
// I want to be able to do this
Job.prototype.test = function() {
return this.db.collection('abc').findAsync()...
};
// Instead of this
Job.prototype.test = function() {
return this.getDb().then(function(db) {
return db.collection('abc').findAsync()...
});
};
My code always calls getDb so the connection does get created so that is not an issue. ex:
this.getDb().then(test.bind(this));
But I actually string many of these calls around so looking for a cleaner approach.
This works - just wondering if there an overall better approach to dealing with this.
Job.prototype.getDb = function(id) {
var self = this;
return new P(function(resolve, reject) {
if (!self.db) {
return Mongo.connectAsync(self.options.connection)
.then(function(c) {
self.db = c;
debug('Got new connection');
resolve(c);
});
}
debug('Got existing connection');
resolve(self.db);
});
};
I suppose this is really just a mongo connection issue, perhaps not just promises. All the Mongo examples I see either just make all their calls inside of the connect callback or use some framework like Express and assign it on start.
I want to be able to do this
return this.db.collection('abc').findAsync()
No, that's impossible when you don't know whether the database is already connected or not. If you might need to connect at first, and that is asynchronous, then this.db must yield a promise, and you'll need to use then.
Notice that with Bluebird you can shorten that code a bit, and avoid that verbose .then() callback by using the .call() method:
Job.prototype.getDb = function() {
if (!this.db)
this.db = Mongo.connectAsync(this.options.connection);
return this.db;
};
Job.prototype.test = function() {
return this.getDb().call('collection', 'abc').call('findAsync');
};