Helo Folks,
I am working on a view in couchdb. And, in the 'extract' list function, I'm trying to filter out some information using that view (myView). From the client that connects to couchdb, I want to do 1 major thing - show the results from the 'extract' list function. But, there are multiple other things that I want to perform on the results returned from the 'extract' function. One simple operation out of all the other operations is 'sum'. But, there are many other features like calculating median/standard deviation etc on the results of the 'extract' list function.
{
"_id": "_design/myDesigndoc",
"lists": {
"extract": "function(head, req){ ...*extract some info the view*: **myView** ...}",
"sum" : "function(head,req) {...**sum up all the values returned from the 'extract' function above**...}"
},
"views": {
"myView" : { "map" : "..." },
}
}
So, I'm stuck at one point:-
As the whole design doc is a Json and the function bodies are javascript, is there a way to call the 'extract' list function in other list functions like 'sum', 'median', 'standard deviation' etc ?
Reason I want to do this:-
All the other list functions: 'sum', 'standard deviation' etc expect the return value of 'extract' function as input. So, just making redundant copies of the code of extract function in other list functions is the last thing I would want to do.
Is there an alternate way to solve this:-
Yes, there is a way. I had thought that I'll use another view functions than 'myView' for all these functionalities and write the same 'map' function as that in 'myView' but, all these views will have separate 'reduce' functions for calculating 'sum', 'standard dev' etc.
But, the calculation of those views caused a lot of resource usage because those many views were getting created each time.
Could you guys provide a better solution than this?
Thanks
My first thought was to implement the views again with reduce functions to do the calculations but you say this is too resource intensive.. I wonder how often are the views used and if there is a heap of changes between accesses?
If they are just used to produce some statistics for reports or something and are rarely accessed that when they do there is a heap of changes it needs to make to the view indexes maybe you could look at running a script that regularly retrieves the views so it keeps the views up to date so when they are accessed they still respond relatively quickly.
This is something we have done with our all of our views in our production environment and it works quite well, I guess it depends on your infrastructure and how much data you are pumping through.
Something else to consider I am not sure if there is any difference/benefit to doing so but maybe the built in reduce functions may offer better performance than your self created ones
http://wiki.apache.org/couchdb/Built-In_Reduce_Functions
Related
Context
I'm building a general purpose game playing A.I. framework/library that uses the Monte Carlo Tree Search algorithm. The idea is quite simple, the framework provides the skeleton of the algorithm, the four main steps: Selection, Expansion, Simulation and Backpropagation. All the user needs to do is plug in four simple(ish) game related functions of his making:
a function that takes in a game state and returns all possible legal moves to be played
a function that takes in a game state and an action and returns a new game state after applying the action
a function that takes in a game state and determines if the game is over and returns a boolean and
a function that takes in a state and a player ID and returns a value based on wether the player has won, lost or the game is a draw. With that, the algorithm has all it needs to run and select a move to make.
What I'd like to do
I would love to make use of parallel programming to increase the strength of the algorithm and reduce the time it needs to run each game turn. The problem I'm running into is that, when using Child Processes in NodeJS, you can't pass functions to the child process and my framework is entirely built on using functions passed by the user.
Possible solution
I have looked at this answer but I am not sure this would be the correct implementation for my needs. I don't need to be continually passing functions through messages to the child process, I just need to initialize it with functions that are passed in by my framework's user, when it initializes the framework.
I thought about one way to do it, but it seems so inelegant, on top of probably not being the most secure, that I find myself searching for other solutions. I could, when the user initializes the framework and passes his four functions to it, get a script to write those functions to a new js file (let's call it my-funcs.js) that would look something like:
const func1 = {... function implementation...}
const func2 = {... function implementation...}
const func3 = {... function implementation...}
const func4 = {... function implementation...}
module.exports = {func1, func2, func3, func4}
Then, in the child process worker file, I guess I would have to find a way to lazy load require my-funcs.js. Or maybe I wouldn't, I guess it depends how and when Node.js loads the worker file into memory. This all seems very convoluted.
Can you describe other ways to get the result I want?
child_process is less about running a user's function and more about starting a new thread to exec a file or process.
Node is inherently a single-threaded system, so for I/O-bound things, the Node Event Loop is really good at switching between requests, getting each one a little farther. See https://nodejs.org/en/docs/guides/event-loop-timers-and-nexttick/
What it looks like you're doing is trying to get JavaScript to run multiple threads simultaniously. Short answer: can't ... or rather it's really hard. See is it possible to achieve multithreading in nodejs?
So how would we do it anyway? You're on the right track: child_process.fork(). But it needs a hard-coded function to run. So how do we get user-generated code into place?
I envision a datastore where you can take userFn.ToString() and save it to a queue. Then fork the process, and let it pick up the next unhandled thing in the queue, marking that it did so. Then write to another queue the results, and this "GUI" thread then polls against that queue, returning the calculated results back to the user. At this point, you've got multi-threading ... and race conditions.
Another idea: create a REST service that accepts the userFn.ToString() content and execs it. Then in this module, you call out to the other "thread" (service), await the results, and return them.
Security: Yeah, we just flung this out the window. Whether you're executing the user's function directly, calling child_process#fork to do it, or shimming it through a service, you're trusting untrusted code. Sadly, there's really no way around this.
Assuming that security isn't an issue you could do something like this.
// Client side
<input class="func1"> // For example user inputs '(gamestate)=>{return 1}'
<input class="func2">
<input class="func3">
<input class="func4">
<script>
socket.on('syntax_error',function(err){alert(err)});
submit_funcs_strs(){
// Get function strings from user input and then put into array
socket.emit('functions',[document.getElementById('func1').value,document.getElementById('func2').value,...
}
</script>
// Server side
// Socket listener is async
socket.on('functions',(funcs_strs)=>{
let funcs = []
for (let i = 0; i < funcs_str.length;i++){
try {
funcs.push(eval(funcs_strs));
} catch (e) {
if (e instanceof SyntaxError) {
socket.emit('syntax_error',e.message);
return;
}
}
}
// Run algorithm here
}
I am currently working on large projects that make use of lots of javascript files.
I then start learning of using local scopes and using objects.
What I do not really understand is how to call them into you local scope?
E.g if I create an object in an local scope in file-a, how can I use them as in a function in the document.ready scope file-b?
I get that you can find this online, but I get demotivation by the high amount of javascript on the internet and can't really find good examples or material. Any help?
Not sure,but I think you might be referring to the use of namespaces within JavaScript as a way to avoid adding all your functions to the main window object.
The Ugly Way
Let's assume you have 3 functions related to cats:
Function AddCat(cat) {
}
Function DeleteCat(catId) {
}
Function BreedCat(cat,cat) {
}
The way these items are coded, they are globally available. Not only does that clutter up your window object, it's hard to share data between these functions in a discreet way.
As long as this js file is loaded, any function in your app can call these functions just by calling AddCat()
Cleaner
To solve that problem, we could create a Cats object that acts as a "namespace" here:
Cats = {
AddCat: function(cat) {
},
DeleteCat: function(catId) {
},
BreedCat: function(cat,cat) {
}
}
Now, you've only added ONE object to the windows class: Cats. In addition, other methods in your web app can call any of those 3 items by calling Cats.AddCat() for example.
This lets you encapsulate all of the Cat data in your entire system within a single "namespace, so it's easier to read.
This can get a lot more detailed. By encapsulating items like this, you can start to hide variables that all your cat routines require from the rest of your code.
There is an excellent set of resources on this type of namespacing (including tons of detail) here and here with links that lead you deeper.
Is that what you were looking for?
I am using protractor to test a series of webpages (actually mostly one webpage of Angular design, but others as well). I have created a series of page objects to handle this. To minimize code maintenance I have created an object with key value pairs like so:
var object = {
pageLogo: element(by.id('logo')),
searchBar: element.all(by.className('searchThing')),
...
};
The assumption being that I will only need to add something to the object to make it usable everywhere in the page object file. Of course, the file has functions (assuming you are not familiar with the page object pattern) as such:
var pageNamePageObject = function () {
var object = {...}; //list of elements
this.get = function() {
brower.get('#/someWebTag');
}
this.getElementText = function(someName){
if (typeof someName == 'number')
... (convert or handle exception, whatever)
return object[name].getText();
}
...
*Note these are just examples and these promises can be handled in a variety of ways in the page object or main tests
The issue comes from attempting to "cycle" through the object. Given that the particular test is attempting to verify, among other things, that all the elements are on the particular web page I am attempting to loop through these objects using the "isPresent()" function. I have made many attempts, and for brevities sake I will not list them here, however they include creating a wrapper promise (using "Q", which I must admit I have no idea how it works) and attempting to run the function in the 'expect' hoping that the jasmine core will wait for all the looping promises resolve and then read the output (it was more of a last ditch effort really).
You should loop like you did before on all of the elements, if you want it in a particular order, create a recursive function that just calls itself with the next element in the JSON.
Now, to handle jasmine specs finishing before and that stuff.
this function needs to be added to protractor's flow control for it to wait to continue, read more about it here. and also, dont use Q in protractor, use protractor's implementation of webdriverJS promises.
Also, consider using isDisplayed instead, assuming you want it to be dispalayed on the page.
So basically, your code skeleton will look like this:
it(.....{
var flow = webdriver.promise.controlFlow();
return webdriver.execute(function () {//your checks on the page here,
//if you need extract to external function as i described in my first paragraph
well i think that should provide you enough info on how to handle waiting for promises in protractor, hope i helped.
I am new to databases and these MongoDB/Mongoose.js async functions have annoyed the hell out of me over the last few hours. I have written and rewritten this bit so many times:
router.get('/districts', function(req, res) {
districtNames = [];
// I'm using mongoose-simpledb, so db.District refers to the districts collection
db.District.find(function(err, found) {
found.forEach(function(element) {
findParentProv(element, districtNames);
});
res.render('districts', {title: "Districts page", district_list: districtNames});
})
});
function findParentProv(element, namesArray) {
db.Province.findById(element.parent, function(err, found) {
console.log(found.name);
namesArray.push(element.name + " " + found.name);
});
}
I want to get all items in the districts collection, follow their parent field (which contains an ObjectID), find that item from the provinces collection and push both their names as a string into districtNames.
How should I do this?
Well, you do seem to be on the right track.
The one major issue I recognize in your solution is that after kicking off all the async queries for parents, you immediately return the (most likely empty) districtNames array, without waiting for the queries to finish.
This is indeed very annoying, and not surprisingly so. MongoDB is a non-relational DB, and so join operations like what you're trying to do aren't easy to get right.
The solution that would probably require the least fundamental changes to what you're doing would be to wait on all the queries before calling res.render. The most basic way to do this would be to check the length of namesArray/districtNames after pushing each element, and once you see it's gotten to the desired size, only then calling render. There are, however, more standardized ways of doing this, and I'd suggest looking into something like Async (specifically async.parallel) or a Promise framework such as Bluebird.
Now, another approach to solving this problem is de-normalizing the data. For someone with a relational background this probably sound appalling, but in Mongo it might actually be a valid solution to just include the province names along with their IDs in the districts collection, in which case your one initial query should be sufficient.
Another approach, which might be suitable if you're dealing with relatively small collections, would be to run 2 queries, 1 for all the districts and 1 for all the provinces, and do the correlation in-app. Obviously, this isn't a very efficient solution, and should definitely be avoided if there's any chance the collections contain, or will in the future contain, more than a handful of objects.
Best bet moving forward is to use ES6 Promise patterns to help with your callback patterns..
suggested modules:
lodash [optional] has a lot of useful methods, not needed here, but you may need, for example _.flatten, or _.assign
i-promise will give you a native Promise (node 0.11.3+) or a scripted implementation
es6-promise is the fallback for i-promise to use
promisify-patch is an inline promisify for specific methods.
Install the modules required for your use (in this example).
npm install --save es6-promise i-promise promisify-patch
Use Promise pattern with your example.
require('promisify-patch').patch();
var Promise = require('i-promise')
;
//returns a promise to resolve to your list for display
function getDistricts() {
//gets all of the db.District
return db.District.find.bind(db.District).promise()
//after districts retrieved
.then(function(districts){
//resolve an array of promises, will return an array of results
return Promise.all(districts.map(getDistrictProv)); //map each district via getDistrictProv
});
}
//returns a promise to resolve a specific district/province name
function getDistrictProv(district){
return db.Provice.findById.bind(db.Province).promise(element.parent)
.then(function(province){
return district.name + ' ' + province.name;
});
}
...
//express handler
router.get('/districts', function(req, res, next) {
//get the district names
getDistricts()
//then process the rendering with the names
.then(function(names){
res.render('districts', {title: "Districts page", district_list: names});
})
//if there was an error in the promise chain
// pass it along, so it can be handled by another express plugin
.catch(next)
});
Disclosure: I made i-promise and promisify-patch to make situations like this easier to convert node-style callbacks into promise chains.
NOTE: If you are creating general purpose libraries for Node or the Browser that are not flow-control related, you should at least implement the node-style callback implementation.
Further, you may wish to look into co, koa for using generators as well.
The question seemed to be how to control the flow of data, in which promises are likely the best answer. If your issue is trying to fit non-relational data into a relational box or vice-versa, may want to re-evaluate your data structure as well...
http://blog.mongodb.org/post/88473035333/6-rules-of-thumb-for-mongodb-schema-design-part-3
You should probably have some key data for parents/children replicated to those affected documents in other collections. There are configuration options via Mongoose to support this, but that doesn't mean you should avoid the consideration.
If you do many/large join operations like this it will negatively affect your performance. This isn't meant to be a religious comment only that MongoDB/SQL or other SQL vs. NoSQL considerations should be made depending on your actual needs.
The data in question seems to be highly cacheable data that may well be better with a relational/sql database.
This is really tricky to get my head around as I'm not used to this style of programming/data management.
All I'm trying to do at the moment is pass a json object returned via breeze into a dynatree or fancytree.
The examples that exist online all assume that the tree will do the ajax call via "initajax" or that some weirdly convoluted custom binding handler is needed into which various objects are passed:
ko.bindingHandlers.dynatree = {
update: function (element, valueAccessor, allBindingsAccessor, viewModel) {
setTimeout(function () {
$(element).dynatree({
noLink: true, // noLink is required to 'unlock' the checkboxes
minExpandLevel: 2
})
// the timeout value shows the correct knockout bindings BEFORE dynatree kicks in.
}, 1000);
}
}
This all seems too complicated to me, surely? I already have the json object, I know that's working. If I use knockout to "foreach" bind it to some plain html then all data is displayed just fine. In my mind all I need to do is initialize the tree div and pass it the json object... It's just that I have no idea how to do that!
I've tried using the jsfiddle here: http://jsfiddle.net/Ebram/UhA3m/5/ but chrome developer tools complain about the element having no "dynatree" method when the custom binding handler fires. It's passing in a "ul" element and that could be the problem - surely it should be passing in the tree div, not the ul element?
Anyhow, if anyone could point me in the right direction I'd hugely appreciate it. As I'm using John Papa's SPA methodology, I'm also unsure as to where I would put any separate js initialization code as the underlying viewmodel isn't the right place for me to be doing a $(#tree).dynatree initialization type call, is it? I must admit I've not got my head around this yet.
I suppose all I'm looking for is something along the lines of "once the viewmodel for this view has finished loading and the knockout binding is being done, initialize the dynatree div and pass this json object to the tree" if that makes sense in pseudocode?
I can hopefully point you in the approximate right direction.
It seems dynatree can also take JSON from a file as well as an AJAX request. In this example Lazy Loading, if you look in the source code, there's:
// In real life we would call a URL on the server like this:
...
// .. but here we use a local file instead:
Storing your data in a file to get it in seems awfully wasteful. Now that we know it's a little flexible in what it gets, let's see where it uses the data and maybe we can get it to use a local variable instead. let see where it loads it
Looking in the dynatree source, there's a function associated with appendAjax. (line 1774 in my source.) A little short on time at the moment, but I'd find where it gets the JSON and what it does with it. Perhaps you can do the same thing outside or mod the handling of ajaxOptions to take a variable with the JSON.