I recently discovered CouchDB and it fits perfectly for what I am working at today. Working with the Futon interface, and calling the http API works fin, but something is missing.
During the design of my application, I sometimes want to apply some changes on all the documents in the database. As a simplified example, lets say all my documents have a field named "type", and I decided to pick strings as type instead of numbers.
Now I have to go over all my documents in Futon, and change the string into a number, which is a silly job.
Another example would be deleting all documents that apply to a certain condition.
The perfect solution would be some kind of engine which can call a javascript function per document, and I can return the new value for the document.
Does this exist?
Everything you need is already there: the API. Note that Futon also only a wrapper around the API.
Not sure what middleware you use (node?, php?) but if you're familiar with the API it should be easy to:
Get all documents
Change field "type" (string instead of number)
Save the docs (use _bulk_docs)
Related
We have a MongoDB database in a development environment. There are a lot of collections that contain names of people. What we want to do is the following:
mask the names in each collection, the fields need to be updated directly in the database, cannot run them through some external pipeline
once masked, it is ok if we are unable to retrieve the original names (so one-way masking)
every unique name should result in the same mask
the masking script can be run on the mongodb cli or a MongoDB gui like Studio3T
I was thinking of maybe using MD5 or SHA, but I am not sure if either is available to use directly in mongo operations like update or even in javascript without external libraries.
Also, since MD5 always produces the same hash, if someone were to get access to the document, since we will not be masking the field name, it would be fairly easy to feed typical names into the algorithm until the hash matches to figure out the name, but I think we may be able to live with this.
An alternative I was thinking of was, to loop through the unique names we have, and create a map from names to UUIDs. Then, go through each collection and use this map to update the names with the UUIDs. The problem with this is that we'll need to keep this mapping dictionary for when we receive additional documents for an existing person.
Our organization has a CRM installation on which we've done extensive customization. Right now I'm trying to implement a solution to enforce a business rule: Prevent users from updating a program to inactive when the program is a designation opportunity on an open opportunity.
I know how to prevent the update; return false from OnSave() in the JavaScript. I haven't been able to find out when that's the case. The best idea I've come up with is to make a SOAP call to the OData endpoint in CRM, but I've come across a sticking point at the last step. (If you've got a better idea I'm totally open to it.)
Here's what I've got. I can get the program in question:
programset(guid'thisone')
.../OrganizationData.svc/uwkc_programSet(guid'F4D75E9D-3A79-E611-80DA-
C4346BACAAC0')
I can get the associated designations:
programset(guid'thisone')/program-desig
.../OrganizationData.svc/uwkc_programSet(guid'F0D75E9D-3A79-E611-80DA-C4346BACAAC0')/uwkc_uwkc_program_uwkc_opportunitydesignation
and the associated Opportunities to those:
programset(guid'thisone')/program-desig?$expand=desig-opportunity
...OrganizationData.svc/uwkc_programSet(guid'F0D75E9D-3A79-E611-80DA-C4346BACAAC0')/uwkc_uwkc_program_uwkc_opportunitydesignation?$expand=uwkc_opportunity_uwkc_opportunitydesignation
... but now I get a little stuck.
I can filter on a primitive value on the Opportunity (link + field)
...$filter=opp-oppdesig/EstimatedCloseDate gt DateTime('2016-07-01')
...OrganizationData.svc/uwkc_programSet(guid'F0D75E9D-3A79-E611-80DA-C4346BACAAC0')/uwkc_uwkc_program_uwkc_opportunitydesignation?$expand=uwkc_opportunity_uwkc_opportunitydesignation&$filter=uwkc_opportunity_uwkc_opportunitydesignation/EstimatedCloseDate%20gt%20DateTime%272016-07-01%27
and I can filter on a complex value on the Designation (field + value)
...$filter=statecode/Value gt 0
...OrganizationData.svc/uwkc_programSet(guid'F0D75E9D-3A79-E611-80DA-C4346BACAAC0')/uwkc_uwkc_program_uwkc_opportunitydesignation?$expand=uwkc_opportunity_uwkc_opportunitydesignation&$filter=statecode/Value%20gt%200
but I can't make a filter work on a complex value on the Opportunity (connection + field + value)
...$filter=opp-oppdesig/statecode/Value gt 0
...OrganizationData.svc/uwkc_programSet(guid'F0D75E9D-3A79-E611-80DA-C4346BACAAC0')/uwkc_uwkc_program_uwkc_opportunitydesignation?$expand=uwkc_opportunity_uwkc_opportunitydesignation&$filter=uwkc_opportunity_uwkc_opportunitydesignation/statecode/Value%20gt%200
No property 'statecode' exists in type 'Microsoft.Xrm.Sdk.Entity' at
position 45.
How can I filter on the state of an entity two away from what I'm looking at? Or, if there's a better way, what's the best way to prevent in-use programs from being deactivated?
First issue is that you should be using the schema name of the attribute (StateCode), and not the logical name (statecode).
However, I believe it will then just return another error message:
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
<code>-2147220989</code>
<message xml:lang="en-US">attributeName</message>
</error>
For some reason, it seems that filtering on complex types in an expanded entity does not work properly for the SOAP endpoint. And from what I have tested, the new Web API does not support this kind of depth in a query either yet.
One solution to your problem is to just fetch all the results, and then perform the filtering manually in your code. This of course works best if you can assume that there are not too many related entities retrieved in this kind of query. Also be sure to use $select to retrieve only the necessary attributes, as it greatly reduces the time it takes for a query to finish.
Another solution is to perform the query using FetchXML instead. This can be done either via the Web API, or as a SOAP request you construct yourself.
A third solution is to split your query into multiple queries, so you don't have to filter on a state two entities away in a query.
I have a question about how to approach a certain scenario before I get halfway through it and figure out it was not the best option.
I work for a large company that has a team that creates tools for the team mates to use that aren’t official enterprise tools. We have no access to the database directly, just access to an internal server to store our files to run and be able to access the main site with javascript etc (same domain).
What I am working on is a tool that has a ton of options in it that allow you to select that I will call “data points” on a page.
There are things like “Account status, Balance, Name, Phone number, email etc” and have it save those to an excel sheet.
So you input account numbers, choose what you need and then using IE Objects it navigates to the page and scrapes data you request.
My question is as follows..
I want to make the scraping part pretty Dynamic in the way it works. I want to be able to add new datapoints on the fly.
My goal or idea is so store the regular expression needed to get the specific piece of data in the table with the “data point option”.
If I choose “Name” it knows the expression for name in the database to run again the DOM.
What would be the best way about creating that type of function in Javascript / Jquery?
I need to pass a Regex to a function, have it run against the DOM and then return the result.
I have a feeling that there will be things that require more than 1 step to get the information etc.
I am just trying to think of the best way to approach it without having to hardcode 200+ expressions into the file as the page may get updated and need to be changed.
Any ideas?
IRobotSoft scraper may be the tool you are looking for. Check this forum and see if questions are similar to what you are doing: http://irobotsoft.org/bb/YaBB.pl?board=newcomer. It is free.
What it uses is not regular expression but a language called HTQL, which may be more suitable for extracting web pages. It also supports regular expression, but not as the main language.
It organizes all your actions well with a visual interface, so you can dynamically compose actions or tasks for changing needs.
I'm working on a JavaScript project which has an existing database (and hence an existing schema).
To add a new feature I must add a new column to the table. This will store a property of objects of a class.
Currently there is a SELECT query in the project which queries a bunch of fields from the database table every time the application starts and then puts these obtained results into the various needed JavaScript objects.
So, now it looks like:
let FIELDS = "field1, field2, field3";
let query = "SELECT " + FIELDS + "FROM FOO_TABLE";
Somewhere else, this query is made whenever needed (usually after app restart).
I thought of changing it to:
let FIELDS = "field1, field2, field3, new_prop";
But it won't work as in the current table, such a table doesn't exist. (Maybe after next restart, things will work, but not the first time).
What is the workaround?
Also, please note that a silent change will be better than one that will show that this property is new to everyone who works on the file.
Well you can't just add the field into the query, that won't work, so you have 2 options.
Use exceptions/errors to work out if something goes wrong. I'm not an sqlite guru (in fact i've never used sqlite directly, always via API's as with ios dev.), but if you try to run a query containing a column name that is non-existent, something will error which you can capture, work out the error code/string and decide to re-run the query minus the bad column.
This is not a good idea, its dirty and expensive. I've used exceptions to save dupe check query's in the past, but even that isn't the correct way to do things... but it does work
The only "proper" way to do what you want is to first check if the column exists. In sqlite, i don't think there's a simply select command like there is in mysql. But you can use the PRAGMA table_info('table-name') statement to get all columns, then check if your column exists prioir to running the query. This is the correct way to do things.
Having said that, if your working in a collaborative/team dev environment, you should have a much cleaner way of upgrading everyone in the group. So i'd be more tempted to address that issue of procedure rather than code my way around it.
On the website I'm developing, I have a search box with which I would like to incorporate a live autosuggest feature. (If you know a better way than the route I'm going please let me know)
Here's what I have in mind:
As the user types in the search bar, when the onchange event is triggered I want to send the query to the server (via ajax or websockets), then build a regex from the query (/^SOMELETTERS/gi), then search multiple fields (product names, brands, product numbers...) and match them to the regex. I want to gather the top 10 results and send them to the client side in json format to be used in an autosuggest script that's kind of like what google has.
These are my questions:
(1) Is there a better way?
(2) If no, how would I build a regex that matches items that begin with whatever the query is? ...I mean, what would the syntax be? I've tried a lot of things but they don't seem to work.
Call the constructor of the RegExp object. MSDN Docs.
If you have req.params.q you can:
var rx = new Regex(req.params.q);