Develop Reference

MySQL query with LIMIT and updating data by chunks

I have a table in my mysql db with columns id and drivers_license_number. Now data in drivers_license_number column is stored in plaintext, and I want to encrypt it.
This table contains about 400000 records. I have read about problems with LIMIT and OFFSET issue and try to use in my query late row lookups.
So first of all I need to get data from db, then encrypt the right field and update db with this encrypted field. I don't understand, how to organize my code to send parameters with limit and offset to db. I need to work in loop in this situation?
function getUpdatedEncryptedField(db_table_name, field_name) {
return getDataFromDb(db_table_name, field_name).then(function(result){
return encryptData(result, field_name).then(function (result){
var promises = result.map(function(item) {
var data = [item[field_name], item.id];
return updateFieldNameData(db_table_name, field_name, data);
});
return q.all(promises);
});
});
}
After the first pass I will get for example first 1000 records and how to move forward to get another 1000 rows?
function getDataFromDb(db_table_name, field_name, limit, offset) {
var sql = 'SELECT ds.id, ' + field_name + ' FROM ( SELECT id FROM ' + db_table_name + 'WHERE ' + field_name +
' IS NOT NULL ORDER BY id LIMIT ' + limit + ', ' + offset + ') d JOIN ' + db_table_name + ' ds ON ds.id = d.id ORDER BY ds.id;'
return db.modio.executeSql(sql);
}

You can use a carring/high order function to update the chunks by e.g 1000 and get the next 1000 records until the 400000, like so:
function getUpdatedEncryptedField(db_table_name, field_name) {
var idFrom = 0;
var idTo = 1;
return function getDataFromDB(db_table_name, field_name){
idFrom = idTo;
idTo += 1000;
console.log(id range is ${idFrom}-${idTo}`);
// call your db to get the recods
}
}
// assign high order func to updateChunks var, which is again a func
var updateChunks = getUpdatedEncryptedField('studentsDB','creditCard')
// so each time you execute updateChunks() the values idFrom & idTo are updated
// updateChunks()
// id range is 1-1001
// updateChunks()
// id range is 1001-2001
// and so on
// create a loop until you get all the table records, e.g.
// pay attention to define 'i' with let and not with var because of async loop
for(let i=0; i <= 400000/1000; i++){
// call the updateChunk func here
// at eaach itteration it will ask for the next chunk of records
updateChunks()
}
`
Of course you should update a little bit your mysql query, to get the range of Ids from - to, this is out of the question scope.
Hope helps, good luck.

Inserting a string at a variable index

I have not been able to find a working example or a good explanation of how I can achieve the following: (I would appreciate if anyone can point me in the right direction.
I have a query string: **"/api/bla?sources=[1,2]&plans=[1,2]&codes=[1,2,3]"**
I will be updating the query string via either javascript or jquery when certain events occur on my page, doesnt matter which.
For example, there is a multi select dropdown on the page which houses [sources] and [plans] and [codes]... These dropdowns have IDs which i am to update my request url with upons selecting items in teh dropdowns.
When a source with ID "3" is selected from the dropdown (or checkbox, doesnt matter what page controls are being used) the query string parameter sources[1,2] will need a "3" appended. Likewise then if the item with an ID of "2" is unselected, it will likewise be removed from the query string leaving the new string as sources[1,3]
I am somewhat new to javascript/jquery and especially more advanced string manipulation. I have been attempting to recreate something to demonstrate this and have gotten to the following which is not fully working.
Basically my initial if statement works as intended, but the moment the else is hit (when another ID needs to be added to an existing model in the query string - like a second ID for [sources] or [codes]) it returns wonky output - seeng as I couldnt get the right formula to update everything correctly.
//TIMEMTABLE QUERY
function updateCalendar(filter_id, filter_element) {
//Run through the filter checks before making final call to query and update timetable?
//GET THE MODEL/OBJECT NAME
var queryName = filter_element.attr('data-owner');
//GET THE IDs //this is either an array of all selected IDs or a single id which is used in the else statement
var queryId = filter_element.attr('value');
var queryIds = $('#'+filter_id).val();
var modelCheckIndex = requestString.toLowerCase().indexOf(queryName.toLowerCase());
//build a request string
if (modelCheckIndex < 0) {
console.info('ADD MODEL TO QUERY STRING');
requestString = requestString + "&" + (queryName.toLowerCase() + "[" + queryIds + "]");
console.log(requestString);
}
else{
console.info('UPDATE MODEL ON QUERY STRING');
var position = requestString.toLowerCase().indexOf(queryName.toLowerCase());
//requestString = requestString.substr(modelCheckIndex -1, requestString.length -1) + "," + queryId + "]";
requestString = requestString.slice(modelCheckIndex.indexOf("]"), modelCheckIndex) + "," + queryId;
console.log(requestString);
}
//MAKE THE API CALL USING CREATED QUERY STRING
}
If anyone has any examples or fiddles lying around I would also appreciate it.
Fiddle I am trying to get to work

It looks like you are just having trouble parsing and updating a query string. In which case, I have a function I've been using for that (thank you Google)
function getUriParams(string) {
var params = {},
queryString = string.slice(string.lastIndexOf('?')).substring(1),
regex = /([^&=]+)=([^&]*)/g,
m;
while (m = regex.exec(queryString)) {
params[decodeURIComponent(m[1])] = decodeURIComponent(m[2]);
}
return params;
}
The input is your requestString and the output is an object of key value pairs of the query string.
To make the object a string, jQuery makes it easy with $.param().
//get key value pairs
var obj = getUriParams(requestString);
//make query string
var str = $.param(obj);

I suggest to change the logic a bit. I would use some data storage for the wanted parameter and rebuild the request string every time when it's necessary, like the below example.
It is much more better, than rebuild the string each time when some value has changed.
var data = {
sources: [1, 2],
plans: [],
codes: [1, 3, 4]
};
function buildStr() {
function get(key) { return key + '=' + JSON.stringify(data[key]); }
return '/api/bla?' + ['sources', 'plans', 'codes'].map(get).join('&');
}
document.write('<pre>' + buildStr() + '</pre>');

How to pass a formatted query to findOne() in mongo using node.js

I've one Restful service developed using node.js which takes some values from a user. According to the values given it reformats a query string. But, Whenever I want to pass this query to findone() of mongoDB and calls that service it showing "query selector must be an object" message on browser.
var query = "{" ;
if( orderNo == " "){
// Don't append anything
}else{
query = query + "'orderNo' : " + orderNo ;
}
.
.
.
query = query + "}";
And, I also tried like below :
var query = {};
query[orderNo] = orderNo;
db.collection('*****').findOne(query, function(err, item) {
console.log(item);
res.jsonp(item);
});
In both cases I'm getting the same result in browser.
Anyone guide me how I can query these kind of queries in MongoDB.

You were almost there on your second attempt. When you do this
var query = {};
query[orderNo] = orderNo;
you create an object with key name the same as the value! For example, if the value of orderNo is 12345, then the above statement will produce the object
query = {
"12345": 12345
}
You could use the same bracket notation to create a proper query object property
query = {
"orderNo": 12345
}
as follows:
var query = {};
if(orderNo != " ") query["orderNo"] = orderNo;
db.collection('*****').findOne(query, function(err, item) {
console.log(item);
res.jsonp(item);
});

eliminating line breaks in an array with javascript and regex

I have a file that looks like so:
QUERY:1
DATABASE:geoquery
NL:What are the capitals of the states that border the most populated states?
SQL:something
QUERY:2
DATABASE:geoquery
NL:What are the capitals of the states bordering New York?
SQL:SELECT state.Capital FROM state JOIN borderinfo ON state.State_Name = borderinfo.State_Name
WHERE borderinfo.Border = 'New York'
QUERY:3
DATABASE:geoquery
NL:Show the state capitals and populations.
SQL:SELECT state.Capital, state.Population FROM state
etc...
The person generating this file refuses to give it to me in a usable format like say, XML or JSON. So I am parsing it a couple of times with REGEX to get results I want.
Strip off Databases to populate select list (works fine):
$.get('inputQueryExamples.txt',
function(data){
var string = data;
var dbDynamo ='';
dbExp = new RegExp('(DATABASE:.*)','gm');
dbDynamo = string.match(dbExp);
cleanBreaks = new RegExp('(\r?\n|\r)','gm');
stringNow = dbDynamo.toString();
//console.log(dbDynamo);
dbDynamo = dbDynamo.map(function(el){return el.replace('DATABASE:','');});
var outArray = [];
for(i=0; i < dbDynamo.length; i++){
if ($.inArray(dbDynamo[i],outArray)== -1){
outArray.push(dbDynamo[i]);
}
}
dbDynamo = outArray.sort();
var options = '';
for(i=0; i<dbDynamo.length; i++){
options += '<option value="' + dbDynamo[i] + '">' + dbDynamo[i] + '</option>';
};
$(select).append(options);
});
The problem comes when I parse a second time to get all of the strings associated with a specific database. I end up with a linebreak in front of every string so that when I fill a textarea with the autocomplete text it starts on the second line:
Array [ "
NL:Show company with complaint against Debt collection product.,DATABASE:consumer", "
NL:Show issues and dates of complaints about HSBC companies.,DATABASE:consumer", "
NL:Show companies, issues and dates with consumer concerns.,DATABASE:consumer", "
NL:Show issues and companies with complaints in MA state." ]
$(document).ready(function(){
$.get('inputQueryExamples.txt',function(data){
var queryString = data;
var cleanString = "";
var db = '';
$('#database-list').change(function(){
db = $('#database-list').val();
// /(^DATABASE:.*\r\n)(^NL.*)/gm
// http://regex101.com/r/mN4hS2
regex = new RegExp('(^DATABASE:'+ db +'\r\n)(^NL.*)' ,'gm');
cleanString = queryString.match(regex);
//console.log(cleanString);
//cleanBreaks = new RegExp('(\r\n|\r|\n)(^NL.*)','gm');
//stringNow = cleanString.toString();
//var dirtyString
//dirtyString = stringNow.match(cleanBreaks);
//console.log(dirtyString);
var nlString = cleanString.map(function(el) {return el.replace('DATABASE:' + db,'');});
nlString = nlString.map(function(el){return el.replace('NL:',''); });
//nlString = nlString.map(function(el){return el.replace('\\r','').replace('\\n','');});
console.log(nlString.pop());
$('#query-list').autocomplete({
source:nlString,
});
}); // end change
I have tried about everything I can think of to get rid of the linebreaks without success. Any ideas would be appreciated. Unfortunately the one where the server side just gives me data in a usable format is not viable. There is a lot of extra code in this just to give you an idea of what I have tried. I have commented out useless things. Thanks in advance.

It would make sense to use JavaScript to parse the data-structure you are given into a JavaScript object first, then you can more easily work with the data.
Here is a jsfiddle that demonstrates parsing your data into a JavaScript object. Below is the relevant code that does the parsing:
var datasources = {};
var parseData = function(block) {
var db = block.match(/DATABASE:(.*)/)[1];
var dbdata = {id: 0, nl: "", sql: ""};
if (!datasources[db]) {
datasources[db] = [];
}
dbdata.id = block.match(/QUERY:(.*)/)[1];
dbdata.nl = block.match(/NL:(.*)/)[1];
dbdata.sql = block.match(/SQL:(.*)/)[1];
datasources[db].push(dbdata);
};
var parseBlocks = function(data) {
var result = data.split('\n\n');
for(var i=0; i < result.length; i++) {
parseData(result[i]);
};
};

Thanks for the very thoughtful and elegant approach. I continued the brute force approach.
replace:
nlString = nlString.map(function(el){return el.replace('NL:',''); });
with:
nlString = nlString.map(function(el){return el.replace('NL:','').trim(); });

Filling arrangement from a database

I need to fill an array with each of the values ​​for "name" and "nickname" taken from a SQLITE database and show it in an alert.
This is for a project written in JavaScript in titanium appcelerator.
This is the code:
var fightersRS = db.execute('SELECT * FROM fighters');
var setFighters = [];
while (fightersRS.isValidRow())
{
var name = fightersRS.fieldByName('name');
var nickname = fightersRS.fieldByName('nickname');
setFighters.push = {
name: name,
nickname: nickname
};
fightersRS.next();
}
alert(setFighters);
The idea is that the alert displays all comma separated values, for example:
"muhammad ali, rocky balboa, ten shin han, etc ...."
How I can do it? my code does not work.

Change your code as follows
var fightersRS = db.execute('SELECT * FROM fighters');
var setFighters = [];
while (fightersRS.isValidRow())
{
var name = fightersRS.fieldByName('name');
var nickname = fightersRS.fieldByName('nickname');
setFighters.push({"name": name, "nickname": nickname}); //This is the correct syntax for pushing an element to the array
fightersRS.next();
}
You can display the elements using a for loop as follows
for(var i=0; i <setFighters.length;i++){
alert("Name = " + setFighters[index].name + " Nickname= " + setFighters[index].nickname);
}
If you want to display their name with comma, try the following
var fightersRS = db.execute('SELECT * FROM fighters');
var setFighters = [];
while (fightersRS.isValidRow())
{
var name = fightersRS.fieldByName('name');
setFighters.push(name);
fightersRS.next();
}
var fighters = setFighters.join(",");//Joining the array elements with comma
alert(fighters); //Will display all names with a comma
Look at javascript join method

Use a for loop or inside the while loop only create a string appending all the values with comma.
Fetching all the data from an array with comma is not possible.
Here's an asccociative array example from stack over flow .. you can have your own for loop or use the while loop you are creating
Associative array and fetching data
Thanks