How to query a bigquery view from bigquery APIs - javascript

I have a view in bigquery which contains fields from different datasets and tables, Now I would like to query this view through my google script. What is correct way of doing that.
Actually Currently I have created a separate table in bigquery and querying the table instead of view but I need view as view will get updated when the dependency tables will be updated.
If I am using the table, It is working fine but in case of view I am getting below error:
Exception: Response Code: 404. Message: Not Found.
Bigquery api's to return the result of query.
try {
var job = BigQuery.newJob();
var config = BigQuery.newJobConfiguration();
var queryConfig = BigQuery.newJobConfigurationQuery();
queryConfig.setQuery(sql);
queryConfig.setMaximumBillingTier(5);
config.setQuery(queryConfig);
job.setConfiguration(config);
var jobid = BigQuery.Jobs.insert(job, projectNumber).jobReference;
queryResults = BigQuery.Jobs.getQueryResults(projectNumber, jobid.jobId);
}
catch (err) {
Logger.log(err);
Browser.msgBox(err);
return;
}
// Check on status of the Query Job : MONTHLY
while (queryResults.getJobComplete() == false) {
try {
queryResults = BigQuery.Jobs.getQueryResults(projectNumber, queryResults.jobId);
//queryResults = BigQuery.Jobs.getQueryResults(projectNumber, job.id);
}
catch (err) {
Logger.log(err);
Browser.msgBox(err);
return;
}
}
return queryResults;
If I comment out my first try clause and use below one
try {
var queryRequest = BigQuery.newQueryRequest();
queryRequest.setQuery(sql).setTimeoutMs(100000);
queryResults = BigQuery.Jobs.query(queryRequest, projectNumber);
//Browser.msgBox(queryResults);
}
catch (err) {
Logger.log(err);
Browser.msgBox(err);
return;
}
then It starts giving me
Exception: Query exceeded resource limits for tier 1. Tier 3 or higher required.

It looks like the 'configuration.query.maximumBillingTier' property isn't getting set when you are inserting the job. The method of using 'JobConfigurationQuery' and other classes seems to have been abandoned as there is no mention of them in the current docs, and I needed to resort to using the Wayback Machine to find them.
The most recently available document from 11/12/2013 only defines getters and setters for a few configuration properties, and 'maximumBillingTier' isn't one of them.
I'd suggest manually setting the request properties as is done in the usage examples from the current documentation, rather than relying on the "old" object constructors, as they seem to only be left around for compatibility purposes and are incomplete.
The reason a view would require a higher billing tier versus a table, by the way, is because a view is only a logical table and the queries which define the view must be re-executed when the view itself is queried.

Related

How can I check if a data search was successful in Firebase

I am checking if a combination of child values exists in my DB.
This works but can't get the proper message to be displayed in the console.
the correct console message is displayed in both else statements. The one that is not being displayed properly is the console.log('yes').
Or actually it is displayed but always followed by aconsole.log('no') and I have no idea why
There is however a match found because the correct data is shown in console.log(details); console.log(snapshot.key);
FBSearch: function(){
var T1 = this
var T2 = this
var T3 = this
var ref = firebase.database().ref("flights");
ref
.orderByChild('date')
.equalTo(T2.flight.dat)
.on('child_added', function(snapshot) {
var details = snapshot.val();
if (details.flight == T1.flight.flt) {
if(details.origin == T3.flight.org){
console.log(details);
console.log(snapshot.key);
console.log('yes')
} else {
console.log('no')
}
} else {
console.log('no')
}
})
}
The desired outcome is a success message when a match is found
Not sure if this is the answer to your question, but something you might want to consider...
Right now you're reading all flights of the specific date, in an effort to check if one of them exists. This is suboptimal, and can waste lot of your users' bandwidth. Ideally a yes/no question like that should require you to pass in the criteria, and get a minimal yes/no type answer.
While the Firebase Realtime Database doesn't support exists() queries, and can't query on multiple conditions, you can make significant improvements by adding a special property to your data to support this query.
As far as I can see, you're filtering on three properties: date, flight, and origin. I recommend adding an extra property to all flights that combines the values of these properties into a single value. Let's call this property "origin_date_flight".
With that property in place you can run the following query to check for the existence of a certain combination:
ref
.orderByChild('origin_date_flight')
.equalTo(`${T3.flight.org}_${T2.flight.date}_${T1.flight.flt}`)
.limitToFirst(1)
.once('value', function(snapshot) {
if (snapshot.exists()) {
console.log('yes')
} else {
console.log('no')
}
})
The main changes here:
The query uses a composite property as a sort-of index, which allows it to query for the specific combination you're looking for.
The query uses limitToFirst(1). Since you're only looking to see if a value exists, there's no need to retrieve multiple results (if those exist).
The query uses once("value", which allows it to check for both existence and non-existence in a single callback. The child_added event only fired if a match exists, so can't be used to detect non-existence.
And as a bonus, this new property also allow you to for example get all flights out of Amsterdam today:
ref
.orderByChild('origin_date_flight')
.startAt(`AMS_20190728_${T2.flight.date}_`)
.endAt(`AMS_20190728_${T2.flight.date}~`)
.once('value', function(snapshot) {
snapshot.forEach(function(flightSnapshot) {
console.log(flightSnapshot.key+": "+flightSnapshot.child("flt").getValue());
})
})
Also see:
Query based on multiple where clauses in Firebase

Meteor remote collection - hooks don’t work

I have to connect to the external database and get access to its collections. It works fine, when I use it, but the problem is when I need collection hooks, e.g. Collection.after.insert(function(userId, doc)). The hook is not being fired. I have following code:
// TestCollection.js
let database = new MongoInternals.RemoteCollectionDriver("mongodb://127.0.0.1:3001/meteor",
{
oplogUrl: 'mongodb://127.0.0.1:3001/local'
});
let TestCollection = new Mongo.Collection("testCollection", { _driver: database });
module.exports.TestCollection = TestCollection;
console.log(TestCollection.findOne({name: 'testItem'})); // writes out the item correctly
// FileUsingCollection.js
import { TestCollection } from '../collections/TestCollection.js';
console.log(TestCollection.findOne({name: 'testItem'})); // writes out the item correctly second time
TestCollection.after.update(function (userId, doc) {
console.log('after update');
}); // this is NOT being fired when I change the content of remote collection (in external app, which database I am connected)
How to make this work?
EDIT:
I have read many hours about it and I think it might be connected with things like:
- oplog
- replicaSet
But I am newbie to Meteor and can’t find out what are those things about. I have set MONGO_OPLOG_URL and I added oplog parameter to database driver as I read here: https://medium.com/#lionkeng/2-ways-to-share-data-between-2-different-meteor-apps-7b27f18b5de9
but nothing changed. And I don’t know how to use this replicaSet, how to add it to the url. Anybody can help?
You can also try something like below code,
var observer = YourCollections.find({}).observeChanges({
added: function (id, fields) {
}
});
You can also have 'addedBefore(id, fields, before)', 'changed(id, fields)', 'movedBefore(id, before)', 'removed(id)'
For more features goto link.

Node js - Request Error transaction was deadlocked

I'm having problems when i insert several data using promise, sometimes it works but other times give me this error:
And my code is this:
return Promise.all([
Promise.all(createBistamp),
Promise.all(createSlstamp),
listOfResults,
i
]).then(function(listOfResults2) {
for(var j=0; j<resultArticle.length; j++) {
if(arm === 'Arm-1') {
}
if(arm === 'Arm-1-11') {
}
}
if(arm === 'Arm-1') {
console.log("PROMISE ARM-1");
return Promise.all([insertBi,insertBi2,insertSl]).then(function (insertEnd) {
res.send("true");
}).catch(function(err) {
console.log(err);
});
}
if(arm === 'Arm-1-11') {
console.log("PROMISE ARM-1-11");
return Promise.all([insertBi,insertBi2,insertSl,insertSlSaida]).then(function (insertEnd) {
res.send("true");
}).catch(function(err) {
console.log(err);
});
}
}).catch(function(err) {
console.log(err);
});
I remove the code line inside ifs and for but it was inserts in database.
Example of insert:
var insertBi2 = request.query("INSERT INTO bi2 (bi2stamp,alvstamp1,identificacao1,szzstamp1,zona1,bostamp,ousrinis,ousrdata,ousrhora,usrinis,usrdata,usrhora)"+
"VALUES ('"+bistamp+"','AB16083056009,454383576','2','Adm13010764745,450449475','1','"+bostamp+"','WWW','"+data+"','"+time+"','WWW','"+data+"','"+time+"')");
Full Code:
http://pastebin.com/DTjtXvDt
This is my structure and i don't know if i'm working well with promises.
Thank you
I have also faced this problem recently.
error: RequestError: Transaction (Process ID 72) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Solution -
There was not a single index on the table. So, I created a non-clustered unique index on the unique identifier column.
I was surprised when this solution worked
There was a single update operation in the code and no select operation. So, it made me curious to do some research. I came across lock granularity mechanism for locking resources. In my case, locking has to be at row level instead of page level.
Note:
For clustered tables, the data pages are stored at the leaf level of the (clustered) index structure and are therefore locked with index key locks instead of row locks.
Further Reading
https://www.sqlshack.com/locking-sql-server/
If you are inserting data or updating data in a loop, then it's better to make all queries in the loop and store it and then execute it all at once in a single transaction. Will save yourself with a lot of issues

can't set headers after they are sent node js when using restify module

I have used tedious to connect to sql server and restify for restful api
here is the server.js
server.get('/getInvoiceData', function (req, res, next) {
repository.GetInvoiceData(function(data){
res.send(data);
next();
});
});
and the invoice.js
exports.GetInvoiceData = function(callback){
var query = "SELECT * FROM [Snapdeal].[dbo].[tbl_Configuration]";
var req = new request(query,function(err,rowcount){
if (err)
{
console.log(err.toString());
}else{
console.log(rowcount+ " rows");
}
});
req.on('row',function(){
callback({customernumber:123});
});
connection.execSql(req);
}
I am getting the error as Cant set the headers after they are sent.
I am not 100% sure as I am not familiar with the SQL lib you are using, however, it looks to me like the problem is your row event would be raised per row, rather than per transaction.
You are ending the response after the first row event therefore if there is more than one row being returned the response will already have been closed (hence the error).
One way of dealing with this is to accumulate the row data as it's being retrieved and then raise the callback after your done
Now that you have stated the lib you are using (Tedius), it would appear my hunch was correct. Looking at the library, here is the simplest approach you can take to returning all the rows in a single callback
exports.GetInvoiceData = function(callback){
var query = "SELECT * FROM [Snapdeal].[dbo].[tbl_Configuration]";
var req = new request(query,function(err, rowcount, rows){
if (err) {
console.log(err.toString());
} else{
callback(rows);
}
});
connection.execSql(req);
}
Note - remember to set config.options.rowCollectionOnRequestCompletion to true otherwise the rows parameter will be empty.
My issue, using mssql, was that I had a default value or binding set (in this case, (getdate())) to one of my columns (modified date column). However, the data I was trying to retrieve had preset NULL values for this particular column.
I put data in those rows and I was good to go.

Add a new field to a document mongodb

I am very new to mongodb and have a basic question that I am having trouble with. How do I get the ID field of a document that has already been created? I need the ID so i can update/add a new field to the document.
//newProfile is an object, one string it holds is called school
if(Schools.find({name: newProfile.school}).fetch().length != 1){
var school = {
name: newProfile.school
}
Meteor.call('newSchool', school);
//Method 1 (doesn't work)
var schoolDoc = Schools.findOne({name: newProfile.school});
Schools.update({_id: schoolDoc._id}, {$set: {enrolledStudents: Meteor.user()}});
//Method 2?
//Schools.update(_id: <what goes here?>, {$push: {enrolledStudents: Meteor.user()}});
}
else {
//Schools.update... <add users to an existing school>
}
I create a new school document if the listed school does not already exist. Schools need to hold an array/list of students (this is where i am having trouble). How do I add students to a NEW field (called enrolledStudents)?
Thanks!
I'm having some trouble understanding exactly what you're trying to do. Here's my analysis and understanding so far with a couple pointers thrown in:
if(Schools.find({name: newProfile.school}).fetch().length != 1){
this would be more efficient
if(Schools.find({name: new Profile.school}).count() != 1) {
Meteor.call('newSchool', school);
Not sure what you're doing here, unless you this will run asynchronously, meaning by the time the rest of this block of code has executed, chances are this Meteor.call() function has not completed on the server side.
//Method 1 (doesn't work)
var schoolDoc = Schools.findOne({name: newProfile.school});
Schools.update({_id: schoolDoc._id}, {$set: {enrolledStudents: Meteor.user()}});
Judging by the if statement at the top of your code, there is more than one school with this name in the database. So I'm unsure if the schoolDoc variable is the record you're after.
I believe you are having trouble because of the asynchronous nature of Meteor.call on the client.
Try doing something like this:
// include on both server and client
Meteor.methods({
newSchool: function (school) {
var newSchoolId,
currentUser = Meteor.user();
if (!currentUser) throw new Meteor.Error(403, 'Access denied');
// add some check here using the Meteor check/match function to ensure 'school'
// contains proper data
try {
school.enrolledStudents = [currentUser._id];
newSchoolId = Schools.insert(school);
return newSchoolId;
} catch (ex) {
// handle appropriately
}
}
});
// on client
var schoolExists = false;
if (Schools.findOne({name: newProfile.school})) {
schoolExists = true;
}
if (schoolExists) {
var school = {
name: newProfile.school
};
Meteor.call('newSchool', school, function (err, result) {
if (err) {
alert('An error occurred...');
} else {
// result is now the _id of the newly inserted record
}
})
} else {
}
Including the method on both the client and the server allows Meteor to do latency compensation and 'simulate' the insert immediately on the client without waiting for the server round-trip. But you could also just keep the method on the server-side.
You should do the enrolledStudents part on the server to prevent malicious users from messing with your data. Also, you probably don't want to actually be storing the entire user object in the enrolledStudents array, just the user _id.
For what you're trying to do, there is no need to get the _id. When you use update, just switch out the {_id: schoolDoc._id} with your query. Looks like using {name: newProfile.school} will work, assuming that the rest of your code does what you want it to do.
While that would work with the normal Mongo driver, I see that Meteor does not allow your update query to be anything but _id: Meteor throws throwIfSelectorIsNotId exception
First, make sure that you're pulling the right document, and you can try something like this:
var school_id = Schools.findOne({name: newProfile.school})._id;
Schools.update({_id: school_id}, { $push: { enrolledStudents: Meteor.user()}});
If that doesn't work, you'll have to do a little debugging to see what in particular about it isn't working.

Categories

Resources