Develop Reference

How should I use refresh tokens with Google Picker and Google Drive?

The problem:
I have a app on google script's platform, that's meant to allow uploads to Google Drive without using any account. The upload feature works well but I'm having issues with very long/big uploads. I'm trying to solve this since a week now, mostly because I need to test the expiration of the tokens.
When a user tries to upload a big file (20/30 GB) to the server, the Auth token expires Error Screenshot 1 and then I get this error Error screenshot 2.
So, what I need is to use a token that would expire in more than 5 hours. I did try to use a refresh token but I ended up very confused. I did created the refresh token in OAuth 2.0 Playground.
Things I've tried:
Pass the refresh token in the setOAuthToken. (Rejected by the function)
Use the refresh token to use setOAuthToken but it failed.
Questions
Can I permanently authorize the app access to Picker? (since it's always the same user accesing the drive on the server side code)?
Should I use a refresh token to obtain an Auth token?
Original Code:
var a = (new google.picker.PickerBuilder)
.addView(t)
.enableFeature(google.picker.Feature.NAV_HIDDEN)
.setOAuthToken("<?= ScriptApp.getOAuthToken(); ?>")
.enableFeature(google.picker.Feature.MULTISELECT_ENABLED)
.hideTitleBar()
.setSize(DIALOG_DIMENSIONS.width - 2, DIALOG_DIMENSIONS.height - 2)
.setCallback(pickerCallback).setOrigin(config.FORM_EMBED_DOMAIN)
.build()
Any help will be extremely appreciated.

afaik, the Picker can't take a Refresh Token and use this to renew its Access Tokens. This is almost certainly by design, since Refresh Tokens should never be on an insecure device such as a browser.
The only approach I can suggest would be to:-
A 1. have a Refresh Token on a secure server
implement your own endpoint to return an Access Token using the stored refresh Token
or
B 1. Use gapi, immediate=true (or however you currently obtain an Access Token)
Have a setTimeout/setInterval function which every 59 minutes, gets a new Access Token using option A or B
Poke this into the Picker object by finding the internal property where the Access Token is stored.
This is fugly and fragile, but I honestly can't think of a better answer.

You can dispose created picker object after 1hr, and create a new one with freshly obtained access_token
https://developers.google.com/picker/docs/reference#picker
Look at method dispose in API description

The suggested solutions did not solve the problem.
I tried the same using Google forms, I tried to upload the same files I used to test the error described in the original question. It turns out, I have the exactly same error!
So, I think is a case of "worked as design". I already sent a error report to Google, we have a G Suite Account, I hope we receive some feedback. But I think is not something easy to solve.
The main problem with the google form alternative, is that it requires a Gmail/Google account, and if the files you want to upload are bigger than your free quota, the upload will fail. I'm trying with a personal account with 21 GB (the uploader) and an unlimited G Suite account (receiver and form owner).
So,
After a lot of testing different options, the easiest/fastest solution is to limit the clients to upload up to 3 files (because you can upload 3 files at the time during the beginning of the process). When you try to upload the 4th file you'll get an authentication error.
Case closed!

I am trying to scrape a website with Node.js, but I got blocked from the website. How can I solve this?

http://pslsource.com/buy_tennessee_titans_psl/
This is the website I'm trying to scrape. It has some tables with prices, and I can't access it through my Node app anymore (which is on Heroku). How can I re-access it again? Any proxies or other suggestions?
Here's the code as well:
var Horseman = require("node-horseman");
var h = new Horseman({timeout: 50000});
h.open("http://pslsource.com/buy_tennessee_titans_psl/")
.waitForSelector("body")
.text("body")
.then(function(data) {
console.log(data);
});
Note: All credits will be paid to the original website, I won't claim the data as my own.

The fact that you have been blocked from the site clearly shows that crawling the site is against the intention of the owners. You could "solve" this by using proxies and send requests to their APIs.
However, if crawling the site is in fact a legitimate action, then send an email to them and explain them why do you intend to crawl their site, convince them that your intentions are not against their interests. They might give you an export. If not, then do not try to extract data from there.

Twitter JS API, update status

I'm at a total loss here. I believe I'm right in thinking there is no longer any JS API for twitter which just sucks hugely.
However I realise looking at this I could just use ajax and react to the responses from this:
https://dev.twitter.com/rest/reference/post/statuses/update
OAuth Signature Generator
Example Request POST
https://api.twitter.com/1.1/statuses/update.json?status=Maybe%20he%27ll%20finally%20find%20his%20keys.%20%23peterfalk
However when I post to that url I get the following:
{"errors":[{"code":215,"message":"Bad Authentication data."}]}

Bad Authentication data -> this is very self explanatory: Your Authentication is Bad; in other word, you are not authorized to use that API method.
Since you use POST method, make sure you set your APP access level to Read & Write.
Sign in to apps.twitter.com, and in the Applications Management page for your app, click the Permissions tab. There you can change your access level.
For further reading, please see this answer.

Simple, library-free, Javascript method for inserting a file into Google Drive

I am using a web scripting service that generates files and can POST them to a URI. If I don't have or provide one, they are stored on an ftp server. What I want to do is upload these files to my Google Drive, and, eventually, other users' Drives as well.
Because I'm using a 'private' host, and 'pure' javascript--no browser-- I can't add any framework or library. I have to do everything by hand.
I've been through all the documentation, read many posts here, found some really good information, and have lost some hair.
Here's where the project is at the moment:
I'm using oauth2, web server flow.
I can request access, and get an auth code, like so:
https://accounts.google.com/o/oauth2/auth?
response_type=code
&scope=https://www.googleapis.com/auth/drive
&redirect_uri=http://myserver.com/
&access_type=offline
&state=testing
&client_id={CLIENT_ID].apps.googleusercontent.com
This gets me an auth code. So far, so good.
Then, I take that code and do:
POST /o/oauth/token HTTP/1.1
Host: https://accounts.google.com
client_id=yada_yada
&client_secret=more_yada
&redirect_uri=http://myserver.com/
&code=[the code I just got]
&grant_type=authorization_code
Now, this step fails if I use XMLHttpRequest; I get an "Access-Control-Allow-Origin' violation. So, I slapped together an HTML page with a form and posted that. I get back a JSON object containing an access_token and a refresh_token. Yay.
{
'access_token' : [Access Yada-Yada],
'expires_in' : 3600,
'token_type' : 'Bearer',
'refresh_token' : '[Refresh Yada-Yada]
}
Now, I can get my script to work for one hour. Yippee.
Now, after all that, how do I use the refresh_token to get a new access_token. I have tried everything. XMLHttpRequests give me the same cross-domain violation, and doing an HTML form gets me an 'invalid_request' response! I tried using curl, never having touched it before, and that did not go well.
Please help.
Edit
The format for the refresh request goes like this:
var xhr = XMLHttpRequest();
xhr.open('POST',"https://accounts.google.com/o/oauth/token?" +
"client_id=[CLIENT_ID]"+
"&client_secret=[CLIENT_SECRET]"+
"&refresh_token=[REFRESH_TOKEN]"+
"&grant_type=refresh_token",
true);
xhr.setRequestHeader('Content-Type','x-www-form-urlencoded');
xhr.send();
I copied the url parameters directly from the OAuth Playground, which I had given my project keys. I get the Access-Control-Allow-Origin violation when I do this.
If I wrap it in a form, like so:
<form method="GET" action="https://accounts.google.com/o/oauth2/token">
<div>REFRESH TOKEN:</div>
<input type="text" name="refresh_token" value="[REFRESH_TOKEN]">
<br><div>CLIENT ID:</div>
<input type="text" name="client_id" value="[CLIENT_ID]">
<br><div>CLIENT SECRET:</div>
<input type="text" name="client_secret" value="[CLIENT_SECRET">
<br><div>GRANT TYPE:</div>
<input type="text" name="grant_type" value="refresh_token">
</form>
I can get a new access token. But, I can't get the same thing to work in JS using XMLHttpRequest. I keep getting XD errors.
I just need a way to post a refresh request to google from javascript without using any libraries.

The webserver flow is not adequate for an application that only has client-side logic. I recommend you use the G+ Sign-In widget (which can be customized for any scope, including Drive and actually works for all Google users whether or not they happen to have a Google+ account, including all Google Apps users). https://developers.google.com/+/
If you'd rather do it all yourself, you can read on how the client-side flow works: Essentially you ask for 'response_type=token' and parse the access_token from the fragment. There's no refresh_token for access_token exchange step. You can periodically refresh the token in the background using an invisible iframe by just repeating the initial authorization request. make sure you add display=none to requests you make in invisible iframe. See https://developers.google.com/accounts/docs/OAuth2UserAgent for more information.

Publish data from browser app without writing my own server

I need users to be able to post data from a single page browser application (SPA) to me, but I can't put server-side code on the host.
Is there a web service that I can use for this? I looked at Amazon SQS (simple queue service) but I can't call their REST APIs from within the browser due to cross origin policy.
I favour ease of development over robustness right now, so even just receiving an email would be fine. I'm not sure that the site is even going to catch on. If it does, then I'll develop a server-side component and move hosts.

Not only there are Web Services, but nowadays there are robust systems that provide a way to server-side some logic on your applications. They are called BaaS or Backend as a Service providers, usually to provide some backbone to your front end applications.
Although they have multiple uses, I'm going to list the most common in my opinion:
For mobile applications - Instead of having to learn an API for each device you code to, you can use an standard platform to store logic and data for your application.
For prototyping - If you want to create a slick application, but you don't want to code all the backend logic for the data -less dealing with all the operations and system administration that represents-, through a BaaS provider you only need good Front End skills to code the simplest CRUD applications you can imagine. Some BaaS even allow you to bind some Reduce algorithms to calls your perform to their API.
For web applications - When PaaS (Platform as a Service) came to town to ease the job for Backend End developers in order to avoid the hassle of System Administration and Operations, it was just logic that the same was going to happen to the Backend. There are many clones that showcase the real power of this strategy.
All of this is amazing, but I have yet to mention any of them. I'm going to list the ones that I know the most and have actually used in projects. There are probably many, but as far as I know, this one have satisfied most of my news, whether it's any of the previously ones mentioned.
Parse.com
Parse's most outstanding features target mobile devices; however, nowadays Parse contains an incredible amount of API's that allows you to use it as full feature backend service for Javascript, Android and even Windows 8 applications (Windows 8 SDK was introduced a few months ago this year).
How does a Parse code looks in Javascript?
Parse works through classes and objects (ain't that beautiful?), so you first create a specific class (can be done through Javascript, REST or even the Data Browser manager) and then you add objects to specific classes.
First, add up Parse as a script tag in javascript:
<script type="text/javascript" src="http://www.parsecdn.com/js/parse-1.1.15.min.js"></script>
Then, through a given Application ID and a Javascript Key, initialize Parse.
Parse.initialize("APPLICATION_ID", "JAVASCRIPT_KEY");
From there, it's all object manipulation
var Person = Parse.Object.extend("Person"); //Person is a class *cof* uppercase *cof*
var personObject = new Person();
personObject.save({name: "John"}, {
success: function(object) {
console.log("The object with the data "+ JSON.stringify(object) + " was saved successfully.");
},
error: function(model, error) {
console.log("There was an error! The following model and error object were provided by the Server");
console.log(model);
console.log(error);
}
});
What about authentication and security?
Parse has a User based authentication system, which pretty much allows you to store a base of users that can manipulate the data. If map the data with User information, you can ensure that only a given user can manipulate specific data. Plus, in the settings of your Parse application, you can specify that no clients are allowed to create classes, to ensure innecesary calls are performed.
Did you REALLY used in a web application?
Yes, it was my tool of choice for a medium fidelity prototype.
Firebase.com
Firebase's main feature is the ability to provide Real Time to your application without all the hassle. You don't need a MeteorJS server in order to bring Push Notifications to your software. If you know Javascript, you are half way through to bring Real Time magic to your users.
How does a Firebase looks in Javascript?
Firebase works in a REST fashion, and I think they do an amazing job structuring the Glory of REST. As a good example, look at the following Resource structure in Firebase:
https://SampleChat.firebaseIO-demo.com/users/fred/name/first
You don't need to be a rocket scientist to know that you are retrieve the first name of the user "Fred", giving there's at least one -usually there should be a UUID instead of a name, but hey, it's an example, give me a break-.
In order to start using Firebase, as with Parse, add up their CDN Javascript
<script type='text/javascript' src='https://cdn.firebase.com/v0/firebase.js'></script>
Now, create a reference object that will allow you to consume the Firebase API
var myRootRef = new Firebase('https://myprojectname.firebaseIO-demo.com/');
From there, you can create a bunch of neat applications.
var USERS_LOCATION = 'https://SampleChat.firebaseIO-demo.com/users';
var userId = "Fred"; // Username
var usersRef = new Firebase(USERS_LOCATION);
usersRef.child(userId).once('value', function(snapshot) {
var exists = (snapshot.val() !== null);
if (exists) {
console.log("Username "+userId+" is part of our database");
} else {
console.log("We have no register of the username "+userId);
}
});
What about authentication and security?
You are in luck! Firebase released their Security API about two weeks ago! I have yet to explore it, but I'm sure it fills most of the gaps that allowed random people to use your reference to their own purpose.
Did you REALLY used in a web application?
Eeehm... ok, no. I used it in a Chrome Extension! It's still in process but it's going to be a Real Time chat inside a Chrome Extension. Ain't that cool? Fine. I find it cool. Anyway, you can browse more awesome examples for Firebase in their examples page.
What's the magic of these services? If you read your Dependency Injection and Mock Object Testing, at some point you can completely replace all of those services for your own through a REST Web Service provider.
Since these services were created to be used inside any application, they are CORS ready. As stated before, I have successfully used both of them from multiple domains without any issue (I'm even trying to use Firebase in a Chrome Extension, and I'm sure I will succeed soon).
Both Parse and Firebase have Data Browser managers, which means that you can see the data you are manipulating through a simple web browser. As a final disclaimer, I have no relationship with any of those services other than the face that James Taplin (Firebase Co-founder) was amazing enough to lend me some Beta access to Firebase.

You actually CAN use SQS from the browser, even without CORS, as long as you only need the browser to send messages, not receive them. Warning: this is a kludge that would make my CS professors cry.
When you perform a GET request via javascript, the browser will always perform the request, however, you'll only get access to the response if it was from the same origin (protocol, host, port). This is your ticket to ride, since messages can be posted to an SQS queue with just a GET, and who really cares about the response anyways?
Assuming you're using jquery, your queue is https://sqs.us-east-1.amazonaws.com/71717171/myqueue, and allows anyone to post a message, the following will post a message with the body "HITHERE" to the queue:
$.ajax({
url: 'https://sqs.us-east-1.amazonaws.com/71717171/myqueue' +
'?Action=SendMessage' +
'&Version=2012-11-05' +
'&MessageBody=HITHERE'
})
The'll be an error in the console saying that the request failed, but the message will show up in the queue anyways.

Have you considered JSONP? That is one way of calling cross-domain scripts from javascript without running into the same origin policy. You're going to have to set up some script somewhere to send you the data, though. Javascript just isn't up to the task.

Depending in what kind of data you want to send, and what you're going to do with it, one way of solving it would be to post the data to a Google Spreadsheet using Ajax. It's a bit tricky to accomplish though.Here is another stackoverflow question about it.
If presentation isn't that important you can just have an embedded Google Spreadsheet Form.

What about mailto:youremail#goeshere.com ? ihihi
Meantime, you can turn on some free hostings like Altervista or Heroku or somenthing else like them .. so you can connect to their server , if i remember these free services allows servers p2p, so you can create a sort of personal web services and push ajax requests as well, obviously their servers are slow for free accounts, but i think it's enought if you do not have so much users traffic, else you should turn on some better VPS or Hosting or Cloud solution.

Maybe CouchDB can provide what you're after. IrisCouch provides free CouchDB instances. Lock it down so that users can't view documents and have a sensible validation function and you've got yourself an easy RESTful place to stick your data in.