Develop Reference

'You Look Like a Robot' Error , while using puppeteer in firebase cloud functions

Im using the puppeteer package , in order to scrap a web page data that is fetched by clicking a button in this page
this are the presetting that I'm using:
const puppeteer = require('puppeteer-extra')
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())
// Add adblocker plugin to block all ads and trackers (saves bandwidth)
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker')
puppeteer.use(AdblockerPlugin({ blockTrackers: true }))
those setting are made in order that I will not be detected as a robot.
here what I'm doing :
(basically , creating a request by clicking a button , then this request return a json with a data that fill up some text info in a label , then I'm reading the data from that label
here's how im clicking the button :
const box = await btn.boundingBox();
const x = box.x + (box.width/2);
const y = box.y + (box.height/2);
console.log(x, y);
page.mouse.move(x,y,{step:1});
page.mouse.click(x,y)
await page.waitForTimeout(4000);
then afterwards ---> I'm getting the data from the data:
const [result] = await page.$x('//*[#id="content"]/div[1]/div[1]/div/div[2]/div');
// const txt = await result.evaluate.toString
let value = await page.evaluate(el => el.textContent, result);
console.log(value);
console.log('done?');
await browser.close();
const dic = {};
dic['status'] = 200;
dic['data'] = {"message": value};
response.send(dic);
I'm also using the 'on' method in order to see if the im getting a response from the action of clicking the button , like so:
await page.on('response', async response =>{
try {
console.succ(await response.json());
} catch (error) {
//
// console.error(error);
}
});
and it sure get one.
the problem is ---> that when I'm deploying it to the firebase cloud functions server,
firebase deploy --only functions
and then triggering the function -->
I'm getting a json that look like this :
{ success: false, message: 'You look like a robot.' }
But when deploying the same code to my local host like so
firebase serve --only functions
and then triggering the function -->
I'm not detected as a robot
and getting the json with a successful result --> and with that data that the clicking of a button supposed to fetch.
this is so weird , I'm trying to think that there's a connections between the firebase cloud functions and reCAPTCHA , because both are a google services
but, its not seem's reasonable for it to be true .
that being said, what could be the reason for this?
all that change is the environment that the code runs from.
do you have any idea why this is happening ?
and how to solve it of course .

Since your function runs properly locally, it's almost certainly not the function itself.
Sites take a variety of different approaches to detect bots, one of which is blocking traffic from known data centers like Google Cloud's. Using a residential IP proxy like those provided by BrightData will probably circumvent this.

I'm facing the same issue while using Puppeteer in Firebase Cloud Functions.
I'm using a residential IP proxy with the following set of packages puppeteer-extra, puppeteer-extra-plugin-stealth, puppeteer-extra-plugin-anonymize-ua', and user-agents`.
On localhost, all is working as expected while running Puppeteer in firebase Cloud Functions I'm getting a 404 response from the requested URL. So there must be some difference.

Want client-side Firebase logs to show up in StackDriver, not users' browsers

I'm using firebase-functions/lib/logger to log client-side firebase/firestore activity like
const { log, error } = require("firebase-functions/lib/logger");
export const addData = async (userId, dataId) => {
try {
const collectionRef = firestore
.collection("docs")
await collectionRef.add({
dataId,
});
log(`Data added`, { userId, dataId });
} catch (err) {
error(`Unable to add new data`, { userId, dataId });
throw new Error(err);
}
};
When I run this on my local, the log shows up in my browser console. Will this happen on non-local environments, ie for real users? Will these logs also show up automatically in Stackdriver, or are they stuck on the client side? I want to be able to view the logs either in Stackdriver or Firebase console but have them not show up in the browser for real users. How should I accomplish this?

Messages logged in Cloud Functions will not show up in the client app at all (that would probably be a security hole for your app). They will show up in the Cloud Functions console in the log tab, and in StackDriver.
Any messages logged in your app will not show up in any Google Cloud product. They are constrained to the device that generated them. If you want cloud logging, you'll need to implement some other solution. Cloud Functions does not support this - you will need to investigate other solutions or build something yourself.

PouchDB remote authentication password storage in browser

I'm playing around with in browser pouchdb and couchdb and and remote sync. This was all straight forward but now I'm at a roadblock that I'm not able to figure out myself.
For security reasons every user of the website has its own datbase that is only accessable via the respective couchdb user.
const localDB = new PouchDB('test');
const remoteDB = new PouchDB(process.env.REMOTE_DATABASE);
const username = process.env.REMOTE_USER;
const password = process.env.REMOTE_PASSWORD;
remoteDB.login(username, password, (error, response) => {
if (error) {
console.error('Login failure', error)
} else {
console.log('Login success');
dispatch('sync');
}
});
console.log('sync');
localDB.sync(remoteDB, {live: true})
.on('change', response => {
console.log(response);
})
.on('error', error => {
console.error('Sync failure', error);
});
To create a user I need a backend layer to handle user registration and creation. THis is now problem sind i can store the admin credentials "secure" on the server. Now my problem is how to handle the pouchdb user credentials in the browser. It would be possible to ask for the credentials on the first loading of the app and then storing it in the ram but this would lead to logging in on each reload of the page. Is there any way to use like cookie/token auth?
Or is there a way to proxy the pouchdb connection through a backend proxy that handles the authentication!?
Any tips would be appreciated.

The simplest way to use cookie authentication is by using the pouchdb-authentication plugin on GitHub. If you do not want to use this then you should read this previous question (#23986912) on Stackoverflow which discusses this in more detail.
Hope this helps.

Meteor.call not working

I had this code working before I wanted to change the client side collection find/insert methods to server side. I removed insecure and autopublish from my meteor project, and changed my code to what it is below.
My angular Code in client/controllers/item-controller.js
angular.module('prototype').controller('ItemController', ['Config','$window','$meteor', function(Config, $window, $meteor) {
this.items = function(){
Meteor.call('getAllItems', function(err, res){
alert("error: " +err + " res: " + res );
return res;
});
}
My item-collection codee in server/item-collection-methods.js
Meteor.methods({
getAllItems : function(){
console.log("i got here")
return Items.find();
}
});
My main file in lib/app.js
Items = new Mongo.Collection("Items");
Before I had 15 items showing, now none of them show.
when I copy my Meteor.call function into the chrome console, all I get back is undefined.
I have a feeling it either has to do with the project structure, or the fact that autopublish and insecure are removed. Any advice would be helpful.
EDIT:
I did get something in my server console
I20150629-00:54:54.402(-4)? Internal exception while processing message { msg: '
method', method: 'getAllItems', params: [], id: '2' } Maximum call stack si
ze exceeded undefined

Meteor data transmission works with a publish/subscribe system. This system is able to replicate part of or all the data that is stored in your MongoDB (server) to the client in an in-memory DB (MiniMongo). Autopublish was publishing everything on the client, as you removed it there is nothing in your Items collection anymore.
In order to publish some data to the client you have to declare a publication on the server side:
Meteor.publish('allItems', function () {
//collection to publish
return Items.find({});
});
And subscribe on the client (either in the router or in a template):
Meteor.subscribe('allItems');
To learn more about this system you can read the official docs.
Concerning your method "getAllItems", you cannot directly send a cursor (Items.find()) on your data, that is why you are getting the error message "Maximum call stack size exceeded".
But you could send an array of these data by returning Items.find().fetch(). Also the call to a Meteor method is asynchronous, so you have to use the callback (more on Meteor methods)
Please note that by sending data over a method (which is perfectly acceptable) you lose the reactivity offered by the publish/subscribe system.

adal javascript windows store app token

I have a javascript windows store app that is authenticathing with AAD via Microsoft.Preview.WindowsAzure.ActiveDirectory.Authentication library. It works against an ASP.NET WebAPI hosted in Azure. It works fine, it prompst the user to login (windows login service), once logged the user can work and is not asked to log in again. The problem I have found is that when the user does not use the app for a time, I´m not sure but I Think is between 20 and 30 minutes, the app gets iddle. It does not respond to user querys. I think comunication with webApi is lost, I'm not sure if the token acquired by the app is expired or Azure itself cuts the connection.
This is how I'm getting the token
var authcontext = new aal.AuthenticationContext(audience);
aal.DefaultTokenCache().clear();
return authcontext.acquireTokenAsync(resource, clientID, redirectUri.absoluteUri, "", null).then(function (result) {
if (aal.AuthenticationStatus.succeeded == result.status) {
accessToken = result.accessToken;
return true;
}
}
and the webApi call
var uri = new Windows.Foundation.Uri(apiUrl + apiName);
var httpClient = new Windows.Web.Http.HttpClient();
httpClient.defaultRequestHeaders.authorization =
new Windows.Web.Http.Headers.HttpCredentialsHeaderValue("Bearer", accessToken);
return httpClient.getAsync(uri).then(function (response) {
if (response.isSuccessStatusCode == true)
return response.content.readAsStringAsync().then(function (responseText) {
var array = JSON.parse(responseText);
return array;
});
else
{
var md = new Windows.UI.Popups.MessageDialog(response, "Error");
md.showAsync();
}
});
I`ve tried adding
<sessionTokenRequirement lifetime="96:00:00" /> and <cookieHandler persistentSessionLifetime="60.0:0:0" requireSsl="true" /> in the web.config but still getting the error.
I am trying unsuccesfully adding ADAL new releases to the project because I read something about refresh tokens, I'm working on it but don´t find any example for windows store apps.
Any clue about the cause of the problem and how to solve it??
Thanks for your time.

The library version you are using is ancient and not supported. Also, cookies play no part in the api communications.
I recommend switching to the latest stable ADAL. Windows Store sample using ADAL: https://github.com/AzureADSamples/NativeClient-WindowsStore