Develop Reference

Puppeteer Web Scraper runs on local machine only but fails to deploy as Google Cloud Function

I have built a simple scraper with Puppeteer which I can run locally on my machine, but when I deploy it as a Google Cloud function, it's not working. The only error message I get from the Google Cloud Logs is:
Function failed on loading user code. This is likely due to a bug in
the user code.
Here are the steps I follow to deploy it the function; code is further below.
(Note: I'm outlining the process of zipping the files; I have tried the Cloud Function inline editor as well but am receiving the same error.)
Run npm install
Run zip -r my-app.zip *
Create new Google Cloud function
-- Name 'newFunction'
-- Memory: 1gb
-- Runtime: Node.js 14
-- Entry point: scrapeFunction
Upload Zip
index.js
const puppeteer = require('puppeteer');
const { BigQuery } = require('#google-cloud/bigquery');
async function scrapeFunction() {
const browser = await puppeteer.launch({ args: ['--no-sandbox', '--disable-setuid-sandbox'] });
const page = await browser.newPage();
await page.goto('<URL>', {waitUntil: 'load', timeout: 0});
await page.waitForSelector('span.text');
const info = await page.evaluate(() => {
return document.querySelector('span.text').innerText;
});
console.log(info);
// Write results to BigQuery table
const bigqueryClient = new BigQuery();
const dataset = bigqueryClient.dataset('xxx');
const table = dataset.table('yyy');
const rows = [{ info: info }];
await table.insert(rows);
await browser.close();
}
scrapeFunction();
package.json
{
"name": "newFunction",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC",
"dependencies": {
"#google-cloud/bigquery": "^6.1.0",
"puppeteer": "^19.7.1"
}
}

ReferenceError: exports is not defined in ES module scope

I am new in typescript, when I compiled code it compiled properly but when I run program using node
I got this error ReferenceError: exports is not defined in ES module scope
ReferenceError: exports is not defined in ES module scope
This file is being treated as an ES module because it has a '.js' file extension and 'D:\projects\pro8\package.json' contains "type": "module". To treat it as a CommonJS script, rename it to use the '.cjs' file extension.
at file:///D:/projects/pro8/await.js:40:1
at ModuleJob.run (node:internal/modules/esm/module_job:185:25)
at async Promise.all (index 0)
at async ESMLoader.import (node:internal/modules/esm/loader:281:24)
at async loadESM (node:internal/process/esm_loader:88:5)
at async handleMainPromise (node:internal/modules/run_main:65:12)
typescript code
import fetch from "node-fetch";
const baseApi = "https://reqres.in/api/users?page=1";
const userApi = "https://reqres.in/api/user";
interface Employee {
id: number
employee_name: string
employee_salary: number
employee_age: number
profile_image: string
}
const fetchAllEmployees = async (url: string): Promise<Employee[]> => {
const response = await fetch(url);
const { data }:any = await response.json();
return data;
};
const fetchEmployee = async (url: string, id: number): Promise<Record<string, string>> => {
const response = await fetch(`${url}/${id}`);
const { data }:any = await response.json();
return data;
};
const generateEmail = (name: string): string => {
return `${name.split(" ").join(".")}#company.com`;
};
const runAsyncFunctions = async () => {
try {
const employees = await fetchAllEmployees(baseApi);
Promise.all(
employees.map(async user => {
const userName = await fetchEmployee(userApi, user.id);
const emails = generateEmail(userName.name);
return emails;
})
);
} catch (error) {
console.log(error);
}
};
runAsyncFunctions();
package.json
{
"name": "pro8",
"version": "1.0.0",
"description": "",
"main": "index.js",
"type": "module",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC",
"devDependencies": {
"#typescript-eslint/eslint-plugin": "^5.21.0",
"#typescript-eslint/parser": "^5.21.0",
"eslint": "^8.14.0",
"eslint-config-standard": "^17.0.0",
"eslint-plugin-import": "^2.26.0",
"eslint-plugin-n": "^15.2.0",
"eslint-plugin-promise": "^6.0.0"
},
"dependencies": {
"#types/node-fetch": "^2.6.1",
"node-fetch": "^3.2.4"
}
}
I tried to solve this error ts An async function or method in ES5/ES3 requires the 'Promise' constructor
any answer not working in my case
so please give me solution if any

Try and remove "type": "module" from package.json. I had this issue before and it seemed to do the trick.

Returning reason for node-fetch query results not matching rather than giving undefined. nodejs node-fetch

Note the API key has had the last characters removed as I was told not to post api keys linked to my personal account.
I am working on getting weather results from openweathermap.org Everything seems to be working except if I input an invalid city name or area code it returns undefined. I would like to return a reason for the failure in the console.
below are the dependencies being used:
"name": "node-final-exercise",
"version": "1.0.0",
"description": "",
"main": "app.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC",
"dependencies": {
"node-fetch": "^2.6.1"
}
}
heres the code snippet I'm working on right now.
// Goals for project
// API's OpenWeatherMap api.openweathermap.org/data/2.5/weather?q=London,uk&APPID=8d0f96c686dbab4b20d5dda13
// API's zip code specific api.openweathermap.org/data/2.5/weather?zip={zip code},{country code}&appid={API key}
// a. use node app.js (areacode) to connect to weather api should also accept ( city ) and (state) like cleveland ohio
// b. parse data from weather api
// c. ouput JSON parsed data into a readable format in the console.
const fetch = require("node-fetch");
const city = process.argv.slice(2);
(async () => {
try {
await fetch(
`http://api.openweathermap.org/data/2.5/weather?q=${city}&units=metric&appid=8d0f96c686dbab4b20d5dda13`
)
.then((res) => res.json())
.then((body) => console.log(body.main))
.catch(error =>{console.error("first catch error : ", error)})
} catch (error) {
console.error("error : ",error.message);
}
})();
console results

Unit testing of HttpsCallable cloud functions - online mode

I am designing a backend api (for android and iOS apps) around HttpsCallable cloud functions. It becomes quite cumbersome to test them through the app, so I wish to switch to unit testing the functions (before production deployment) using the firebase-functions-test tool. I have been following this unit testing tutorial.
I am having some issues running the unit tests in online mode. Let me provide some details.
Here is my `package.json' content:
{
"name": "functions",
"description": "Cloud Functions for Firebase",
"scripts": {
"lint": "eslint .",
"serve": "firebase serve --only functions",
"shell": "firebase functions:shell",
"start": "npm run shell",
"deploy": "firebase deploy --only functions",
"logs": "firebase functions:log",
"test": "mocha --reporter spec"
},
"engines": {
"node": "8"
},
"dependencies": {
"#google-cloud/tasks": "^1.9.0",
"#googlemaps/google-maps-services-js": "^2.0.2",
"chai": "^4.2.0",
"firebase-admin": "^8.6.0",
"firebase-functions": "^3.3.0",
},
"devDependencies": {
"eslint": "^5.12.0",
"eslint-plugin-promise": "^4.0.1",
"firebase-functions-test": "^0.1.7",
"mocha": "^7.1.1"
},
"private": true
}
I am using google APIs (Directions, Geocoding etc) from firebase backend, therefore to be able to access them while running my tests, I configured my tests located at test/index.test.js as recommended in the Unit testing tutorial as follows:
const firebaseConfig = {
apiKey: ...,
authDomain: ...,
databaseURL: "https://my-project.firebaseio.com",
projectId: "my-project",
storageBucket: "my-project.appspot.com",
messagingSenderId: ********,
appId: *********
};
const test = require('firebase-functions-test')(firebaseConfig
, 'path_to_json_keyfile/myproject.json');
Below is the sample test code. Note that all my callable functions return HttpsError, and the test below, simply checks for the code field of HttpsError, if its ok, the test passes. The functions I am testing reside in a separate rides.js file, which the index.js exports as exports.rides = require(./rides.js) (not shown below)
const chai = require('chai');
const assert = chai.assert;
describe('Cloud functions', () => {
let rideFunctions;
before(() => {
rideFunctions = require('../index.js');
});
after(() => {
test.cleanup();
});
describe('getRideEstimate', () => {
it('should return ok', async () => {
data = {
riderType: 1,
pickup_lat: 37.0,
pickup_lng: -122,
dropoff_lat: 37.01,
dropoff_lng: -122.01,
scheduledTime: 1585939000,
ts: 1585939000,
tz_id: "uslax"
}
context = {
auth: {
uid: AUTH_ID_FOR_USER_ACCOUNT
}
};
const wrappedGetRideEstimate = test.wrap(rideFunctions.rides.getRideEstimate);
let httpsError = await wrappedGetRideEstimate(data, context);
return assert.equal(httpsError.code, 'ok');
});
})
describe('testCallable', () => {
it('should return ok', async () => {
data = {}
context = {}
const wrappedTestCallable = test.wrap(rideFunctions.rides.testCallable);
let httpsError = await wrappedTestCallable(data, context);
return assert.equal(httpsError.code, 'ok');
})
})
})
Problem
My simple testCallable function of the form
exports.testCallable = functions.https.onCall((data, context) => {
console.log('testCallable');
return new functions.https.HttpsError('ok', '', '');
})
passes the test (as expected) but inexplicably, it seems it is running in the offline mode, as there is no record of it on cloud function logs in Firebase Console. Also, if I disable connectivity of my laptop, the test result remains the same suggesting that somehow this function is running in the offline mode.
My getRideEstimate function which calls the google Directions API, returns a lengthy 400 error indicating some issue with forming the request to Directions API. I don't know if this error is related to the first problem, but since the Directions API call is embedded deeper inside the getRideEstimate function, it does suggest that the function is getting called, but somehow the API call is failing.
Is there any other way to test HttpsCallable functions in online mode?

For me this works:
import * as firebaseSetup from 'firebase-functions-test';
export const testEnvironment = firebaseSetup({
databaseURL: "https://project-id.firebaseio.com",
projectId: 'project-id'
}, './service-account-project-id-firebase-adminsdk.json');
You can find a full YouTube tutorial under this: https://youtu.be/UDMDpdu5-rE?t=1122

(node .) command: "throw new errors.ERR_INVALID_ARG_TYPE('listener', 'Function', listener);"

I've been trying to make my bot go live on Discord, but I got this error and don't know how to fix it:
Also, here is the "package.json" file:
{
"name": "botName",
"version": "1.0.0",
"main": "raidBot.js",
"scripts": {
"text": "echo \"Error: no test specified\" && exit 1"
},
"author": "uwu || dm 2 partner#7245 & lottie#8696",
"license": "ISC",
"dependencies": {
"discord.js": "^11.4.2"
},
"description": ""
}
"botName.js" file:
const discord = require('discord.js');
var client = new discord.Client();
const token = "instertedTokenHere";
client.on ("ready", () => {
console.log("the bot is ready...");
client.user.setGame ("prefix is $");
});
const prefix = "$";
client.on ("message"), (message) => {
if (message.author.bot) return;
if (message.content.startsWith (prefix + "bot")) {
message.reply ('Running...');
}
};
NOTE: This is what I did that resulted in the error:
cd D:\bot004
npm install
npm install --save discord.js
node .
Everything seems fine though. I have the node_modules folder and package.json, package-lock.json, and botName.js files.

// Your code
const discord = require('discord.js');
var client = new discord.Client();
const token = "instertedTokenHere";
client.on ("ready", () => {
console.log("the bot is ready...");
client.user.setGame ("prefix is $");
});
const prefix = "$";
client.on ("message"), (message) => {
if (message.author.bot) return;
if (message.content.startsWith (prefix + "bot")) {
message.reply ('Running...');
}
};
This looks a little suspect. You are not passing in the callback function to the .on() method below. That's where your error is coming from.
client.on ("message"), (message) => {
if (message.author.bot) return;
if (message.content.startsWith (prefix + "bot")) {
message.reply ('Running...');
}
};
Try refactoring to this:
client.on ("message", (message) => {
if (message.author.bot) return;
if (message.content.startsWith (prefix + "bot")) {
message.reply ('Running...');
}
});
I am able to reproduce the error when I run your code as is. The error subsides when I fix the code.

Place your client variable under the token. I had the same error when I made my discord bot.