I'm learning about puppeteer and firebase at the moment. What I am trying to do is create a pdf of a web page and upload to firebase storage. This is my code.
const puppeteer = require('puppeteer');
const fs = require('fs').promises;
const firebase = require('firebase');
require("firebase/storage");
const url = process.argv[2];
if (!url) {
throw "Please provide URL as a first argument";
}
var firebaseConfig = {
#Firebase Config Goes here
};
// Initialize Firebase
firebase.initializeApp(firebaseConfig);
#Function to generate PDF file
async function run () {
const browser = await puppeteer.launch();
const page = await browser.newPage();
//await page.goto(url);
await page.goto(url, {waitUntil: 'domcontentloaded', timeout: 60000} );
//await page.pdf({ path: 'api.pdf', format: 'A4' })
const myPdf = await page.pdf();
await browser.close()
return myPdf;
}
const myOutput = run();
#Upload to Firebase based on the instruction here https://firebase.google.com/docs/storage/web/upload-files
var storageRef = firebase.storage().ref();
// Create a reference to 'mountains.jpg'
storageRef.child("Name.pdf").put(myOutput)
However, I'm running into this error when executing my code
$ node screenshot.js https://google.com
Promise { <pending> }
(node:20636) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'byteLength' of undefined
at C:\Users\ppham\NucampFolder\Test\node_modules\#firebase\storage\dist\index.cjs.js:833:40
at Array.forEach (<anonymous>)
at Function.FbsBlob.getBlob (C:\Users\ppham\NucampFolder\Test\node_modules\#firebase\storage\dist\index.cjs.js:832:25)
at multipartUpload (C:\Users\ppham\NucampFolder\Test\node_modules\#firebase\storage\dist\index.cjs.js:1519:24)
at C:\Users\ppham\NucampFolder\Test\node_modules\#firebase\storage\dist\index.cjs.js:2003:31
at C:\Users\ppham\NucampFolder\Test\node_modules\#firebase\storage\dist\index.cjs.js:1900:21
at processTicksAndRejections (internal/process/task_queues.js:85:5)
(node:20636) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:20636) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
This looks to imply that myOutput doesn't contain anything. I thought that I have created the pdf file after executing the run() function, assigned it to the myOutput variable and passed it to the upload function? I've been reading the Puppeteer documentation and couldn't find any reason why this wouldn't work. Anyone knows why this is not valid?
The difference between your code and the example code at https://www.toptal.com/puppeteer/headless-browser-puppeteer-tutorial, for example, is trivial, but there's one significant difference:
run is defined as an async function -- in the example code, this may be inconsequential, as the last line of the example just calls run() and nothing happens afterwards. But, in your code, you expect to do something with that output. So, you'll need to call it with:
const myOutput = await run();
But, Node won't want you to use await at the top level -- await can only be used inside an async function. So, you can either use then() or just define another async wrapper. So, probably something like this:
async function runAndUpload() {
const myOutput = await run();
console.log(myOutput); //let's make sure there's output
var storageRef = firebase.storage().ref();
storageRef.child("Name.pdf").put(myOutput)
}
runAndUpload();
Context and info:
I recently made a simple script that logs into a exterior website and gets some data. The purpose of this script was to get a students grades and then turn it into plottable data. To make the process of getting the data much easier I used the npm library: selenium-webdriver. The reason I used that library instead of request (for example) is because I need to login, and so that I don't get cross origin error (and yes I would get a cross origin error because my server is already connected to a front end app). Note that all of my code is inside an async function and that it is being called like a promise would (meaning that it is being called with a .then and not with a await inside of an async function.).
The issue:
The script works perfectly giving me the exact results I want, but I am still getting an error. This error is confusing to me because of twofold: first because all of my code is in a try block with a catch attached as well as having a .catch() where it is called; and second the error is logged after the function has resolved.
The error message:
(node:17908) UnhandledPromiseRejectionWarning: NoSuchSessionError: invalid session id
(Driver info: chromedriver=73.0.3683.68
(47787ec04b6e38e22703e856e101e840b65afe72),platform=Windows NT 10.0.17134 x86_64)
at Object.checkLegacyResponse (C:\Users\Redacted\Desktop\Application\Code\node_modules\selenium-webdriver\lib\error.js:585:15)
at parseHttpResponse (C:\Users\Redacted\Desktop\Application\Code\node_modules\selenium-webdriver\lib\http.js:533:13)
at Executor.execute (C:\Users\Redacted\Desktop\Application\Code\node_modules\selenium-webdriver\lib\http.js:468:26)
at processTicksAndRejections (internal/process/next_tick.js:81:5)
(node:17908) UnhandledPromiseRejectionWarning: Unhandled promise rejection.
This error originated either by throwing inside of an async function without a catch block,
or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:17908) [DEP0018] DeprecationWarning: Unhandled promise rejections are
deprecated. In the future, promise rejections that are not handled will
terminate the Node.js process with a non-zero exit code.
My code:
'use strict'
const {Builder, By, Key, until, Capabilities} = require('selenium-webdriver')
const Chrome = require('selenium-webdriver/chrome')
exports.simpleGradebookGetGrades = async function(username, password) {
const driver = new Builder().forBrowser('chrome').withCapabilities(Capabilities.chrome()).setChromeOptions(new Chrome.Options().addArguments('--remote-debugging-port=25470')).build()
try {
let retval = []
await driver.get('https://simplegradebook.ca/gradebook/login.php')
async function login(username, password) {
await driver.findElement(By.name('userid')).sendKeys(username)
await driver.findElement(By.name('password')).sendKeys(password)
await driver.findElement(By.name('login')).click()
await driver.wait(until.titleMatches(/.{20,}/))
return
}
await login(username, password)
for(let i of Object.keys(await driver.findElements(By.name('viewclasses')))) {
await driver.wait(until.elementsLocated(By.name('viewclasses')));
(await driver.findElements(By.name('viewclasses')))[i].click()
await driver.wait(until.elementLocated(By.tagName('tbody')))
retval.push(await driver.findElement(By.tagName('tbody')).getText())
await driver.get('https://simplegradebook.ca/gradebook/login.php')
login(username, password)
}
return retval
} catch(err) {
console.log(err)
} finally {
await driver.quit()
}
}
exports.simpleGradebookGetGrades('Redacted', 'Redacted').then(result => {
console.log(result)
}).catch(err => {
console.log(err)
})
My question:
Why is the error occuring? How can I remove or Ignore this error? Why is my catch block not catching this error?
Additional info:
Node version: 11.8.0
Selenium-webdriver version: 4.0.0-alpha.1
Do any of the errors go away if you define your driver with the following option for Chrome?
const chrome = require('selenium-webdriver/chrome')
const webdriver = require('selenium-webdriver')
let options = new chrome.Options()
let nextPort = 9222 //for example
options.addArguments(["--remote-debugging-port=" + nextPort])
let driver = new webdriver.Builder()
.withCapabilities(webdriver.Capabilities.chrome())
.setChromeOptions(options)
.build()
I am trying to write tests for electron using spectron.
This is my code.
describe ('Application launch', function(done) {
this.timeout(30000);
const app = new Application({
path: electronBinary,
args: [baseDir],
});
before(() => app.start());
after(() => app.stop());
it('shows an initial window', async () => {
await app.client.waitUntilWindowLoaded();
const count = await app.client.getwindowcount();
assert.equal(count,1);
});
});
However, When I run npm test The error I get is
1) Application launch "before all" hook:
Error: Timeout of 30000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
2) Application launch "after all" hook:
Error: Application not running
at Application.stop (node_modules\spectron\lib\application.js:58:48)
at Context.after (test\spec.js:19:19)
Do I need to add any functions to the existing hooks?
It appears to be happening in your before all method. Notice that your error says 1) Application launch "before all" hook:.
So your actual test function looks fine to me. And I don't see a beforeAll anywhere in this example code so I would say there is one of two possible problems.
There is a beforeAll method with an issue somewhere not in this code sample.
The before hook shown here is returning a non-promise object.
In your code you are using a lambda function to do the work of before but if app.start() is returning an object which is not a promise then that would be your issue. Try refactoring it more like this:
before(() => {
app.start()
})
If your app.start() function is asynchronous you may need to pass the done handler into it:
before((done) => {
app.start(done)
})
Or possibly convert your app.start() function to return a promise which should resolve it. You may need to add async () => await app.start() but I don't think that would be necessary for a single expression like this.
You didn't use "done" as a callback in your it-function. You also don't have to use done in the describe callback. Also, since done() already makes your code asynchronous, you don't have to use the async keyword.
My solution:
describe ('Application launch', function() {
this.timeout(30000);
const app = new Application({
path: electronBinary,
args: [baseDir],
});
before(() => app.start());
after(() => app.stop());
it('shows an initial window', (done) => {
await app.client.waitUntilWindowLoaded();
const count = app.client.getwindowcount();
assert.equal(count,1);
done();
});
});
Hope it helps!
I have been going over async/await and after going over several articles, I decided to test things myself. However, I can't seem to wrap my head around why this does not work:
async function main() {
var value = await Promise.resolve('Hey there');
console.log('inside: ' + value);
return value;
}
var text = main();
console.log('outside: ' + text);
The console outputs the following (node v8.6.0) :
> outside: [object Promise]
> inside: Hey there
Why does the log message inside the function execute afterwards? I thought the reason async/await was created was in order to perform synchronous execution using asynchronous tasks.
Is there a way could I use the value returned inside the function without using a .then() after main()?
I can't seem to wrap my head around why this does not work.
Because main returns a promise; all async functions do.
At the top level, you must either:
Use top-level await (proposal, MDN; ES2022, broadly supported in modern environments) that allows top-level use of await in a module.
or
Use a top-level async function that never rejects (unless you want "unhandled rejection" errors).
or
Use then and catch.
#1 top-level await in a module
You can use await at the top-level of a module. Your module won't finish loading until the promise you await settles (meaning any module waiting for your module to load won't finish loading until the promise settles). If the promise is rejected, your module will fail to load. Typically, top-level await is used in situations where your module won't be able to do its work until the promise is settled and won't be able to do it at all unless the promise is fulfilled, so that's fine:
const text = await main();
console.log(text);
If your module can continue to work even if the promise is rejected, you could wrap the top-level await in a try/catch:
// In a module, once the top-level `await` proposal lands
try {
const text = await main();
console.log(text);
} catch (e) {
// Deal with the fact the chain failed
}
// `text` is not available here
when a module using top-level await is evaluated, it returns a promise to the module loader (like an async function does), which waits until that promise is settled before evaluating the bodies of any modules that depend on it.
You can't use await at the top level of a non-module script, only in modules.
#2 - Top-level async function that never rejects
(async () => {
try {
const text = await main();
console.log(text);
} catch (e) {
// Deal with the fact the chain failed
}
// `text` is not available here
})();
// `text` is not available here, either, and code here is reached before the promise settles
// and before the code after `await` in the main function above runs
Notice the catch; you must handle promise rejections / async exceptions, since nothing else is going to; you have no caller to pass them on to (unlike with #1 above, where your "caller" is the module loader). If you prefer, you could do that on the result of calling it via the catch function (rather than try/catch syntax):
(async () => {
const text = await main();
console.log(text);
})().catch(e => {
// Deal with the fact the chain failed
});
// `text` is not available here, and code here is reached before the promise settles
// and before the code after `await` in the main function above runs
...which is a bit more concise, though it somewhat mixes models (async/await and explicit promise callbacks), which I'd normally otherwise advise not to.
Or, of course, don't handle errors and just allow the "unhandled rejection" error.
#3 - then and catch
main()
.then(text => {
console.log(text);
})
.catch(err => {
// Deal with the fact the chain failed
});
// `text` is not available here, and code here is reached before the promise settles
// and the handlers above run
The catch handler will be called if errors occur in the chain or in your then handler. (Be sure your catch handler doesn't throw errors, as nothing is registered to handle them.)
Or both arguments to then:
main().then(
text => {
console.log(text);
},
err => {
// Deal with the fact the chain failed
}
);
// `text` is not available here, and code here is reached before the promise settles
// and the handlers above run
Again notice we're registering a rejection handler. But in this form, be sure that neither of your then callbacks throws any errors, since nothing is registered to handle them.
2021 answer: you can now use top level await in the current stable version of node
Most of the answers above are a little out of date or very verbose, so here's a quick example for node 14 onwards.
Make a file called runme.mjs:
import * as util from "util";
import { exec as lameExec } from "child_process";
const exec = util.promisify(lameExec);
const log = console.log.bind(console);
// Top level await works now
const { stdout, stderr } = await exec("ls -la");
log("Output:\n", stdout);
log("\n\nErrors:\n", stderr);
Run node runme.mjs
Output:
total 20
drwxr-xr-x 2 mike mike 4096 Aug 12 12:05 .
drwxr-xr-x 30 mike mike 4096 Aug 12 11:05 ..
-rw-r--r-- 1 mike mike 130 Aug 12 12:01 file.json
-rw-r--r-- 1 mike mike 770 Aug 12 12:12 runme.mjs
Errors:
Top-Level await has moved to stage 3 stage 4 (see namo's comment), so the answer to your question How can I use async/await at the top level? is to just use await:
const text = await Promise.resolve('Hey there');
console.log('outside: ' + text)
Of if you want a main() function: add await to the call to main() :
async function main() {
var value = await Promise.resolve('Hey there');
console.log('inside: ' + value);
return value;
}
var text = await main();
console.log('outside: ' + text)
Compatibility
v8 since Oct 2019
the REPL in Chrome DevTools, Node.js and Safari web inspector
Node v13.3+ behind the flag --harmony-top-level-await
TypeScript 3.8+ (issue)
Deno since Oct 2019
Webpack#v5.0.0-alpha.15
To give some further info on top of current answers:
The contents of a node.js file are currently concatenated, in a string-like way, to form a function body.
For example if you have a file test.js:
// Amazing test file!
console.log('Test!');
Then node.js will secretly concatenate a function that looks like:
function(require, __dirname, ... perhaps more top-level properties) {
// Amazing test file!
console.log('Test!');
}
The major thing to note, is that the resulting function is NOT an async function. So you cannot use the term await directly inside of it!
But say you need to work with promises in this file, then there are two possible methods:
Don't use await directly inside the function
Don't use await at all
Option 1 requires us to create a new scope (and this scope can be async, because we have control over it):
// Amazing test file!
// Create a new async function (a new scope) and immediately call it!
(async () => {
await new Promise(...);
console.log('Test!');
})();
Option 2 requires us to use the object-oriented promise API (the less pretty but equally functional paradigm of working with promises)
// Amazing test file!
// Create some sort of promise...
let myPromise = new Promise(...);
// Now use the object-oriented API
myPromise.then(() => console.log('Test!'));
It would be interesting to see node add support for top-level await!
You can now use top level await in Node v13.3.0
import axios from "axios";
const { data } = await axios.get("https://api.namefake.com/");
console.log(data);
run it with --harmony-top-level-await flag
node --harmony-top-level-await index.js
The actual solution to this problem is to approach it differently.
Probably your goal is some sort of initialization which typically happens at the top level of an application.
The solution is to ensure that there is only ever one single JavaScript statement at the top level of your application. If you have only one statement at the top of your application, then you are free to use async/await at every other point everwhere (subject of course to normal syntax rules)
Put another way, wrap your entire top level in a function so that it is no longer the top level and that solves the question of how to run async/await at the top level of an application - you don't.
This is what the top level of your application should look like:
import {application} from './server'
application();
i like this clever syntax to do async work from an entrypoint
void async function main() {
await doSomeWork()
await doMoreWork()
}()
Other solutions were lacking some important details for POSIX compliance:
You need to ...
Report a 0 exit status on success and non-zero on fail.
Emit errors to stderr output stream.
#!/usr/bin/env node
async function main() {
// ... await stuff ...
}
// POSIX compliant apps should report an exit status
main()
.then(() => {
process.exit(0);
})
.catch(err => {
console.error(err); // Writes to stderr
process.exit(1);
});
If you're using a command line parser like commander, you may not need a main().
Example:
#!/usr/bin/env node
import commander from 'commander'
const program = new commander.Command();
program
.version("0.0.1")
.command("some-cmd")
.arguments("<my-arg1>")
.action(async (arg1: string) => {
// run some async action
});
program.parseAsync(process.argv)
.then(() => {
process.exit(0)
})
.catch(err => {
console.error(err.message || err);
if (err.stack) console.error(err.stack);
process.exit(1);
});
Node -
You can run node --experimental-repl-await while in the REPL. I'm not so sure about scripting.
Deno -
Deno already has it built in.
For Browser you need to add type="module"
without type="module"
<script>
const resp = await fetch('https://jsonplaceholder.typicode.com/users');
const users = await resp.json();
console.log(users)
</script>
with type="module"
<!--script type="module" src="await.js" -->
<script type="module">
const resp = await fetch('https://jsonplaceholder.typicode.com/users');
const users = await resp.json();
console.log(users)
</script>
You need to add type in package.json
"type": "module"
You are good to go.
import axios from 'axios';
const res = await axios.get('https://api.github.com/users/wesbos');
console.log(res.data);
Remember if you change type of document then you must have to write code in ES6 way.
Now with ECMAScript22, we can use await at the top-level module.
This is an example with ( await top-level ):
const response = await fetch("...");
console.log(response):
an other example without (await top-level )
async function callApi() {
const response = await fetch("...");
console.log(response)
}
callApi()
In NodeJS 14.8+, you can use top-level await module (#3 solution). You can rename also .js to .mjs (ES module) instead of .js (.cjs CommonJS).
If your only goal is to control the execution order of asynchronous code mixed with other code for testing purposes, you could wrap the entire top-level code inside of an immediately-invoked function expression (IIFE) defined as an async function. In the example from the question, you would then add await before calling main().
You can use this pattern when your code is not already in an async function or at the top level body of a module. In other words, if you're just testing a bunch of code inside of a js file and using tools like Live Server, RunJs, or any other type of JavaScript playground to watch the console window, wrap all of your code in an IIFE defined as async and use the await keyword when you want to wait for asynchronous code to finish before executing the next line.
let topLevelIIFE = (async () => {
async function main() {
var value = await Promise.resolve('Hey there');
console.log('inside: ' + value);
return value;
}
var text = await main();
console.log('outside: ' + text);
})()
You would not need to use this pattern when running the code specified in the body of the IIFE inside of the REPL in Chrome DevTools or another browser REPL tool that behaves similarly.
Since main() runs asynchronously it returns a promise. You have to get the result in then() method. And because then() returns promise too, you have to call process.exit() to end the program.
main()
.then(
(text) => { console.log('outside: ' + text) },
(err) => { console.log(err) }
)
.then(() => { process.exit() } )
I am trying to test some functionality. Inside the routes.js file, i placed this code:
async function getPic(arg) {
const browser = await puppeteer.launch(/*{headless: false}*/);
const page = await browser.newPage();
await page.goto(arg);
await page.setViewport({width: 1000, height: 500})
await page.screenshot({path: 'pic.png'});
await broswer.close();
}
I read about async/await (it is the first time i use it). Though i get an error message and the code does not work:
(node:5896) UnhandledPromiseRejectionWarning: Error: Navigation Timeout Exceeded
: 30000ms exceeded
at Promise.then (C:\...\node_modules\puppeteer\lib\NavigatorWatcher.js:73:21
)
at <anonymous>
(node:5896) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This
error originated either by throwing inside of an async function without a catch
block, or by rejecting a promise which was not handled with .catch(). (rejection
id: 1)
(node:5896) [DEP0018] DeprecationWarning: Unhandled promise rejections are depre
cated. In the future, promise rejections that are not handled will terminate the
Node.js process with a non-zero exit code.
Unfortunately i do not know what this means. I found out a similar question on stackoverflow here:
NodeJS Unhandled Promise Rejection
But unfortunately does not make things clear for me on how to solve this error.
When i tested this snippet of code in a standalone node environment - meaning just this code without anything else, as i do in my project, then somehow it works.
When i place this function inside my routes.js file, and then invoke the function when a post event happens then i get the error.
Here is the code that invokes this function:
app.post('/sc', function(req, res){
var url = req.body.convo
console.log(url)
getPic(url);
})
You should use await before call async function and wrapp await call into try catch construction.
app.post('/sc', async (req, res) => {
const url = req.body.convo
try {
var picture = await getPic(url);
//some logic or render response
} catch (error){
//here you should handle error
}
})
You need to await getPic(url);.
Also you have browser misspelled in await broswer.close();