I am trying to use page.evaluate in Pyppeteer and capture js script response but I am unable to capture it. In the following code, I am trying to capture the result returned by js script in dimensions variable, but its capturing as None
import asyncio
from pyppeteer import launch
async def hmm():
browser = await launch(headless=False)
page = await browser.newPage()
await page.goto('https://jobs.chegg.com')
dimensions = await page.evaluate("""async () => {
var script = document.createElement('script');
script.type = 'text/javascript';
script.src = 'https://cdnjs.cloudflare.com/ajax/libs/axe-core/3.1.2/axe.min.js';
document.head.appendChild(script);
var result = new Promise((resolve, reject) => {
axe.run(document, {
runOnly: {
type: "tag",
values: ["wcag2a", "wcag2aa", "best-practice"]
},
"rules": {
"skip-link": { enabled: false }
}
}, function(err, results) {
if (err) throw err;
resolve(results);
});
});
let test = await result.then((res) => {
return res;
}).catch(err => {
console.log(err);
});
console.log(test);
return test;
}
""")
print(dimensions) # None
return dimensions
asyncio.get_event_loop().run_until_complete(hmm())
Note :- Open console in any website and run the js script, then, an object/dictionary is returned.
Please suggest a workaround for this problem.
Not sure what you are trying to accomplish but if you want to capture response from the site, you should listen to response. Here is an example with your site. It will print out every response object and related information.
import asyncio
from pyppeteer import launch
def interception_fun(response):
# Response logic goes here
print("URL:", response.url)
print("Method:", response.request.method)
print("Response headers:", response.headers)
print("Request Headers:", response.request.headers)
print("Response status:", response.status)
return
async def hmm():
browser = await launch(headless=False)
page = await browser.newPage()
await page.goto('https://jobs.chegg.com')
page.on('response', interception_fun)
await browser.close()
return
asyncio.get_event_loop().run_until_complete(hmm())
Update:
As of pyppeteer version 0.2.5. page.on() should be lambda function like this:
page.on('response', lambda res: asyncio.ensure_future(interception_fun(res)))
Related
app.prepare().then(() => {
createServer(async (request, result) => {
try {
const parsedUrl = parse(request.url, true);
const { pathname, query } = parsedUrl;
// ...
// here config
const routes = await serverRouterApiWorker.getRoutes();
const clonedReactRoutes = Object.assign({}, reactRoutes.routes);
for (const reactRouteName of Object.keys(clonedReactRoutes)) {
const reactRoute = clonedReactRoutes[reactRouteName];
if (routes[reactRouteName] !== undefined && pathname === routes[reactRouteName].path) {
await app.render(request, result, reactRoute, query); // Here I choose which path to render
delete clonedReactRoutes[reactRouteName];
}
}
// ...
} catch (err) {
console.error('Error occurred handling', request.url, err)
result.statusCode = 500
result.end('internal server error')
}
}).listen(port, (err) => {
if (err) throw err
console.log(`> Ready on http://${hostname}:${port}`)
})
})
I need to get this exact routes config in my next.js application. I need this data to be loaded only when a request is made to the server (reloading the page, opening the page through a browser). If the page opens without reloading the request for new data should not do.
I can't use MyApp.getInitialProps because it makes a request every time the page is opened, not just when the page is reloaded. I need something like MyApp.getServerSideProps but I see that it is not possible https://github.com/vercel/next.js/discussions/13199
I would appreciate it if you could show me an example of how to transfer data from server js to application and how to use them (I need to use them in a component)
Need some guidance here with service worker.
When the service worker is installed, it caches the assets. On next reload, when any request is made, it is intercepted by service worker, which first checks in cache, if it isn't found, then we make a network call. But this second network call is again being intercepted by service worker and thus it has turned into an infinite loop.
I don't want the next fetch call, to be intercepted again. I hope I'm able to explain the issue here.
Here is the serviceWorker.js
const cacheVersion = "v11";
self.addEventListener('install',(event)=>{
self.skipWaiting();
event.waitUntil(caches.open(cacheVersion).then((cache)=>{
cache.addAll([
'/',
'/index.html',
'/style.css',
'/images/github.png',
])
.then(()=>console.log('cached'),(err)=>console.log(err));
}))
})
self.addEventListener('activate',event=>{
event.waitUntil(
(async ()=>{
const keys = await caches.keys();
return keys.map(async (cache)=>{
if(cache !== cacheVersion){
console.log("service worker: Removing old cache: "+cache);
return await caches.delete(cache);
}
})
})()
)
})
const cacheFirst = async (request) => {
try{
const responseFromCache = await caches.match(request);
if (responseFromCache) {
return responseFromCache;
}
}
catch(err){
return fetch(request);
}
return fetch(request);
};
self.addEventListener("fetch", (event) => {
event.respondWith(cacheFirst(event.request));
});
The reason here is your cacheFirst, it's a bit wrong. What do we want to do inside it (high-level algorithm) ? Should be something like this, right?
check cache and if match found - return
otherwise, fetch from server, cache and return
otherwise, if network failed - return some "dummy" response
const cacheFirst = async (request) => {
// First try to get the resource from the cache
const responseFromCache = await caches.match(request);
if (responseFromCache) {
return responseFromCache;
}
// Next try to get the resource from the network
try {
const responseFromNetwork = await fetch(request);
// response may be used only once
// we need to save clone to put one copy in cache
// and serve second one
putInCache(request, responseFromNetwork.clone());
return responseFromNetwork;
} catch (error) {
// well network failed, but we need to return something right ?
return new Response('Network error happened', {
status: 408,
headers: { 'Content-Type': 'text/plain' },
});
}
};
This is not ready-to-use solution !!! Think of it as a pseudo-code, for instance you might need to impl putInCache first.
In my full stack application, I got the app folder and a server and client folder inside.
When going to X URL a useEffect is triggered, and it should fetch data from localhost:4000/pets/cats, but instead I get an error:
http://localhost:3000/localhost:4000/pets/cats 404 (Not Found)
For some reason it's trying to look for this URL: http://localhost:3000/localhost:4000/pets/cats (that doesn't really exist). How to fix it?
Here's the get method
The reducer
The useEffect where I should be getting the info
You need to set a proper base URL for axios:
An example using full URL:
useEffect(() => {
(async () => {
try {
const url = "http://localhost:4000/pets/cats"; // Full URL
await axios.get(url);
} catch (err) {
console.log("Some error: ", err);
}
})();
}, []);
An example using base URL:
useEffect(() => {
(async () => {
try {
const url = "/pets/cats";
await axios.get(url, {
baseURL: "http://localhost:4000", // Base URL
});
} catch (err) {
console.log("Some error: ", err);
}
})();
}, []);
you try to call api twice so api is not found or 404, try this localhost:4000/pets/cats
I made a web app that uses an API.
For hides the key of the API and host it on netlify I've used a lambda function:
exports.handler = async event => {
const apiKey = process.env.apiKey
const response = await fetch(`https://api.waqi.info/feed/${cityName}/?token=${apiKey}`)
const result = await response.json()
const pass = (body) => {
return {
statusCode: 200,
body: JSON.stringify(body)
}
}
return pass(result)
}
that makes the call to the API and share the result of the call to my main script, that one elaborates this response.
async function checkAir() {
let cityName = document.getElementById("cityName").value;
// Call API
const response = await fetch("../netlify/functions/lambda")
const result = await response.json()
console.log("response" + response)
console.log("result" + result)
}
When it runs, doesn't works, and gives the error:
GET 'url/.netlify/functions/lambda' 404
Try using the developer tools in your Browser.
Inspect > Network Tab. and refresh your page to run the script.
Find the http request for lambda, and investigate the request, maybe the path isn't correct.
This used to work for me but stopped a couple of months ago and I've tinkered my way right out of being able to figure this out anymore. What am I doing wrong here?
Call the service worker template, no problem:
if(navigator.serviceWorker){
window.addEventListener('load',() => {
navigator.serviceWorker
.register('/sw.js')
.then(console.log('[ServiceWorker] Registered Successfully'))
.catch(err => console.log(`[ServiceWorker] Error: ${err}`));
});
} else {
console.log('Service Worker not supported.');
}
Setup a cache version and preloaded the cache, no problem:
const cacheName='2020.10.06-01';
var cacheFiles = ['/offline.html'];
Installed the Services Worker, no problem:
addEventListener('install', e => {
e.waitUntil(
caches.open(cacheName).then(cache => {
return cache.addAll(cacheFiles);
})
);
});
Activated the Services Worker for auto cache rollover, no problem:
addEventListener('activate', e => {
e.waitUntil(
caches.keys().then(keyList => {
return Promise.all(keyList.map(key => {
if(key !== cacheName) {
return caches.delete(key);
}
}));
})
);
});
Fetching from cache or network, no problem:
addEventListener('fetch', e => {
e.respondWith(async function() {
try {
const cache = await caches.open(cacheName);
const cachedResponse = await cache.match(e.request);
const networkResponsePromise = fetch(e.request);
e.waitUntil(async function() {
const networkResponse = await networkResponsePromise;
await cache.put(e.request, networkResponse.clone());
}());
// Returned the cached response if we have one, otherwise return the network response.
return cachedResponse || networkResponsePromise;
} catch (error) {
console.log('Fetch failed; returning offline page instead.', error);
const cache = await caches.open(cacheName);
const cachedResponse = await cache.match('/offline.html');
return cachedResponse;
}
}());
});
But if the page/resource I'm trying to request is not already in the cache AND the network is not available it refuses to display my 'offline.html' page. (Which I know IS in the cache)
Any ideas?
Here's the Fetch code I wrote in the end that works perfectly for me:
self.addEventListener('fetch', (event) => {
event.respondWith((async() => {
const cache = await caches.open(cacheName);
try {
const cachedResponse = await cache.match(event.request);
if(cachedResponse) {
console.log('cachedResponse: ', event.request.url);
return cachedResponse;
}
const fetchResponse = await fetch(event.request);
if(fetchResponse) {
console.log('fetchResponse: ', event.request.url);
await cache.put(event.request, fetchResponse.clone());
return fetchResponse;
}
} catch (error) {
console.log('Fetch failed: ', error);
const cachedResponse = await cache.match('/en/offline.html');
return cachedResponse;
}
})());
});
This does everything I need, in a very specific order. It checks the cache first, if found it's returned. It checks the network next, if found it caches it first then returns it. Or it displays a custom offline page with a big Reload button to encourage visitors to try again when they are back online.
But the most important this to realise is that doing it this way alows me to display a page and all it's resources with or without network access.
UPDATE: In order to deal with changes to CORS security requirements that where implemented in all browsers between March and August of 2020, I had to make one small change to the 'fetch' event.
Changed from:
const fetchResponse = await fetch(event.request);
To:
const fetchResponse = await fetch(event.request, {mode:'no-cors'});
Replace your fetch event code with this one. For every request your fetch event will be invoked and it will check if your request is found in the cache file list then it will serve the file from there otherwise it will make the fetch call to get the file from server.
self.addEventListener("fetch", function (event) {
event.respondWith(
caches.match(event.request)
.then(function (response) {
if (response) {
return response;
}
return fetch(event.request);
})
);
});
Also you don't need a separate "offline.html" file in your cache file list. Instead add your main application html file and your relevant css and js files in that list. That will make your application completely offline in case of no network.