I've created a service worker that performs a fetch and then immediately stores the data in the cache.
self.addEventListener('install', async function(e) {
try {
const fileCache = await caches.open(CACHE_NAME);
await fileCache.addAll(FILES_TO_CACHE);
const dataCache = await caches.open(DATA_CACHE_NAME);
const dataFetchResponse = await fetch('/api/transaction');
return await dataCache.put('/api/transaction', dataFetchResponse);
} catch (error) {
console.log(error);
}
});
After performing this I'm trying to make it so that I can test how long it's been since the last data fetch in order to determine if the data needs to be updated. My Transaction model adds a timestamp onto the data so I'd ideally want to test the timestamp of the last transaction against the current time but I'm not sure how to do it.
I've attempted to do cache.match() but it doesn't return the entire object of the matched key. I know localStorage has a .getItem() method but I don't see any methods for cache that are similar.
Any ideas on this one?
Figured out how to get the information. You have to .match() the key of the data stored in the cache and since it returns a promise you convert that into a .json() object.
async function getCachedData(cacheName, url) {
const cacheStorage = await caches.open(cacheName);
const cachedResponse = await cacheStorage.match(url); // Returns a promise w/ matched cache
if(!cachedResponse || !cachedResponse.ok) {return false}
console.log(await cachedResponse);
console.log(await cachedResponse.json()); // prints json object with value of key matched
return await cachedResponse.json();
};
Related
I'm trying to perform a fetch request within a transaction but when the code executes I receive the following error.
Error: Cannot modify a WriteBatch that has been committed.
The steps the function is performing are the following:
Compute document references (taken from an external source)
Query the documents available in Firestore
Verify if document exists
Fetch for further details (lazy loading mechanism)
Start populating first level collection
Start populating second level collection
Below the code I'm using.
await firestore.runTransaction(async (transaction) => {
// 1. Compute document references
const docRefs = computeDocRefs(colName, itemsDict);
// 2. Query the documents available in Firestore
const snapshots = await transaction.getAll(...docRefs);
snapshots.forEach(async (snapshot) => {
// 3. Verify if document exists
if (!snapshot.exists) {
console.log(snapshot.id + " does not exists");
const item = itemsDict[snapshot.id];
if (item) {
// 4. Fetch for further details
const response = await fetchData(item.detailUrl);
const detailItemsDict = prepareDetailPageData(response);
// 5. Start populating first level collection
transaction.set(snapshot.ref, {
index: item.index,
detailUrl: item.detailUrl,
title: item.title,
});
// 6. Start populating second level collection
const subColRef = colRef.doc(snapshot.id).collection(subColName);
detailItemsDict.detailItems.forEach((detailItem) => {
const subColDocRef = subColRef.doc();
transaction.set(subColDocRef, {
title: detailItem.title,
pdfUrl: detailItem.pdfUrl,
});
});
}
} else {
console.log(snapshot.id + " exists");
}
});
});
computeDocRefs is described below
function computeDocRefs(colName, itemsDict) {
const identifiers = Object.keys(itemsDict);
const docRefs = identifiers.map((identifier) => {
const docId = `${colName}/${identifier}`
return firestore.doc(docId);
});
return docRefs;
}
while fetchData uses axios under the hood
async function fetchData(url) {
const response = await axios(url);
if (response.status !== 200) {
throw new Error('Fetched data failed!');
}
return response;
}
prepareMainPageData and prepareDetailPageData are functions that prepare the data normalizing them.
If I comment the await fetchData(item.detailUrl), the first level collection with all the documents associated to it are stored correctly.
On the contrary with await fetchData(item.detailUrl) the errors happens below the following comment: // 5. Start populating first level collection.
The order of the operation are important since I do now want to make the second call if not necessary.
Are you able to guide me towards the correct solution?
The problem is due to the fact that forEach and async/await do not work well together. For example: Using async/await with a forEach loop.
Now I've completely changed the approach I'm following and now it works smoothly.
The code now is like the following:
// Read transaction to retrieve the items that are not yet available in Firestore
const itemsToFetch = await readItemsToFetch(itemsDict, colName);
// Merge the items previously retrieved to grab additional details through fetch network calls
const fetchedItems = await aggregateItemsToFetch(itemsToFetch);
// Write transaction (Batched Write) to save items into Firestore
const result = await writeFetchedItems(fetchedItems, colName, subColName);
A big thanks goes to Doug Stevenson and Renaud Tarnec.
I have to loop through an array of objects and modify one single property in each object. I modify this property with a function that connects to the Twitter API. My problem is that I must be using async and await wrongly because I am getting a promise pending.
This is my code:
getProfile:(req,res)=>{
try {
const userId=req.params.id
const profile=db.query('SELECT * FROM profiles WHERE user_id=?',
[userId],async (err,result)=>{
if(err) return res.status(404).send(err)
const profiles= await result.map( obj=>{
const container={}
container['name']=obj.profile_name
container['desc']=obj.profile_desc
container['twitter']= connectTwitt.getTwitt(obj.twitt)//calls to api
return container
})
console.log(profiles)// promise pending
res.send(profiles)
This is the structure of the array of object that I am mapping:
[
{profile_name:`Elon Musk`, profile_desc:'enterpreneur',twitt:636465}
]
Yes, you are using the async/await syntax a little bit incorrectly.
Right now, you are calling await on the Array.map() method. However, that method is not promise-based.
Instead, you have to add the await keyword to the getTwitt() method, and await for all promises to complete.
With those changes, it should look like below.
const profiles = await Promise.all(result.map(async (obj) => { // This line has been modified
const container = {};
container["name"] = obj.profile_name;
container["desc"] = obj.profile_desc;
container["twitter"] = await connectTwitt.getTwitt(obj.twitt); // This line has been modified.
return container;
}));
Hopefully this helps with your <pending> issue!
I'm trying to practice some web scraping with prices from a supermarket. It's with node.js and puppeteer. I can navigate throught the website in beginning with accepting cookies and clicking a "load more button". But then when I try to read div's containing the products with querySelectorAll I get stuck. It returns undefined even though I wait for a specific div to be present. What am I missing?
Problem is at the end of the code block.
const { product } = require("puppeteer");
const scraperObjectAll = {
url: 'https://www.bilkatogo.dk/s/?query=',
async scraper(browser) {
let page = await browser.newPage();
console.log(`Navigating to ${this.url}`);
await page.goto(this.url);
// accept cookies
await page.evaluate(_ => {
CookieInformation.submitAllCategories();
});
var productsRead = 0;
var productsTotal = Number.MAX_VALUE;
while (productsRead < 100) {
// Wait for the required DOM to be rendered
await page.waitForSelector('button.btn.btn-dark.border-radius.my-3');
// Click button to read more products
await page.evaluate(_ => {
document.querySelector("button.btn.btn-dark.border-radius.my-3").click()
});
// Wait for it to load the new products
await page.waitForSelector('div.col-10.col-sm-4.col-lg-2.text-center.mt-4.text-secondary');
// Get number of products read and total
const loadProducts = await page.evaluate(_ => {
let p = document.querySelector("div.col-10.col-sm-4.col-lg-2").innerText.replace("INDLÆS FLERE", "").replace("Du har set ","").replace(" ", "").replace(/(\r\n|\n|\r)/gm,"").split("af ");
return p;
});
console.log("Products (read/total): " + loadProducts);
productsRead = loadProducts[0];
productsTotal = loadProducts[1];
// Now waiting for a div element
await page.waitForSelector('div[data-productid]');
const getProducts = await page.evaluate(_ => {
return document.querySelectorAll('div');
});
// PROBLEM HERE!
// Cannot convert undefined or null to object
console.log("LENGTH: " + Array.from(getProducts).length);
}
The callback passed to page.evaluate runs in the emulated page context, not in the standard scope of the Node script. Expressions can't be passed between the page and the Node script without careful considerations: most importantly, if something isn't serializable (converted into plain JSON), it can't be transferred.
querySelectorAll returns a NodeList, and NodeLists only exist on the front-end, not the backend. Similarly, NodeLists contain HTMLElements, which also only exist on the front-end.
Put all the logic that requires using the data that exists only on the front-end inside the .evaluate callback, for example:
const numberOfDivs = await page.evaluate(_ => {
return document.querySelectorAll('div').length;
});
or
const firstDivText = await page.evaluate(_ => {
return document.querySelector('div').textContent;
});
I have two calls to Firebase: one to get the existing data and one to listen for updates in the data. When those updates happen, instead of replacing the existing data for some reason I see to be adding the two datasets together. Can't figure out why as I'm directly updating state with new data in my second function.
Here are the functions called on mounted():
mounted() {
this.getImages();
this.refreshImages();
},
And the two functions in question:
async getImages() {
let snapshot = await db
.collection("Maria")
.orderBy("timestamp", "desc")
.get();
snapshot.forEach((doc) => {
let appData = doc.data();
appData.id = doc.id;
this.picturesData.push(appData);
});
this.dataLoaded = true;
},
async refreshImages() {
await db
.collection("Maria")
.orderBy("timestamp", "desc")
.onSnapshot((snapshot) => {
let newPicturesData = [];
snapshot.forEach((doc) => {
let newPictureData = doc.data();
newPictureData.id = doc.id;
newPicturesData.push(newPictureData);
});
this.picturesData = newPicturesData; // this should overwrite the data in my state, right? But instead it's appending.
});
},
It's difficult to tell you exactly what's happening without thoroughly testing your code but you have to note that the two calls (to getImages() and refreshImages()) may not be done in the order you expect.
Since in getImages() you push the data to picturesData and in refreshImages() you replace picturesData, I suspect that the listener set through refreshImages() returns data before you get the result of the query triggered by getImages().
Actually, since onSnapshot() triggers an initial call that returns the entire result of the query, you only need to call refreshImages() (you don't need the initial call to getImages()).
Note that onSnapshot() is not an asynchronous method like get(), so you don't need to make refreshImages() async.
I'm currently trying to track the progress of a chain of native es6 promises, and am wondering what the 'correct' way to go about this is.
I've simplified the actual code to thie following example, which is a basic set of chained promises (in reality, the promise chain is longer, and the session status value changes in more places depending on progress through the chain):
let sessions = {}
const asyncFunc = () => {
// Get a new id for state tracking
let session_id = getID()
sessions.session_id = 'PENDING'
// Fetch the first url
let result = api.get(url)
.then(res => {
// Return the 'url' property of the fetched data
return res.url
})
.then (url => {
// Fetch this second url
let data = api.get(url)
sessions.session_id = 'SUCCESS'
// Return the whole data object
return data
})
.catch(err => {
console.log("ERR", err)
sessions.session_id = 'ERROR'
})
return result
}
asyncFunc()
.then(res => {
console.log("URL", url)
})
This code tracks the state of the functions and stores them to the global sessions object - but the session_id isn't being passed back for inspection of status while the function is 'in-flight'.
One option I'm considering is adding the session_id as a property of the promise when it is returned, so this can be inspected - however I'm not sure if adding a property to a promise is a risky/hacky thing to do? Something like (simplified from above):
const asyncFunc = () => {
// Get a new id for state tracking
let session_id = getID()
sessions.session_id = 'PENDING'
// Fetch the first url
let result = api.get(url)
.then(...)
.then(...)
.catch(...)
// Add the session_id to the promise
result.session_id = session_id
return result
}
let func = asyncFunc()
let status =sessions[func.session_id]
func.then(...)
Any thoughts on the validity of this approach? I can see that I would probably also need to push the session id into the final return value as well, (so that the property exists in both the promise, and the resulting value of the resolved/rejected promise).
Alternatively, any other ways of handling this?
The obvious one is to make the function always return an array of arguments (promise and session_id) but I'd prefer to avoid having to always do e.g.:
let func = asyncFunc()
let status =sessions[func[1]]
func[0].then(...)