I'm writing a web application in HTML/Javacript that records audio and uploads it to a server. Now, I would like to put it also in cache so it's available to service.workers for an offline scenario. What's the best way to do this?
Program flow:
Record audio
Capture data in a Blob
Save data on server
Listen to recorded stuff
If you are online, of course, all works well.
I would like to have the file locally available for listening before it is remotely saved, and backup it on server ASAP.
This is my routine:
function mettiincache(name, myBlob) {
var resp = new Response(myBlob)
var CACHE_NAME = 'window-cache-v1';
caches.open(CACHE_NAME).then(function (cache) {
cache.put(name, resp)
}).catch(function (error) {
ChromeSamples.setStatus(error);
});
}
When I look in Application/cache storage using Chrome DevTools, I find an entry with the correct path/name and content-Type, but with a content-Length of 0 bytes
Note that you might create / use a separate worker 'audioWorker.js' from the SW.js for running the apps audio cache because IMO its easier to test and the SW lifecycle is pretty involved and pretty oriented to its own app cache of 'real' urls used by the app.
Also note an inconsistency with allowable protocols used in the normal Service-Worker implementation that intercepts calls to 'fetch' - blob protocol used by the browser on your audio blobs will be rejected as invalid Urls by the browser.SW implementation. You cannot simply feed your blobs url to the normal SW lifecycle because its URL starts with 'blob://'.
The url from the audioBlob is fine if you choose NOT to use a SW for the cache. However, you might want to suffix it with a mimeType...
url = URL.createObjectURL(audio); // protocol of this is 'blob://'
wUrl = url +"?type=" + {$audio.blob.data.type};
console.log("SW CACH1 " +wUrl);
myCacheWorker.postMessage({ action: 'navigate', url: wUrl });
in the cacheWorker, onMessage , write to the cache:
onmessage = function( e ){
switch( e.data.action ){
case 'navigate':
upcache(e.data.url).then(() => {
postMessage({done: 'done'});
});
break;
}}
//boiler plate cache write below from any example
var upcache = function( url ){
return caches.open($cname)
.then((openCache) => {
return fetch(fetchUrl).then(function(resp) {
if (!resp.ok) {
throw new TypeError('Bad response status');
}
return openCache.put(url, resp);
})
});
}
you can use sql lite to store data in browser , there is a punch of tools that might help you in development to do that ,
i am using this tool my self , https://sqlitebrowser.org/ for debugging, testing and reading data from browsers ,
you can use Data series and publish it to client side as well .
you may refer to this link also reference how to use sql lite in browser , are you storing audio files as binary files? ,
generally , sql lite is good but you have to take care of storing sensitive data without encrypting it other wise it will be compromised also you may use Indexed Database API v 2.0 .
here is a link for more information about this .
https://www.w3.org/TR/IndexedDB/
Related
I've been trying to make offline only PWAs for Android, but the site's cache keeps clearing every so often. Is there any way to make the cache stay permanently?
You can define caching strategies for static assets and data requests for your service worker.
In the following article about service workers and caching strategies I list the different strategies and describe when it makes more sense to implement a specific one.
You can cache static assets and provide them offline when the SW is installing. Those files should be only the "minimum" version of your app (usually called app shell). Because of this, the cache.addAll method rejects the promise if it is not possible to get any of the resources. This means the service worker will install successfully only if all the targeted resources are cached.
self.addEventListener('install', function(event) {
event.waitUntil(
caches.open('staticAssetsCache').then(function(cache) {
return cache.addAll(
[
'/css/bootstrap.css',
'/css/styles.css',
'/js/jquery.min.js',
'/offline.html'
]
);
})
);
});
You can also cache HTTP GET Requests, for example below the stale while revalidate strategy that returns the data from the cache, if available, and in the background attempts to fetch and cache a newer version from the network:
self.addEventListener('fetch', function(event) {
event.respondWith(
caches.open('www.my-web-app.com')
.then(function(cache) {
return cache.match(event.request)
.then(function(response) {
var fetchPromise = fetch(event.request).then(function(networkResponse) {
cache.put(event.request, networkResponse.clone());
return networkResponse;
})
// response contains cached data, if available
return response || fetchPromise;
})
})
);
});
If you are using Angular or Workbox library, https://dev.to/paco_ita/create-progressive-web-apps-with-angular-workbox-pwa-builder-step-4-27d for more details.
I believe I read somewhere iOS Safari and Chrome would invalidate the cache frequently to get new updates. No logic behind it, just re-fetching the files.
Solution:
(In a recent Chrome devlog, it mentions a reduction in frequency from 3 days to 1)
to prevent the clearing of the cache / IndexDB I found this.
if (navigator.storage && navigator.storage.persist)
// '.persist()' will silently pass or trigger a dialog
navigator.storage.persist().then(function(persistent) {
alert(persistent ? 'persistent' : 'denied');
})
else
alert('not available - iOS / ancient Android?');
I have this one-page app with a dynamic URL built with a token, like example.com/XV252GTH and various assets, like css, favicon and such.
Here is how I register the Service Worker:
navigator.serviceWorker.register('sw.js');
And in said sw.js, I pre-cache the assets while installing:
var cacheName = 'v1';
var cacheAssets = [
'index.html',
'app.js',
'style.css',
'favicon.ico'
];
function precache() {
return caches.open(cacheName).then(function (cache) {
return cache.addAll(cacheAssets);
});
}
self.addEventListener('install', function(event) {
event.waitUntil(precache());
});
Note that the index.html (that registers the Service Worker) page is just a template, that gets populated on the server before being sent to the client ; so in this pre-caching phase, I'm only caching the template, not the page.
Now, in the fetch event, any requested resource that is not in the cache gets copied to it:
addEventListener('fetch', event => {
event.respondWith(async function() {
const cachedResponse = await caches.match(event.request);
if (cachedResponse) return cachedResponse;
return fetch(event.request).then(updateCache(event.request));
}());
});
Using this update function
function updateCache(request) {
return caches.open(cacheName).then(cache => {
return fetch(request).then(response => {
const resClone = response.clone();
if (response.status < 400)
return cache.put(request, resClone);
return response;
});
});
}
At this stage, all the assets are in the cache, but not the dynamically generated page. Only after a reload, can I see another entry in the cache: /XV252GTH. Now, the app is offline-ready ; But this reloading of the page kind of defeats the whole Service Worker purpose.
Question: How can I send the request (/XV252GTH) from the client (the page that registers the worker) to the SW? I guess I can set up a listener in sw.js
self.addEventListener('message', function(event){
updateCache(event.request)
});
But how can I be sure that it will be honored in time, ie: sent by the client after the SW has finished installing? What is a good practice in this case?
OK, I got the answer from this page: To cache the very page that registers the worker at activation time, just list all the SW's clients, and get their URL (href attribute).
self.clients.matchAll({includeUncontrolled: true}).then(clients => {
for (const client of clients) {
updateCache(new URL(client.url).href);
}
});
Correct me if I understood you wrong!
You precache your files right here:
var cacheAssets = [
'index.html',
'app.js',
'style.css',
'favicon.ico'
];
function precache() {
return caches.open(cacheName).then(function (cache) {
return cache.addAll(cacheAssets);
});
}
It should be clear that you cache the template since you cache it before the site gets build and this approach is not wrong, at least not for all types of files.
Your favicon.ico for example is a file that you would probably consider as static. Also, it does not change very often or not at all and it isn't dynamic like your index.html.
Source
It should also be clear why you have the correct version after reloading the page since you have an update function.
The solution to this problem is the answer to your question:
How can I send the request (/XV252GTH) from the client (the page that registers the worker) to the SW?
Instead of caching it before the service-worker is installed you want to cache it if the back end built your web page. So here is how it works:
You have an empty cache or at least a cache without your index.html.
Normally a request would be sent to the server to get the index.html. Instead, we do a request to the cache and check if the index.html is in the cache, at least if you load the page for the first time.
Since there is no match in the cache, do a request to the server to fetch it. This is the same request the page would do if it would load the page normally. So the server builds your index.html and sends it back to the page.
After receiving the index.html load it to the page and store it in the cache.
An example method would be Stale-while-revalidate:
If there's a cached version available, use it, but fetch an update for next time.
self.addEventListener('fetch', function(event) {
event.respondWith(
caches.open('mysite-dynamic').then(function(cache) {
return cache.match(event.request).then(function(response) {
var fetchPromise = fetch(event.request).then(function(networkResponse) {
cache.put(event.request, networkResponse.clone());
return networkResponse;
})
return response || fetchPromise;
})
})
);
});
Source
Those are the basics for your problem. Now you got a wide variety of options you can choose from that use the same method but have some additional features. Which one you choose is up to you and without knowing your project in detail no one can tell you which one to choose. You are also not limited to one option. In some cases you might combine two or more options together.
Google wrote a great guide about all the options you have and provided code examples for everything. They also explained your current version. Not every option will be interesting and relevant for you but I recommend you to read them all and read them thoroughly.
This is the way to go.
I'm currently considering adding service workers to a Web app I'm building.
This app is, essentially, a collection manager. You can CRUD items of various types and they are usually tightly linked together (e.g. A hasMany B hasMany C).
sw-toolbox offers a toolbox.fastest handler which goes to the cache and then to the network (in 99% of the cases, cache will be faster), updating the cache in the background. What I'm wondering is how you can be notified that there's a new version of the page available. My intent is to show the cached version and, then, if the network fetch got a newer version, to suggest to the user to refresh the page in order to see the latest edits. I saw something in a YouTube video a while ago but the presenter gives no clue of how to deal with this.
Is that possible? Is there some event handler or promise that I could bind to the request so that I know when the newer version is retrieved? I would then post a message to the page to show a notification.
If not, I know I can use toolbox.networkFirst along with a reasonable timeout to make the pages available even on Lie-Fi, but it's not as good.
I just stumbled accross the Mozilla Service Worker Cookbooj, which includes more or less what I wanted: https://serviceworke.rs/strategy-cache-update-and-refresh.html
Here are the relevant parts (not my code: copied here for convenience).
Fetch methods for the worker
// On fetch, use cache but update the entry with the latest contents from the server.
self.addEventListener('fetch', function(evt) {
console.log('The service worker is serving the asset.');
// You can use respondWith() to answer ASAP…
evt.respondWith(fromCache(evt.request));
// ...and waitUntil() to prevent the worker to be killed until the cache is updated.
evt.waitUntil(
update(evt.request)
// Finally, send a message to the client to inform it about the resource is up to date.
.then(refresh)
);
});
// Open the cache where the assets were stored and search for the requested resource. Notice that in case of no matching, the promise still resolves but it does with undefined as value.
function fromCache(request) {
return caches.open(CACHE).then(function (cache) {
return cache.match(request);
});
}
// Update consists in opening the cache, performing a network request and storing the new response data.
function update(request) {
return caches.open(CACHE).then(function (cache) {
return fetch(request).then(function (response) {
return cache.put(request, response.clone()).then(function () {
return response;
});
});
});
}
// Sends a message to the clients.
function refresh(response) {
return self.clients.matchAll().then(function (clients) {
clients.forEach(function (client) {
// Encode which resource has been updated. By including the ETag the client can check if the content has changed.
var message = {
type: 'refresh',
url: response.url,
// Notice not all servers return the ETag header. If this is not provided you should use other cache headers or rely on your own means to check if the content has changed.
eTag: response.headers.get('ETag')
};
// Tell the client about the update.
client.postMessage(JSON.stringify(message));
});
});
}
Handling of the "resource was updated" message
navigator.serviceWorker.onmessage = function (evt) {
var message = JSON.parse(evt.data);
var isRefresh = message.type === 'refresh';
var isAsset = message.url.includes('asset');
var lastETag = localStorage.currentETag;
// ETag header usually contains the hash of the resource so it is a very effective way of check for fresh content.
var isNew = lastETag !== message.eTag;
if (isRefresh && isAsset && isNew) {
// Escape the first time (when there is no ETag yet)
if (lastETag) {
// Inform the user about the update.
notice.hidden = false;
}
//For teaching purposes, although this information is in the offline cache and it could be retrieved from the service worker, keeping track of the header in the localStorage keeps the implementation simple.
localStorage.currentETag = message.eTag;
}
};
Right now I send my requests via ajax to the backend server, which does some operations, and returns a response:
function getData() {
new Ajax().getResponse()
.then(function (response) {
// handle response
})
.catch(function (error) {
// handle error
});
}
The thing is that each time a user refreshes the website, every request is sent again. I've been thinking about caching them inside the local storage:
function getData() {
if (Cache.get('getResponse')) {
let response = Cache.get('getResponse');
// handle response
return true;
}
new Ajax().getResponse()
.then(function (response) {
// handle response
})
.catch(function (error) {
// handle error
});
}
This way if a user already made a request, and the response is cached inside the localStorage, I don't have to fetch data from the server. If a user changes values from the getResponse, I would just clear the cache.
Is this a good approach? If it is, is there a better way to do this? Also, should I cache backend responses the same way? What's the difference between frontend and backend caching?
Is this a good approach? It depends on what kind of data you are storing
Be aware that everything stored on frontend can be changed by the user so this is potential security vulnerability.
This is the main difference between backend and frontend caching, backend caching can't be edited by the user.
If you decide to do frontend caching here is a code how to do it:
localStorage.setItem('getResponse', JSON.stringify(response));
For retrieving stored data from local storage
var retrievedObject = localStorage.getItem('getResponse');
NOTE:
I assume that you are storing object not a string or integer . If you are storing a string, integer, float... Just remove JSON.stringify
The best practice is to use The Cache API "a system for storing and retrieving network requests and their corresponding responses".
The Cache API is available in all modern browsers. It is exposed via the global caches property, so you can test for the presence of the API with a simple feature detection:
I have a Meteor Mobile app that accesses a lot of photos stored in my S3 bucket. These photos are user uploaded and change frequently. I don't want these photos to be accessible to anyone that isn't using my app. (ie: these photos are only viewable from my application and going to the url directly in a browser won't load them).
What is the best way to accomplish this? AWS Cognito seems to be the logical choice, but it doesn't seem easy to implement and I'm not exactly sure how to authenticate to AWS from the client once it gets a Cognito identity.
My other thought was putting a read only AWS Key on every url and authenticating that way, but that's almost pointless. It would be really easy to find out the key and secret.
EDIT:
To be specific, the URLs for the images are in a Mongo collection and I pass them into a template. So, the S3 resources are just loaded up with an image tag (<img src="). Something like AWS STS sounds like a great option, but I don't know of a way to pass the tokens in the headers when I'm loading them like this. Doing them as a pre-signed query string seems inefficient.
Another option is to restrict access with the referrer header, like this issue. But like Martijn said, it isn't really a secure way of doing it.
After some research and testing I solved this myself. My ultimate solution was to use the referer header to limit access to my S3 bucket. I created a more secure and detailed solution (see below), but it came with a performance hit that wouldn't work for my app. My app is based around viewing photos and videos, and not being able to have them load near instantly wasn't in the cards. Although, I feel like it could be a sufficient solution for most use-cases. Because my app isn't highly sensitive, the referer header is sufficient for me. Here is how to use the http header referer to limit access to a bucket.
Solution using Amazon's STS:
First, you need to have the AWS SDK on both the server and the client. There was no up to date packages for Meteor available, so I created my own. (I'll publish it shortly and put a link here once I do.)
On the server, you must use credentials that have the ability to assume a role. The role to be assumed must have a Trust Relationship with the user that is assuming the role. Article on using IAM. - Article on using credentials with SDK
In the server.js file I created a Meteor Method that I can call from the client. It first checks if a user is logged in. If that's true, it checks to see if it's current temp-credentials are expiring in the next 5 minutes. If they are, I issue new credentials and either write them to the user document or return them as a callback. If they aren't expiring in the next 5 minutes, I return their current temp-credentials.
You must use Meteor.bindEnvironmentfor the callback. See docs
Meteor.methods({
'awsKey': function(){
if (Meteor.userId()){
var user = Meteor.userId();
var now = moment(new Date());
var userDoc = Meteor.users.findOne({_id: user});
var expire = moment(userDoc.aws.expiration);
var fiveMinutes = 5 * 60 * 1000;
var fut = new Future();
if(moment.duration(expire.diff(now))._milliseconds < fiveMinutes ){
var params = {
RoleArn: 'arn:aws:iam::556754141176:role/RoleToAssume',
RoleSessionName: 'SessionName',
DurationSeconds: 3600 //1 Hour
};
var sts = new AWS.STS();
sts.assumeRole(params, Meteor.bindEnvironment((err, data) => {
if (err){
fut.throw(new Error(err));
}else{
Meteor.users.update({_id: user}, {$set: {aws: {accessKey: data.Credentials.AccessKeyId, secretKey: data.Credentials.SecretAccessKey, sessionToken: data.Credentials.SessionToken, expiration: data.Credentials.Expiration}}});
fut.return(data.Credentials);
}
}));
return fut.wait();
}else{
return userDoc.aws;
}
}
}
}
});
Then you can invoke this method manually or in an setInterval on Meteor.startup.
Meteor.setInterval(function(){
if(Meteor.userId()){
Meteor.call('awsKey', function(err, data){
if (err){
console.log(err);
}else{
if(data.accessKey){
Session.set('accessKey', data.accessKey);
Session.set('secretKey', data.secretKey);
Session.set('sessionToken', data.sessionToken);
}else{
Session.set('accessKey', data.AccessKeyId);
Session.set('secretKey', data.SecretAccessKey);
Session.set('sessionToken', data.SessionToken);
}
}
});
}
}, 300000); //5 Minute interval
This way just sets the keys in a Session variable from the callback. You could do this by querying the user's document to get them as well.
Then, you can use these temporary credentials to get a signed URL for the object you are trying to access in your bucket.
I put this in a template helper by passing the object name to it in the template:
{{getAwsUrl imageName}}
Template.templateName.helpers({
'getAwsUrl': function(filename){
var accessKey = Session.get('accessKey');
var secretKey = Session.get('secretKey');
var sessionToken = Session.get('sessionToken');
var filename = filename;
var params = {Bucket: 'bucketName', Key: filename, Expires: 6000};
new AWS.S3({accessKeyId: accessKey, secretAccessKey: secretKey, sessionToken: sessionToken, region: 'us-west-2'}).getSignedUrl('getObject', params, function (err, url) {
if (err) {
console.log("Error:" +err);
}else{
result = url;
}
});
return result;
}
});
That's all there is to it! I'm sure this can be refined to be better, but this is just what I came up with in testing it really fast. Like I said, it should work in most use cases. My particular one didn't. For some reason, when you tried to toggle the visibility: visible|hidden; on an img src of these signedURLs they would take a lot longer to load than just setting the URL directly. It must be because Amazon has to decrypt the signed URL on their side before return the object.
Thanks to Mikkel for the direction.