I have following question:
In my first scenario i have an S3 bucket. I can use the Storage API from the amplify SDK
const result = await Storage.put("profile.png", profileImage, { level: "protected" });
and later i can get it
const result = await Storage.get("profile.png", { level: "protected" });
Everything works fine. Everybody can read it, but only i can read/delete/update it.
But now here my question....
In my application a user can be the admin of a group. He can see all group members.
What if he fetch the list of all users. Lets say he fetch 10 users. Does this mean i need to make 10 requests for each image? This also means i need to save somewhere the ID of the other user.
Is this the correct way?
According to the docs Storage.get returns a pre-signed URL. A pre-signed URL is for a single object so yes, for 10 users you'd need to make 10 get requests.
However, there may be alternatives depending on your app requirements. For example, a common approach for S3 is to use a key prefix to group items. You could prepend the group-name/group-id for all profile pictures of people within that group.
S3 allows listing keys by a prefix, which is the Storage.list call. E.g. you could use a format like s3://my-bucket/{group-id}/{user-id}/profile.png then for the admin user you can get all items in the group with a call like const result = await Storage.list('{group-id}/');
Related
I would like a suggestion.
I'm developing a web app using react-js as frontend and firebase as backend.
For sake of semplicity, say the aim of the web-app is to allow the user to upload some items which are then displayed on the homepage
An item consists of two string: 1) name of item 2) url pointed to an image stored on IPFS
Once the user submit a form, the frontend creates a new doc in a collection in firestore:
await addDoc(collectionRef, newDoc)
Then, in the homepage, I display all the items in the collection
const querySnapshot = await getDocs(collectionRef);
querySnapshot.forEach((doc) => {
...
});
This works; however, if I've understood properly, every time the homepage is refreshed the frontend makes N read calls (where N is the number of docs in the collection). Therefore, I'm wondering if its the right approach, is there a better way?
I really appreaciate any suggestion (also regarding potential major safety flaws in this setting)
Every time the homepage is refreshed the frontend makes N read calls
(where N is the number of docs in the collection).
With const querySnapshot = await getDocs(collectionRef); this is indeed the case: all the docs in the collection are fetched.
If your collection contains a lot of documents and is fetched frequently this will cost you money and, at one moment, the performance of you app may be degraded (duration of the Firestore query execution and lot of docs transferred from the back-end to the front-end).
One approach to avoid that is to use pagination and fetch only the first X documents and display a link to fetch the X next ones, etc.
There is a page in the Firestore documentation dedicated to pagination implementation.
You could also implement an infinite scroll mechanism instead of a link, to load the next set of X document continuously as the user scrolls down the page. That's just a UI variation and is based on the exact same approach detailed in the Firestore documentation.
Also regarding potential major safety flaws in this setting
Fetching all the docs of a collection or using pagination does not make any difference in terms of security. In any case you should implement security according to your requirements by using security rules (and potentially Firebase Auth as well). And don't forget that security rules are not filters.
I am creating a multiplayer game website. I am using three features of firebase.
Firebase authentication
Firestore
Real time database
The data which is permanent is stored in firestore. Like profile image, username etc. The data in firestore is stored in collection users and the key is same as the authentication id which we get from user.uid
The second data is temporary data which contains the chat messages and current game situation and player turn etc. This data is stored in real time database.
There are two base objects in real time data base. One is rooms and other is users. When a user logs in to website the permanent data is taken from the firestore and placed with temporary data(because we might need to display the permanent data again and again). The function I am using to get permanent data and create combination with temp data is
//'uid' is the permanent id which is used in firestore and also in authentication.
export const addUser = async (uid: string) => {
//gets the permanent data from firestore
const data = await getUserData(uid);
//Set it to realtime database my adding one more temp prop
return await dbUsers.child(uid).set({...data, messages: []});
};
Till now everything is fine problem comes when I have to remove the user on disconnection. I used t
export const removeUser = async (uid: string) => {
return await dbUsers.child(uid).remove();
};
The above way doesn't work for multiple tabs. Consider if user had opened multiple tabs and he just closed one server then realtime database will consider it logged out.
Do I need to create realtime data on the basis of another id using push() method. Kindly guide me to correct path.
If I understand correctly you're trying to track the user's online status using Firebase's onDisconnect handlers. To do this:
You write a value for the user's UID when they connect.
You then delete that value using an onDisconnect handler.
This indeed will not work when the user opens the app in multiple locations (tabs, browsers, or devices). The reason is that a user can be online in multiple locations, and your code and data structure needs to cater for this.
The idiomatic approach is the one outlined in the sample presence app in the Firebase documentation, and works with a data structure like this:
"OnlineUsers": {
"uidOfUser1": {
"-LKeyOfConnection1": true,
"-LKeyOfConnection2": true
},
"uidOfUser2": {
"-LKeyOfConnection3": true,
"-LKeyOfConnection4": true
}
}
In this structure, if a user has two open connections (on different tabs, browsers, devices), they have two nodes under their UID, each with its own onDisconnect handler. When both connections are closed, with connection keys disappear, and thus their /OnlineUsers/$uid node also disappears automatically.
So to detect if a user is online in the above structure, you'd check if there is a node under /OnlineUsers with their UID.
I want to get all subscription groups (with product ids) in itunes connect from my backend side. I am looking at the documentation but I can not find any endpoint for that.
I mean, something similar to google which you can specify the packageName and it returns all product ids:
https://developers.google.com/android-publisher/api-ref/inappproducts/get
this.androidpublisher.inappproducts.list({
auth: this.authClient,
packageName: packageName
},
Does Apple provide an endpoint for this?
This can only be done from within your app with an SKProductsRequest. The products could be different based on the user's country (availability, pricing, etc.).
It may be easier to just store a hardcoded mapping of your product IDs -> data on your server.
I'm a relatively new Javascript programmer and I'm experimenting with the Marvel API (I need to access the images for a project) and having a little trouble wrapping my head around the requirements.
As I understand it, you need to pass a hash and a ts (timestamp, I presume), when calling the API from a server-side app. But I don't see in the documentation that this is required when using a client-side app.
I tried to do some basic endpoint testing with Insomnia and I receive the message "You must provide a hash.". Apparently I need the hash for client-side access as well?
I have seen some NodeJS examples that show you how to generate the hash (for example, https://www.raymondcamden.com/2014/02/02/Examples-of-the-Marvel-API), but nothing for the client side (that I could find). I also don't know how I would generate this within Insomnia (or Postman). Any pointers in the right direction would be appreciated.
I'd also like to ask what role the authorized domains play when accessing the Marvel API from a local machine. Do I need to add localhost to this list?
Thanks for any help!
Follow the steps:
Pick an API Endpoint. eg: https://gateway.marvel.com:443/v1/public/characters
Use a query value for ts. ts could be timestamp or any long string.
eg: ts=thesoer
Generate a MD5 hash of ts+privatekey+publickey through code or preferrably online. eg: md5(ts + privKey + pubKey)
For md5 hash: http://www.md5.cz/
Join the dots. URL?ts=val&apikey=key&hash=md5Hash.
eg. https://gateway.marvel.com:443/v1/public/characters?ts=thesoer&apikey=001ac6c73378bbfff488a36141458af2&hash=72e5ed53d1398abb831c3ceec263f18b
Add a pre-requisite script to your postman collection.
var pubkey = "your_public_key";
var pvtkey = "your_private_key";
var ts = new Date().getTime();
pm.environment.set("ts", ts)
pm.environment.set("apikey", pubkey)
var message = ts+pvtkey+pubkey;
var a = CryptoJS.MD5(message);
pm.environment.set("hash", a.toString())
And then you can make your calls like such
https://gateway.marvel.com/v1/public/characters?ts={{ts}}&apikey={{apikey}}&hash={{hash}}
See this collection.
Regarding your authorized domains, add your public IP.
We have our bucket with new Aws SDK API on AWS S3. We uploaded and tagged lots of files and folders with tags.
How can we filter on key-value tag, or only one of them? I'd like to find all the objects with key = "temp", or key = "temp" and value = "lol".
Thanks!
I also hoped that AWS will eventually support "search files by tags" because that would open up possibilities like e.g. having a photo storage with the names, descriptions, location stored in tags so I wouldn't need a separate database.
But, apparently AWS explicitly is not supporting this, and will probably never do so. Quoting from their storage service white paper:
Amazon S3 doesn’t suit all storage situations. [...] some storage needs for which you should consider other AWS storage options [...]
Amazon S3 doesn’t offer query capabilities to retrieve specific objects. When you use Amazon S3 you need to know the exact bucket name and key for the files you want to retrieve from the service. Amazon S3 can’t be used as a database or search engine by itself.
Instead, you can pair Amazon S3 with Amazon DynamoDB, Amazon CloudSearch, or Amazon Relational Database Service (Amazon RDS) to index and query metadata about Amazon S3 buckets and objects.
AWS suggests using DynamoDB, RDS or CloudSearch instead.
There seems to be one way to achieve what you're looking for, although it's not ideal, or particularly user-friendly.
The AWS S3 tagging documentation says that you can grant accounts permissions for objects with a given tag. If you created a new account with the right permissions then you could probably get the filtered list.
Not particularly useful on an ongoing basis, though.
AFAIK - Resource Groups don't support tags on an S3 Object level only on a bucket level.
Source: https://aws.amazon.com/blogs/aws/new-aws-resource-tagging-api/ (scroll down the page to the table).
This is now possible using AWS Resource Tagging API and S3 Select (SQL). See this post: https://aws.amazon.com/blogs/architecture/how-to-efficiently-extract-and-query-tagged-resources-using-the-aws-resource-tagging-api-and-s3-select-sql/.
However, the Resource Tagging API supports only tags on buckets for the S3 service, not on objects: New – AWS Resource Tagging API
There's no way to filter/search by tags. But you can implement this yourself using S3.
You can create a special prefix in a bucket, e.g. /tags/. Then for each actual object you add and want to assign a tag (e.g. Department=67), you add a new object in /tags/, e.g: /tags/XXXXXXXXX_YYYYYYYYY_ZZZZZZZZZ, where
XXXXXXXXX = hash('Department')
YYYYYYYYY = hash('67')
ZZZZZZZZZ = actualObjectKey
Then when you want to get all objects that have a particular tag assigned (e.g. Department), you have to execute the ListObjectsV2 S3 API for prefix /tags/XXXXXXXXX_. If you want objects that have particular tag value (e.g. Department=67), you have to execute the ListObjectsV2 S3 API for prefix /tags/XXXXXXXXX_YYYYYYYYY_
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html
It's not that fast but still does the job.
Obvious downside is that you have to remove the tags yourself. For example, you can do all of this above with a S3 triggers and lambda.
You should be able to query tags and values that you added
using resource-groups/query resource:
https://${region}.console.aws.amazon.com/resource-groups/resources
There is many way to get filter list of s3 by tag. I used in my code:
import boto3
from botocore.exceptions import ClientError
def get_tag_value(tags, key):
for tag in tags:
if tag["Key"] == key:
return tag["Value"]
return ""
def filter_s3_by_tag_value(tag_key,tag_value):
s3 = boto3.client('s3')
response = s3.list_buckets()
s3_list=[]
for bucket in response["Buckets"]:
try:
response_tags = s3.get_bucket_tagging(Bucket=bucket["Name"])
if get_tag_value(response_tags["TagSet"],tag_key) == tag_value:
s3_list.append(bucket["Name"])
except ClientError as e:
print(e.response["Error"]["Code"])
return s3_list
def filter_s3_by_tag_key(tag_key):
s3 = boto3.client('s3')
response = s3.list_buckets()
s3_list=[]
for bucket in response["Buckets"]:
try:
response_tags = s3.get_bucket_tagging(Bucket=bucket["Name"])
if get_tag_value(response_tags["TagSet"],tag_key) != "":
s3_list.append(bucket["Name"])
except ClientError as e:
print(e.response["Error"]["Code"])
return s3_list
print(filter_s3_by_tag_value(tag_key,tag_value))
print(filter_s3_by_tag_key(tag_key))
AWS now supports tagging of S3 images.
They have APIs to add/remove tags.
Amazon S3 Select, Amazon Athena can be used to search for S3 resources with TAGS.
Currently the max number of tags per resource is 10 (thanks Kyle Bridenstine for pointing out the correct number).