We are looking to use Algolia Search for an application. We like the convenience of Algolia but are stuck on one point. We have custom user groups and each user group can only see a subset of the records. When we are pushing records to Algolia all the records show up. How do we pair that with our custom logic of specific users can see specific records and we dont those to show up on the search lists.
The best way to handle this use case is to encode the permission information directly inside your records (like a group or a user). You can for example add a permission array on your record:
"permission": ["group1", "user42"]
You then just need to add this permission attribute in the list of attributes for faceting and apply the restriction in your query via a facetFilters argument.
I would also recommend to use the secured-API key feature that allows to apply this restriction in a secure way even if the query come from a browser or mobile app. A HMAC-SHA 256 signature is computed in your backend between the API key and the restriction to ensure no-one can change this restriction.
Related
I am new to Amazon Dynamo DB, I have created a user table and address table.and I want to retrieve all users with their particular address as I have assigned user_id in address table to each address. So how can I get user info with address with one query rather than querying both table and merge after. Is their any way like in MySQL we can use JOIN?
Dynamodb is not meant for these types of queries; especially aggregation queries are not ideal. DynamoDB is mainly good for fast lookups for predefined access patterns (e.g. get all items in shopping cart for user ID X).
Since addresses are unique properties of users, you might be able to add an attribute to the user ID table. So basically you have one table with all user data and their properties, including address, that you can query by user ID.
If you need to support different queries I'd suggest you note them all down first before deciding on your data model. (E.g. get all users and sort by last name, get all users living in city X). If the data model in dynamodb is too complex to support all these access patterns you might need to change to a SQL-like db instead.
Edit: note that there are ways to model relationships in dynamodb but they are not trivial. For some examples see the link below. But as suggested above, first define your access patterns before deciding on your data model.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-modeling-nosql-B.html
In my react based single page application, my page is divided in two panes.
Left Pane: Filter Panel.
Right Pane: Grid (table containing data that passes through applied filters)
In summary, I have an application that looks very similar to amazon.com. By default, when user hits an application's root endpoint (/) in the browser, I fetch last 7 days of data from the server and show it inside the grid.
Filter panel has couple of filters (e.g. time filter to fetch data that falls inside specified time interval, Ids to search data with specific id etc.) and a search button attached in the header of filter panel. Hitting search button makes a post call to a server by giving selected filters inside post form body, server returns back data that matches filters passed and my frontend application displays this data returned back from the server inside grid.
Now, when someone hits search button in the filter panel I want to reflect selected filters in the query parameter of the URL, because it will help me to share these URLs with other users of my website, so that they can see filters I applied and see data inside the grid matching these filters only.
Problem here is, if on search button click, I use http get with query parameters, I will endup breaking application because of limit imposed on URL length by different browsers.
Please suggest me correct solution to create such URLs that will help me to set the selected filters in the filter panel without causing any side effect in my application.
Possible solution: Considering the fact that we cannot directly add plain strings in query parameter because of URL length limitation from different browsers (Note: Specification does not limit the length of an HTTP Get request but different browsers implement their own limitations), we can use something like message digest or hash (convert input of arbitrary length into an output of fixed length) and save it in DB for server to understand the request and serve content back. This is just a thought, I am not sure whether this is an ideal solution to this problem.
Behavior of other heavily used websites:
amazon.com, newegg.com -> uses hashed urls.
kayak.com -> since they have very well defined keywords, they use
short forms like IN for INDIA, BLR for Bangalore etc. and combine
this with negation logic to further optimize maximum url length. Not
checked but this will ideally break after large selection of filters.
flipkart.com -> appends strings directly to query parameters and breaks
after limit is breached. verified this.
In response to #cauchy's answer, we need to make a distinction between hashing and encryption.
Hashing
Hashes are by necessity irreversible. In order to map the hash to the specific filter combination, you would either need to
hash each permutation of filters on the server for every request to try matching the requested hash (computationally intensive) or
store a map of hash to filter combination on the server (memory intensive).
For the vast majority of cases, option 1 is going to be too slow. Depending on the number of filters and options, option B may require a sizable map, but it's still your best option.
Encryption
In this scheme, the server would send its public key to the client, then the client could use that to encrypt its filter options. The server would then decrypt the encrypted data with its private key. This is good, but your encrypted data will not be fixed length. So, as more options are selected, you run into the same problem of indeterminate parameter length.
Thus, in order to ensure your URL is short for any number of filters and options, you will need to maintain a mapping of hash->selection on the server.
How should we handle permanent vs temporary links?
You mentioned in your comment above
If we use some persistent store to save the mapping between this hash to actual filters, we would ideally want to segregate long-lived "permalinks" from short-lived ephemeral URLs, and use that understanding to efficiently expire the short-lived hashes.
You likely have a service on the server that handles all of the filters that you support in your application. The trick here is letting that service also manage the hashmap. As more filters and options are added/removed, the service will need to re-hash each permutation of filter selections.
If you need strong support for permalinks, then whenever you remove filters or options, you'll want to maintain the "expired" hashes and change their mapping to point to a reasonable alternative hash.
When do we update hashes in our DB?
There are lots of options, but I would generally prefer build time. If you're using a CI solution like Jenkins, Travis, AWS CodePipeline, etc., then you can add a build step to update your DB. Basically, you're going to...
Keep a persistent record of all the existing supported filters.
On build, check to see if there are any new filters. If so...
Add those filters to the record from step 1.
Hash all new filter permutations (just those that include your new filters) and store those in the hash DB
Check to see if any filters have been removed. If so...
Remove those filters from the record from step 1.
Find all the hashes for permutations that include those filters and either...
remove those hashes from the DB (weak permalinks), or
Point that hash to a reasonable alternative hash in the DB (strong permalinks)
Lets analyse your problem and the solution possible.
Problem : You want a URL which has information about the filter applied so that when you share that URL user doesn't land on arbitrary page.
Solutions:
1) Append filter applied with URL. To achieve this you will need to shorten the key of type of filter and the value of filter so that Length of URL don't exceed much for each filter.
Drawback: This is not most reliable solution as the number of filter increase URL length has to increase no other option.
2) Append a unique key of filter applied(hash) with URL. To achieve this you will need to do some changes on server and client both. On client side you will need a encoding algorithm which convert filter applied to unique hash. On server side you will need decoding algorithm which convert unique hash to filter applied. SO now client whenever a URL like this is hit you can make a POST api call which take this hash give you the array of filter applied or on client side only put the logic to convert this hash.
Do all this in componentWillMount to avoid any side effect.
I think 2nd solution is scalable and efficient in almost all cases.
I am making an application which uses both the accounts package and facebook's graph api. Specifically the friends api. The friends api returns all facebook friends that have used the application. The problem is that it returns facebook id's, and the accounts package generate application specific id's. This is problematic when i want to retrieve information from a collection containing a friends information, but stored with the application specific id. I have worked around this by storing both the fb id and the accounts id in the collection.
But i still can't update a user data based on their fb id, as update is only permitted using the application specific id. What i want, but not allowed:
UserData.update({fbId: friend.fbId},{$push: {some: data}});
The only solution i could think of is to get each user id first, like this:
var friendId = UserData.findOne({fbId: friend.fbId})._id;
This is obviously not a good solution as it needs one extra db call for every update.
Is there a way of setting the accounts id equal to the facebook id upon creation? Or do you have any other suggestions.
Extending on the comment above:
MoeRum: #Xinzz UserData is a custom collection. If try updating with fbId I get the
following error: Uncaught Error: Not permitted. Untrusted code may
only update documents by ID. [403]
That is because you're trying to update on the client-side. You can only update by ID on the client-side. What you're trying to do should not be a problem as long as you do it on the server.
From the Meteor docs (for more reference: http://docs.meteor.com/#/full/update):
The behavior of update differs depending on whether it is called by
trusted or untrusted code. Trusted code includes server code and
method code. Untrusted code includes client-side code such as event
handlers and a browser's JavaScript console.
Trusted code can modify multiple documents at once by setting multi to
true, and can use an arbitrary Mongo selector to find the documents to
modify. It bypasses any access control rules set up by allow and deny.
The number of affected documents will be returned from the update call
if you don't pass a callback.
Untrusted code can only modify a single document at once, specified by
its _id. The modification is allowed only after checking any
applicable allow and deny rules. The number of affected documents will
be returned to the callback. Untrusted code cannot perform upserts,
except in insecure mode.
My company has a social networking platform that is accessed via a URL.
We are trying to find a way to advertise our URL in sports stores, with access to our site only possible if you come to the store - we do not want the sharing of the URL to anyone, anywhere.
We have considered QR codes and wonder if it's possible our site can only be accessed when a provided QR code is scanned.
Please let me know if you have any suggestions.
You are basically looking for keys/ access codes that give your customers access to a site.
Those might have to be on a per-user basis, as otherwise one might just leak an access code for the whole public to use.
If sharing keys should be disallowed:
You need a database (sql) to store your cusstomers' information.
Depending on how you generate a key (dynamically, by a set of rules or randomly, using a catalogue of valid keys) you might need a further table to store the keys separately (in case you choose the more secure option of generating a predefined set of random keys)
You can then include those keys in your QR Codes' target URL like www.example.com?key=1jh303u or something similar.
(This means of course that you have to produce customized QR Codes, which in turn means they cannot be printex as a standard mass-produced offset job, but as a customized digital print - so you'd have to send all the different generated QR Codes to your printer)
Once the user visits this URL containing the query string, your site can then check to see if the key is a) valid (in the table) and b) unused, by taking the &_GET["key"] variable and querying the database.
If the key is invalid, output an "access denied" page.
If the key exists but has already been checked in, you can use a user-based login system to handle the login.
If the key is valid and hasn't been used yet, you can output your exclusive content at last.
If it doesn't matter whether people will be able to share their key, you don't need a database at all. You could build a keygen which creates keys after a certain destinct pattern, and use that same set of rules to validate against the entered key.
I'm building a node.js application that opens up a connection to the Twitter Streaming API (v1.1)
I would like to filter multiple keywords (hashtags & words) as separate queries. My original idea was to have multiple public streams.
However, I understand that I can only have one open connection to the Twitter streaming api per application and per IP address and that Twitter encourages us to come up with creative solutions to get what we want.
So my question is this:
If I stream with no filters, such as using statuses/sample (which I believe is 1%) and use custom javascript to filter the output, would I get the same tweets if I used the API method of filtering (i.e track='twitter').
Edit: I have created a diagram explaining this:
As you can see, I want to know if the two outputs wil be the same. I suspect that they won't be because although both outputs are effectively the same filter, one source is a 1% sample, and maybe the other source is a 100% sample but only delivering 1% tweets from that.
So can someone please clarify if both outputs are the same?
Thank you.
According to the Twitter streaming api rules, if the keywords that you track doesn't exceed 1% of the whole global traffic you will receive all data (some tweets might be lost due to network issues etc but it is not significant). This is called garden-hose (firehose is a special filter which gives you all the data but it is given as a paid service through third parties such as http://datasift.com/)
So if a tweet is filtered through public stream then it would be part of your custom filter too unless your keyword set is too broad.
By using custom filters you can track multiple search keywords, and if you miss some data because your keyword set is too broad twitter sends a track limitation notice indicating how much data you are missing.
My suggestion to you would be to use a custom filter and analyze what you get from the stream and what you get as a result for the same keywords from twitter. And when you start getting track limitation notice from twitter, it is time for you to split your keyword set into chunks and start streaming through different streamers by running them from different machines.
The details of the filter streaming is below (taken from official website https://dev.twitter.com/docs/api/1.1/post/statuses/filter)
Returns public statuses that match one or more filter predicates. Multiple parameters may be specified which allows most clients to use a single connection to the Streaming API. Both GET and POST requests are supported, but GET requests with too many parameters may cause the request to be rejected for excessive URL length. Use a POST request to avoid long URLs.
The default access level allows up to 400 track keywords, 5,000 follow userids and 25 0.1-360 degree location boxes. If you need elevated access to the Streaming API, you should explore our partner providers of Twitter data here.
I would like to answer my question with the results of my findings.
I tested both side by side in the same time frame and concluded that the custom filter method, whilst it supports multiple filters does not provide enough tweets to create an interesting enough visualisation.
I think the only way to get something more interesting with concurrent filters is to look at other methods but I am wondering if its not possible. Maybe with a third party.
I have attached a screenshot of the visualisation tracking 'barackobama' The left is the custom filter, the right is statuses/filter.
The statuses/filter api operate on all tweets, instead of those returned by statuses/sample, you can tell by looking at their tweet id's: sample tweets all come from a specific time window. So from millisecond-resolution creation time, you can definitely tell that filter returns tweets outside of sample.
For more details about getting creation time from tweet id and the time window on sample tweets, consult this post: http://blog.falcondai.com/2013/06/666-and-how-twitter-samples-tweets-in.html