I am looking to implement/add realtime push notification updates from node.js server to browser (client).
I looked into socket.io (http://socket.io/docs/rooms-and-namespaces/)
business requirement is - users will visit a page displaying customer info & their orders. There will be ~10,000 users visiting the page at any given time (all 10,000 could be for different customers or sometimes a user may have opened the same page in 2 or 3 tabs)
When orders flow in to elasticsearch for a customer (my datastore) i want to push notification to users who have that customer's page opened.
Questions:
Is socket.io the correct framework for this case?
Am I correct in understanding I have to use socket.io' rooms functionality to implement this? (each room identifier equals customer ID?)
Is this implementation scalable and would it be memory intensive for 10k users on node.js server?
Thanks!
Yes, but you could consider socksJS as well and write your own simple back-end.
Yes, it's the easiest way if you need authentication.
Worst case scenario you'd need to cluster your socket.io servers and use a back-end adapter. Redis should be fast enough for 10.000 connections.
EDIT: memory will depend on your specific implementation.
Also consider https://github.com/Automattic/socket.io/issues/1393
Related
We are creating a video chat web app that is using Tokbox (PHP/JS).
We will have the same 5 "rooms" (sessions) all the time. A user can join one of the 5 rooms where a moderator will be conducting the chat.
We have gone through the hello world demos and made some various test apps and all seems to be working as we need it.
However, my question lies in managing the sessions/ids for our 5 chat rooms.
Is it (A) necessary to create unique session ids dynamically with the server PHP code? Or, since we are only going to have 5 rooms, can we (B) generate the session ids in our account control panel and hard code the session ids for the five rooms into the client side?
Option B, seems pretty straightforward.
Option A, I am thinking we would have to build an interface for the moderators to choose a room and then generate the room's session id when the moderator "activates" the room for users to then join.
Does anybody have any best practices on how to manage Tokbox session ids when you are using the same sessions over and over?
Thank you!
Adam here from the OpenTok team.
It should be fine to just generate the sessionIds once and reuse them (option B). You are not going to be able to hard-code the tokens though, you will still need to generate those with a server-side SDK in your app server. This is because the tokens expire, by default after 24 hours but you can configure that time. You generally want to generate a new token for every user each time they want to connect to a session.
I am working with a research group that needs to collect Facebook post data for a group of participants over the course of a few months. The goal is let the participants authorize us to collect this information via facebook, and when they post to Facebook, send that information to a secured server.
After reading the Facebook documentation, I'm mostly just confused about how to provide a mechanism for the user to agree to the terms of collecting his/her data. I'm planning on using webhooks to collect the data, which is independent of the client platform, but do I really need to build separate apps for web, iOS and Android just for the approval process?
I'm planning on using webhooks to collect the data, which is independent of the client platform, but do I really need to build separate apps for web, iOS and Android just for the approval process?
You most likely won't get this approved in review anyway.
Permissions must, for the most part, be used to provide a direct benefit to the in-app user experience; collecting data for analytical measures only is explicitly mentioned as a not allowed use case. Whether the users would agree is not relevant here.
For example, user_posts permission, has "non-visible use of this data such as sentiment analysis or guarding against spam bots" marked with a red "X" for nope.
So the only way of getting your app to request the necessary permission(s) from users, would be to add them as testers to the app via app dashboard/API. Anyone with a role in the app which can be asked for any permission, reviewed or not. Those users would however have to have their accounts verified (mobile/text, credit card) and sign up as a "developer" on the FB platform, before your app could send them requests to become testers.
You'd be flying under the radar with that, so to speak. Facebook offers this "loophole" mainly for the purpose of letting developers test and develop their apps properly, before review. It is also explicitly mentioned in the app development FAQ as a way around the need for review, for specific use cases such as wanting to get your blog posts published to your own FB page automatically. It would not cover what you are trying to do so, but it will likely "work" as long as none of your test users specifically raises a complaint with Facebook ...
Not sure if there is any published limit on the number of test users that can be added to an app, people have asked about that in the past, but AFAIK none is documented or otherwise published by Facebook. If there is one, and it is not completely stopping this approach dead in it's tracks (say, a few hundred per app rather than, five), you could use multiple app ids, if you don't need to relate data between user accounts too much - because the app-scoped user ids will be different per app id, so that could making connections between friends that are in different "app-id test groups" etc. difficult. Or you might have to refer to other, less reliable measures of uniquely identifying them, such as email or profile name.
I'm working on a pretty fun web app project that can become rather big, and I have a chance to play around with this handy thing called PubNub as main real-time engine of application.
So it's a web application with Node.js backend, involves potentially huge amount of chat rooms between users and realtime notifications sent to users by backend when some data in DB is updated.
Usually, developing with Sockets.io, I will just subscribe each user to channel of his unique DB id, and also to chanels representing different chat rooms.
This way I can handle chat rooms and authentication on backend and after storing some personal notification in DB I can easily push them to channel named by user id, so if user is online - he gets it, if not - fine, he will see it on next login, notification is already in DB. And theoretically this monstrocity should scale just fine horizontally with help of redis pub/sub.
Thing that worries me about PubNub in this case is scalability. As I obviously have no insight on what is going on in PubNub backend's dark corners, I want to make sure that app is built in the way that it will be prepared to handle some obscure enormously huge amount of simultaneous users.
My question is, what is the best approach to building such a system with PubNub?
Am I correct assuming that it will be better, in need of pushing notification to specific user, to subscribe to this user's pubnub, push note and unsubscribe. As if I will keep all online-user channels open - then there is no point in PubNub instead of websockets on my server, as server will be anyway under load of all of those opened online-user channels and should be scaled just to maintain huge quantity of them.
What about user authorisation? Without involving my backend how can I be sure that user posting some message will not be able to fake his personality and will have exatly the same as he have authenticated inside application?
And generally (and via PubNub) what is the best practice to tackle huge amounts of chats per users? As say during application life each user may accumulate some decent amount of garbage chat rooms that have some users in it, though havn't been touched by anyone for a long time, and users just way too lazy to leave it manually?
Thanks for Your patience in reading this wall of text!
UPDATED Dec 5, 2021
If you are implementing a chat app, please refer to the PubNub Chat use-case documentation for full details. It has new features and UI components that are built upon the PubNub Platform.
UPDATED May 15, 2020
We have some new docs that will explain much of the below in much clearer terms.
And new features that can be applied to many of the questions/answers below:
Message Actions
Message Counts
Batch History (multi-channel message fetch)
Objects (Users, Channels and Memberships Metadata)
NOTE: I've sprinkled some of the above links down in the answers below.
First, let's address this...
Thing that worries me about PubNub, in this case, is scalability. As I
obviously have no insight on what is going on in PubNub backend's dark
corners, I want to make sure that the app is built in a way that it will
be prepared to handle some obscure enormously huge amount of
simultaneous users.
and this...
then there is no point in PubNub instead of WebSockets on my server,
as the server will be anyway under the load of all of those opened online-user
channels and should be scaled just to maintain the huge quantity of them
This is sort of backward because you would use a service like PubNub to ensure that your application scales to handle millions of users. PubNub has thousands of customers that scale to millions of users and 100's of billions of messages. Not knowing how PubNub does this frees you to implement the biz logic of your application.
But I think I get what you are saying. You are under the impression that your server has to be involved in each and every chat room interaction for every user, but that is only partially true. Mostly, your server will be used for authentication, some subscription maintenance (optional), and probably for sending messages out to one, many, or all end users, as required (depends on your requirements).
Here are some attempts to answer your questions although they are kind of a bit all over the place so I will do my best to answer what I think it is you are asking.
Question 1
This question seems to be directed at maintaining lots of subscriptions to channels and the scalability of that.
Generally speaking, every end-user initializes PubNub and subscribes to channels they need to listen to and publish to channels they need to send messages on. Typically, the channels (chat rooms in your case, I assume) they are publishing on are the same channels they are subscribing to, but they are different kinds of use cases. And you can subscribe to thousands of channels at a time (up to 20K per client). If you did this with WebSockets, how would you go about scaling this to millions of users? You would implement and operate (to scale) something similar to PubNub (not easy and not cheap).
Now, if a user is subscribed to a bunch of chat room channels but some or many are stale (user hasn't viewed or posted to in a while), you could have some code on your server (or the client) that monitors the activity of users and unsubscribes them from those stale channels. This is possible using channels groups. Each end-user would have their own channel group that contains all the channels they are listening to. And the client code or the server code and add and remove channels to/from those end users' channel groups.
Question 2
UPDATED DOCS: https://www.pubnub.com/docs/platform/security/access-control
Now this question is a bit more clear and focused and is asking about authentication (login) and how to ensure someone is who they say they are and how to handle authorization (what they can and cannot do) and where/who controls this.
The answer is, you control the authentication (login) to prove that the person is what they say they are. Your log-in process checks for a valid username/password and in the user record, you will have a list of access controls for that user. With that, you generate an auth-key that you grant read and/or write access to one or more channels. This grant is a PubNub operation that your server invokes. The auth-key is passed back to the client and the client code initializes PubNub instance using the pub/sub keys and this auth-key that PubNub servers use to check for access based on the channel and the operation being requested (subscribe to this channel, publish to that channel, etc). If the auth-key does not have the proper access, the PubNub server will deny access (403 response).
There's more to all of this but this is a good start. Read up on PubNub Access Manager for the SDK you will be using on our docs page. For example, you can start with the JavaScript SDK Access Manager docs and tutorials.
Question 3
UPDATED DOCS: https://www.pubnub.com/docs/platform/channels/receive#subscribe-to-channels
I believe I answered this sufficiently with question 1 - Channel Groups. Start with the JavaScript SDK Stream Controller (which provides Channel Group feature) docs and tutorials.
I hope I have managed to move you a few steps further along your journey to a highly successful real-time, data stream application using PubNub. Please reply with any additional questions you may still have.
*Answers to your new comments:*
Thanks for your follow-up comments. It is very clear what you are asking now.
I will need to compare chat room timestamp with personal user last-read timestamp for this, so it seems that I need to listen to those channels from back-end and update user's last-reads, or to trust into the front-end, and get timestamps from a user directly
No, you do not have to listen to the channels on your server. Yes, from the client app, you will keep the timestamp of the last received message. When the user comes back online, you use this timestamp to get history for the channels the client was subscribed to. Many have done this successfully and we are going to be releasing some amazing features in the coming months that will simplify this considerably.
pushing real-time notifications to users from the back-end. Do I need to be subscribed to all of my user channels if I want to push notes to them at any time?
You can publish on any channel without actually subscribing to it first. So your server can publish to channels as it needs to.
And as before, keep coming with more questions as you require.
*Great follow-up questions again. Here's what I suggest*
... it makes sense to not request all of those chat rooms from DB and join via pubnub all of them, but rather implement pagination... how user can be aware of new messages that may appear in his old chat rooms?
Again, you can stay subscribed to 20K channels using channel groups. You can subscribe to 10 channel groups with 2K channels per channel group - but I'd recommend just limiting the user to 100 or less because that seems like a sufficient limit to impose in your app. But pick whatever upper limit you want and when the user hits that limit, force them to leave another chat room first or suggest they leave one of the top 10 most inactive, or some algorithm that makes sense for your app.
UPDATED DOCS: https://www.pubnub.com/docs/platform/channels/receive#subscribe-to-channels
Getting the # of missed messages does require a full history fetch, but we are going to be providing improved APIs to make this simpler in the near future. But if the user is registered for push notifications on all these channels, the device would be able to receive these push messages and your app can keep that count locally. We will have a "how to update the badge count in background" article being published soon. You could also use that to keep track of the number of missed messages per channel (chat room).
For now I just want to limit the number of rooms available for users to let's say a hundred and request and join them without pagination.
UPDATED DOCS: https://www.pubnub.com/docs/platform/channels/retrieve
We do have customers that do this without worrying about pagination. They just retrieve history on the 100 channels the device is subscribed to. With the background badge count updater strategy, you will have the advantage to know which channels to fetch from when the app becomes active. I will post the link to that article here once it is published.
I am currently trying to make a browser plugin using crossrider that will sync bookmarks but as there is no "on new bookmark" event how do I sync all upload new bookmarks to the server and update bookmarks lists on other connected machines in realtime? I would be inclined to do this with a websocket but as I said it is not event based. So do I poll all the clients every n seconds? That seems like a lot of data being moved and it seems taxing on the server as the process presumably wold include: first requesting all the clients bookmarks every n seconds and then comparing them to a list of the clients logged bookmars and, in the event of a match, update the clients browser's bookmarks. So in short, what is the best solution to the conundrum?
Given the scenario you describe, I think that polling it the best solution.
If I correctly understand the intention of your extension, the one thing I would do differently is how you compare the bookmarks. In your solution, you mention sending the bookmarks data to the server, comparing, and sending data back to the client to update the browser bookmarks.
I think a better more efficient solution would be to compare the bookmarks on the client-side and send new bookmarks to the server. You can store a snapshot of the bookmarks for use when comparing using our local database appAPI.db.async API and our appAPI.request API to send the new bookmarks to your server.
We at Crossrider are here to help you with any issues or questions you might have. Hence, if you need any assistance, please do not hesitate to contact our support (support#crossrider.com).
Disclaimer: I am a Crossrider employee.
I am building a node.js application. When a user connects, they transmit their state as online. When they disconnect, the state gets switched to offline. Also, when a user connects, he gets a list of his connections, along with the states of these friends(online or offline). Ok, great, no problem with this.
But the trouble I'm running into is how to dynamically send the state of a user's freind while I'm logged in. So, the steps would look like:
I log in.
I get a list of all my friends: user1, user2, user3
I get a list of states for all users: user1:online, user2:online user3:offline
user1 logs off
How do I dynamically get the state change of user 1?
Remember that these users aren't connected to each other, and also remember that a user's state needs to get transmitted to all the other user's who have him as a friend. Let's say that user1 has 100 friends, I am one of them. I (along with the other 99) need to get the notification that user1 is now offline. Also note that these people may not be on the same physical server (multiple servers handling X number of connections each). What is the best theoretical way to solve this problem?
The only answer that I could come up with is that I save the state in a DB, and that node will query the db, and when the state changes, it will update. But it seems crazy that node would have to watch for changes to the db, that can't scale. There's got to be a better way to do this - maybe its an obvious one and I'm missing it.
As hinted by ebohlman in the comments, the publish/subscribe pattern (also known as "pub/sub") sounds applicable to your problem. In a pub/sub architecture, instead of the publisher sending messages to specific clients, the publisher instead sends messages to some central pub/sub server. Subscribers notify this server for the kinds of messages, or "topics," they're interested in. The key is that the publishers and the subscribers never know, or even care, about each other, making the pattern very scalable (there can be any number of publishers or subscribers).
In your case, each of the servers that handle user login/logout might subscribe to the "login_changed" topic. Whenever a user logs in or out, the server that processed the change would send a message to the pub/sub server with the topic "login_changed" along with some data indicating what happened to whom (e.g. with a JSON-encoded string). Each of the servers that have subscribed to the "login_changed" topic will receive the message, and they can each determine if they're responsible for tracking the user in question.
There are several common tools for managing the pub/sub pattern. Redis, a common key-value store, has support for pub/sub and is a personal favorite tool of mine for the task. AMQP is an open standard that includes many kinds of messaging-related behavior, including pub/sub, and has many implementations, the most common of which is probably RabbitMQ. If you're interested in hooking up web clients over HTTP, you might be interested in something like Faye. There's a lot of literature on the subject, as well as many implementations in many languages; a Google search should take you far.