Develop Reference

How does one create a webhook for a personal project? (JavaScript, Node)

I recently learned about webhooks and am trying to implement one for a full-stack app that I'm building. However, I haven't been able to find information on how to create my own webhook. This is how my app is expected to work:
My client side is a website built with React, and when it loads, it will make a GET request to my server (built with Node and Express). The response will provide data that will be displayed on the website. The user will click a button on the website, which will send a (POST? GET?) request to my server, which will make its own POST request to send some of the data to an external API. When that external API data completes work on the data, it will send a POST request to my webhook URL (which I assume is part of my server), and the data from the external API needs to be sent to the client, which will display it alongside the original data. Additionally, when work on the first batch of data is completed, my server sends another POST request (with some more of the original data) to the external API, which will then make a POST request to my webhook/server with new data that then gets passed to the client again, and this repeats until all the original data from my server has been passed to and processed by the external API, and the all the processed data gets sent to my server and then to the client.
From my understanding, the webhook will allow my client to make a single request to my server, and then, my server can make multiple response to my client: one response for each POST request received from the external API. However, I have not been able to find information on how to create a webhook. Everything that I have found has been about how to use webhooks with pre-existing apps/websites like Dischord or GitHub or Stripe. Or using 3rd-party services to use their webhooks to connect pre-existing app/websites.
How do I go about making a webhook for my little project that will update my website every time that my server receives a POST request from the external API? Or am I misunderstanding how a webhook works? Can they only be created by 3rd-party services and/or only used on pre-existing apps?

For a simple use case, it just means another endpoint on your server that will receive a request from the external API when it's done with the task. It's a common way REST APIs work to let you know they are done with something, or communicate anything back to you in general. You have one endpoint that will receive a request from the React client, and send information off to the external API (as you said), and another endpoint (this "webhook") that will receive requests from the external API. That's it.
You described the client-server communications for this problem just fine with the exception that the server cannot send requests to the client over HTTP. If the client needs anything from the server, it needs to send a request and ask for it. A common way to do for this would be to store the information on the server once the external API hits your "webhook", and the client can ask the server if the data is back every X seconds (long-polling).
Example (payment flow):
Client -> Server: Heres my credit card number
Server -> Client: Thanks! Let me process that.
Server -> External API: Please process this credit card for the amount of $X and ping me at this webhook when you're done.
External API -> Server: OK, will let you know when I'm done.
Client -> Server: You done yet?
Server -> Client: Nope!
External API -> Server (to webhook): Hey, I'm done, here's the receipt.
Server -> External API: Thanks!
Client -> Server: You done yet?
Server -> Client: Yes! Here's your receipt.
More advanced ways to solve the problem would be using a long-lived bidirectional communication protocol such as websockets. And as others have mentioned, all of the endpoints don't have to be on the same server. You can have one server for each endpoint, and you can have turn them on only when they get a request and then go back to sleep (that's a Lambda). Pros and cons to this approach. That's more advanced stuff and if you're just learning I wouldn't really worry about it now.

AFAIK, webhooks are just normal web APIs. Properties that make an API a webhook:
The client sending requests to your server, in this case, is another server
This client will send you a request when there's a triggered event
In your case, your server POSTing data to the external API. What makes webhook different is that the external API's server doesn't have to respond right away, instead, it sends back you, for example, an HTTP 200 OK to acknowledge that it received the request and has started to process the data. When the external API's server finishes processing the data (which is an event), it will call your server's endpoint to inform you about the process along with the processed data (if succeeded).
Your server, correspondingly, will then have to inform your client (the user) about the processed data. I would suggest implementing some kinds of persistent connection (WebSocket or SSE), or simply just make the client poll request every some second.

"(which I assume is part of my server)"-- The webhook is part of the third party API that points back to either your server or maybe some serverless function/lambda(that you set up) that in turn can parse and perform computations on the data and then send it back to your server.
It really depends on the use case and architecture of your API. If you don't want your main API server to get overloaded, if say you need to perform heavy I/O, CPU/GPU processing, you could create Serverless Lambda Functions to handle this data processing and be the endpoint you supply to the third party's webhook post endpoint.
If you're not concerned with it, you could just create another endpoint in your server to handle this webhook.
Just keep in mind, when you're trying to scale up, you want your systems to be as distributed as possible-- within reason-- so your API can handle more requests and not be overloaded with a bunch of processes.

Sending object from Node.js server to JS client [duplicate]

I'm using node to make an email client. I want to update the browser window as soon as any new email arrives. I'm already fetching the emails on server side using mail-listener2. I want to push that data to client side. How do i do that?

Look at using websockets to push data in realtime to the client (browser). I would recommend using http://socket.io. They have some really great demos on there to show how you would do something like this.
Your server would send socket.io messages called message, containing the message details as JSON. Your client would listen for message events and update the UI as they come in.

Can Nodejs validate a PHP user's session?

What am I working on?
I am trying to establish a communication between a PHP app and a telephony system's REST API.
Since I my site is all written in PHP, I decided to build the communication using PHP by making cURL calls to the API.
to bring you up to speed, there are 2 types of communication between the user and the API and I like to put them into two different categories
Send Once / Receive Once Example of this would, be a user attempt to dial a new phone number "dial 800-123-4567." The API takes the request and return back an interaction id to allow the user to control the call (i.e. disconnect, mute, put on hold.... )
Send Once / Receive Every Second In this communication, I will create "persistent" connection between the user's session and the API. Then every second, I will check the API for new messages. After the message from the API is received, I must update the user's cache, read the latest user's cache, and finally send the browser the cache data.
Problem? HTTP is stateless.
Every request the user send to the web server, it generates a new TCP connection. The issue with this is that every second I query the API for new messages I will have a new TCP connection. On average about 200 TCP connections are needed at any giving time per user. So if I have 300 users using the app/server, then that is about 60,000 TCP connection open for the web server. As you can clearly see the solution does not scale well here and it is a matter of time before the server blow up in my face... :(
Another issue is that PHP is not asynchronous which cause problem if the communication to the API took longer or return errors.
FWIW, I have tried to user JavaScript SharedWorker to eliminate some of the overhead. I every tried Server-sent-events but a user still generated too many TCP connections to the server. nothing first the problem I was only able to reduce the connection a little.
Can Nodejs help?
I was advised by couple of people to use Nodejs instead of PHP for this task. Of course, I am not going to change my PHP application into Nodejs as this would be insane since my app is huge.
I would like to consider running a nodejs server as a middle man between the PHP server and the API service. The idea is to have a WebSocket running on the node server. Then, the client will pass any communication to the websocket and the websocket will then send the communication to the server. It does not sound bad at a high level but once a dig deeper, it seems to be getting trickier.
Nodejs Challenge
When a user logs into my PHP App, I validate their credentials and once they are in then I create a session which is stored into MySQL database. A session to be valid the following must be correct
IP Address must match the Ip which created the session
The agent data must also match (I can live without it for nodejs)
The idle time of the session must be less that 900 seconds.
In order for Nodejs to start communication it must first create a new connection to the API. After the connection is accepted, nodejs must keep track of the following data "received by the API"
CSFR token
Session Id
Http Cookie
In order for Nodejs to make a connection to the API it must pass a username, password, server name, port, and a station name. I have all the needed info stored into MySQL database and I can easily get that using PHP.
The challenge is that NodeJS have to take the PHP session, validates it, pull the API needed info from the database then establish connection to the API.
Questions
Can nodejs use the PHP session to validate the user? If so how?
How will can nodejs use the TCP connection to prevent me from overloading the server?

This is your arrangement:
[User browser] -> [PHP] -> [Node.js] -> [API]
When your user's browser sends a request to your PHP server the request includes a cookie - one of the values of this cookie is the session id which PHP then uses to look up the session. The session id acts like a password, PHP will assume that if you've got the session id then you are the original user that was issued that id.
When your PHP script communicates with Node it needs to pass along that session id as part of the request. In Node you just then need to do a lookup of your sessions table in MySQL for the corresponding session.
PHP session data is stored as the $_SESSION array serialised. To extract data from it you will need to unserialise it first. There are a number of libraries out that can provide this functionality (e.g. https://github.com/naholyr/js-php-unserialize, https://github.com/kvz/phpjs/blob/master/functions/var/unserialize.js). However if the session data is simple and conforms to a known format you could 'hand parse' the data.

Sending/Emitting a message to a single specific client using node.js asocket.io

I'm currently starting working with node.js and as a first little project have decided to code a little chat application. I am using socket.io and have followed the chat tutorial on http://socket.io/get-started/chat/ so far. Now I want to expand on the script, but I am stuck.
I fail to understand how (or if at all) socket.io is managing all connected clients on the server side. I can use the io.emit function to send a message to all connected users. But what if I want to send a message to one specific user?
I could of course "fake" it by sending the message to all clients and then on the client side check for an identifyer, etc. and only process the message if there is a match. But this would still send the message to everyone, and it could easily be circumvented by users with minimal JS knowledge.
The "filtering"/"targeting" needs to happen on the server side for security reasons. But I fail to see/understand how (or if at all) socket.io actually manages all client connections. Is there a way to get a list of all current connected clients? Can you interate over all connected clients in a loop? Can you assign custom identifiers/name to connections? And - as the practical application - can you then emit/send a message to a single connected user?

Every socket connection has a unique socket id assigned with it at the very first time when a new connection is established.You can emit the data using that particular id.
var socketId=socket.id;
socket.broadcast.to(socketId).emit("messageToReceiver",data);
Thus, you can send it to a particular person.
To get all the connected clients check this URL:
Socket.IO - how do I get a list of connected sockets/clients?

Send message from Node.js Server to Client via function call

I want to send messages from my server to my client when a function is called. Using the code from this answer messages can be successfully sent from Server to Client every second.
I am building an application that runs node in the background, ideally I would like to be able to click a button that will call a function in the node server.js file which takes a parameter and sends that message to the client. The function in question would look like this
function sendToClient(message) {
clients[0].emit('foo', msg);
}
This would send the passed in message to the first client. How can I go about this?
In terminal, after you run node server.js is there a way to call a function from the server file using terminal, this could be a possible solution if so.

The best way to send messages from server to client right now is using webSockets. The basic concept is this:
Client A loads web page from server B.
Client A runs some javascript that creates a webSocket connection to server B.
Server B accepts that webSocket connection and the socket stays open for the duration of the life of the web page.
Server B registers event handlers to handle incoming messages from the web page.
Client A registers event handlers to handle incoming messages from the server.
At any point in time, the server can proactively send data to the client page and it will receive that data.
At any point in time, the client may sent data to the server and it will receive that data.
A popular node.js library that makes webSocket support pretty easy is socket.io. It has both client and server support so you can use the same library for both ends of the connection. The socket.io library supports the .emit() method mentioned in your question for sending a message over an active webSocket connection.
You don't directly call functions from client to server. Instead, you send a message that triggers the server to run some particular code or vice versa. This is cooperative programming where the remote end has to be coded to support what you're asking it to do so you can send it a message and some optional data to go with the message and then it can receive that message and data and execute some code with that.
So, suppose you wanted the server to tell the client anytime a temperature changed so that the client could display in their web page the updated temperature (I actually have a Raspberry Pi node.js server that does exactly this). In this case, the client web page establishes a webSocket connection to the server when the page loads. Meanwhile, the server has its own process that is monitoring temperature changes. When it sees that the temperature has changed some meaningful amount, it sends a temperature change message to each connected client with the new temperature data. The client receives that message and data and then uses that to update it's UI to show the new temperature value.
The transaction could go the other way too. The client could have a matrix of information that it wants the server to carry out some complicated calculation on. It would send a message to the server with the type of calculation indicated in the message type and then send the matrix as the data for the message. The server would receive that message, see that this is a request to do a particular type of calculation on some data, it would then call the appropriate server-side function and pass it the client data. When the result was finished on the server, it would send a message back to the client with the result. The client would receive that result and then do whatever it needed to with the calculated result.
Note, if the transactions are only from client to server with a response then coming back from the server, a webSocket is not needed for that type of transaction. That can be done with just an Ajax call. Client makes ajax call to the server, server formulates a response and returns the response. Where webSockets are most uniquely useful is if you want to initiate the communication from the server and send unsolicited data to the client at a time that the server decides. For that, you need some continuous connection between client and server which is what a webSocket is designed to be.
It appears there may be more to your question about how to communicate from a C# server to your node.js server so it can then notify the client. If this is the case, then since the node.js server is already a web server, I'd just add a route to the node.js server so you can simply do an http request from the C# server to the node.js server to pass some data to the node.js server which it can then use to notify the appropriate client via the above-described webSocket connection. Depending upon your security needs, you may want to implement some level of security so that the http request can only be sent locally from your C# server, not from the outside world to your node.js server.

In order to send a command to a client via the console there are two options, single process or multiprocess:
Single Process
When the command is run from console, temporary socket.io server starts listening on a port.
Once the client connects, send the message to the client.
Disconnect and stop the console app.
The reason this works is that socket.io clients are always trying to connect to the server. As long as the browser is open, they will try to connect. So even if the server only comes on for a few seconds, it should connect and receive messages. If the client is not running then simply create a timeout that will stop the console app and inform the user that it failed to broadcast the command.
While this approach is very easy, it's not robust nor efficient. For small projects this would work, but you'll have better luck with the next approach:
Multi-Process
This approach is much more reliable, expandable, and just better looking when you are talking about architecture. Here's the basic summary:
Spin up a stand-alone server that connects with clients.
Create a very similar console node app that will send a message to the server to forward on to clients.
Console app completes but the main server stays up and running.
This technique is just interprocess communication. Luckily you already have Socket.IO on the primary server, so your console app just needs to be another socket.io client. Check out this answer on how to implement that.
The downside to this is that you must secure that socket communication. Maybe you can enforce it to just allow localhost connections, that way you need access to the server to send the run command message (you absolutely don't want web clients executing code on other web clients).
Overall it comes down to the needs of your project. If this is a quick little experiment you want to try out, then just do it single process. But if will be hosting an express server (other webservers are available) and need to be running anyways, then multi-process is the way to go!
Example
I've created a simple example of this process using only Socket.io. Instructions to run it all are in the readme.
Implementations
In order to have C# (app) -> Node.js (server) -> Browser (client) communication then I would do one of the following:
Use redis as a message queue (add items to the queue with the app, consume with the server, which sends commands to client).
Live on the wild side and merge your NodeJS and C# runtimes with Edge.js. If you can run NodeJS from C# you will be able to send messages from your app to the server (messages are then handled by the server, just like any other socket.io client-server model).
Easier, but still kinda hacky, use System.Diagnostics.Process to start a console tool explained in the Multi-Process section. This would simply run the process with arbitrary parameters. Not very robust but worth considering, simple means harder to break (And again, messages are then handled by the server, just like any other socket.io client-server model).

I would create a route for sending the message and send message from post parameter. From CLI you can use curl or from anywhere really:
app.get('/create', function(req, res) {
if( data.type && data.content && data.listeners){
notify( data );
}
});
var notify = function( notification ){
ns_mynamespace.in(notification.listeners.users)
.emit("notification", {
id: notification.id,
title: 'hello', text: notification.content });
}
}