Develop Reference

Test Execution Error in Lambda RDS Data API for Javascript

I am trying to connect the AWS Lambda with SQL query to an AWS RDS (MySQL) using the Data API and return the query result for a user with a particular id.
This is how the handler looks like:
'use strict';
const AWS = require('aws-sdk')
const RDS = new AWS.RDSDataService({ endpoint: '******.cluster-*********.us-west-2.rds.amazonaws.com' })
module.exports.fetchById = async (event, context, callback) => {
const req_id = event.pathParameters.id;
try {
const params = {
resourceArn: 'arn:aws:rds:us-west-2:***********',
secretArn: 'arn:aws:secretsmanager********',
sql: `SELECT * FROM user WHERE user_id = :id`,
database: '*********',
includeResultMetadata: true,
parameters: [
{ id: req_id },
]
}
const db_res = await rdsDataService.executeStatement(params).promise();
const response = {
body: JSON.stringify({
message: 'Data fetched!!',
data: db_res.records
})
};
callback(null, response);
} catch (error) {
console.log('Error Received', error)
}
};
serverless.yml
functions:
fetchByIdId:
handler: handler.fetchById
events:
- http:
path: user/{id}
method: get
authorizer:
name: cognito-authorizer
arn: arn:aws:***********
request:
parameters:
paths:
id: true
Few issues that I need to work upon:
If I instantiate like:
const RDS = new AWS.RDSDataService({ endpoint: '******.cluster-*********.us-west-2.rds.amazonaws.com' })
by including an endpoint cluster as a parameter, the handler function does not execute at all. It just keeps throwing:
{"errorMessage": "2020-10-30T07:31:12.258Z c4b4ca2d-3cbb-4733-8cfe-0c7aad228c29 Task timed out after 6.01 seconds"}.
Tried increasing the timeout also but it didn't made any difference & the error still perists.
But if endpoint is removed & only used like:
const RDS = new AWS.RDSDataService()
, the function does not throw timeout error, but these two new issues are faced:
The id is required. I passed the required config to the yml file, but it
doesn't seem to mark it as required. If the http endpoint is executed as
/user/, it does not throw any error.
I need to perform input data validation/sanitization for the request
parameters. On executing the endpoint /user/123, it throws an error:
INFO Error Received UnexpectedParameter: Unexpected key 'id' found in params.parameters[0].
I read out in the documentation but could not find any particular clue to complete the same.
Any help to resolve this is appreciated.

How do I listen for new uploads from a specific channel in the YouTube API?

I am making a Discord bot, and I want it to be able to use the YouTube API to fetch new uploads from a specific channel.
I have searched elsewhere, but they all say how to upload videos, not how to track uploads.
Is this possible, and how can I do it?
Edit: Tried PubSubHubbub but it was very confusing and I couldn't get it to work

Here an example built on top of Node.js (v12) and Fastify and published with ngrok:
I wrote some comments explaining what it is happening:
const fastify = require('fastify')({ logger: true })
const xmlParser = require('fast-xml-parser')
const { URLSearchParams } = require('url')
const fetch = require('node-fetch')
// add an xml parser
fastify.addContentTypeParser('application/atom+xml', { parseAs: 'string' }, function (req, xmlString, done) {
try {
const body = xmlParser.parse(xmlString, {
attributeNamePrefix: '',
ignoreAttributes: false
})
done(null, body)
} catch (error) {
done(error)
}
})
// this endpoint needs for authentication
fastify.get('/', (request, reply) => {
reply.send(request.query['hub.challenge'])
})
// this endpoint will get the updates
fastify.post('/', (request, reply) => {
console.log(JSON.stringify(request.body, null, 2))
reply.code(204)
reply.send('ok')
})
fastify.listen(8080)
.then(() => {
// after the server has started, subscribe to the hub
// Parameter list: https://pubsubhubbub.github.io/PubSubHubbub/pubsubhubbub-core-0.4.html#rfc.section.5.1
const params = new URLSearchParams()
params.append('hub.callback', 'https://1f3dd0c63e78.ngrok.io') // you must have a public endpoint. get it with "ngrok http 8080"
params.append('hub.mode', 'subscribe')
params.append('hub.topic', 'https://www.youtube.com/xml/feeds/videos.xml?channel_id=UCfWbGF64qBSVM2Wq9fwrfrg')
params.append('hub.lease_seconds', '')
params.append('hub.secret', '')
params.append('hub.verify', 'sync')
params.append('hub.verify_token', '')
return fetch('https://pubsubhubbub.appspot.com/subscribe', {
headers: { 'content-type': 'application/x-www-form-urlencoded' },
body: params,
method: 'POST'
})
})
.then((res) => {
console.log(`The status must be 204. Received ${res.status}`)
// shows the error if something went wrong
if (res.status !== 204) {
return res.text().then(txt => console.log(txt))
}
})
I used my channel id to do some testing, consider that the notification is not in real-time, the POSTs are triggered after several minutes usually.

What extra fields does the AWS Amplify API include when invoking a Lambda function?

I have this AWS Lambda function to create a note object in my DynamoDB table:
import * as uuid from "uuid";
import AWS from "aws-sdk";
const dynamoDb = new AWS.DynamoDB.DocumentClient();
export function main(event, context, callback) {
// Request body is passed in as a JSON encoded string in 'event.body'
const data = JSON.parse(event.body);
const params = {
TableName: process.env.tableName,
// 'Item' contains the attributes of the item to be created
// - 'userId': user identities are federated through the
// Cognito Identity Pool, we will use the identity id
// as the user id of the authenticated user
// - 'noteId': a unique uuid
// - 'content': parsed from request body
// - 'attachment': parsed from request body
// - 'createdAt': current Unix timestamp
Item: {
userId: event.requestContext.identity.cognitoIdentityId,
noteId: uuid.v1(),
content: data.content,
attachment: data.attachment,
createdAt: Date.now()
}
};
dynamoDb.put(params, (error, data) => {
// Set response headers to enable CORS (Cross-Origin Resource Sharing)
const headers = {
"Access-Control-Allow-Origin": "*",
"Access-Control-Allow-Credentials": true
};
// Return status code 500 on error
if (error) {
const response = {
statusCode: 500,
headers: headers,
body: JSON.stringify({ status: false })
};
callback(null, response);
return;
}
// Return status code 200 and the newly created item
const response = {
statusCode: 200,
headers: headers,
body: JSON.stringify(params.Item)
};
callback(null, response);
});
}
What it does is not really relevant to the question. The important thing to note here is that it can be successfully executed offline with the command serverless invoke local --function create --path mocks/create-event.json and an example event create-event.json:
{
"body": "{\"content\":\"hello world\",\"attachment\":\"hello.jpg\"}",
"requestContext": {
"identity": {
"cognitoIdentityId": "USER-SUB-1234"
}
}
}
However, when I invoke this Lambda function with a POST request through the aws-amplify API, I only define a body field in the init object, i.e.
import { API } from "aws-amplify";
...
function createNote(note) {
return API.post("scratch-notes", "/scratch-notes", {
body: note
});
}
Which leads to the following questions...
How did the Lambda function get the needed requestContext field
from the aws-amplify API POST method that invoked it?
Was it appended to the init object by the aws-amplify API?
That would be the obvious answer, but that leads me to another question...
What other fields are appended to the init object by the aws-amplify API?

The event object also contains two extra structures, afaik: a requestContext object and a pathParameters object.
The requestContext object is just the context object included in the event object for testing purposes and for our general convenience.
The pathParameters object is a list of fields that are extracted from paths with special tokens.
get:
handler: get.main
events:
- http:
path: scratch-notes/{id}
method: get
cors: true
authorizer: aws_iam
A handler like that would take a request with the URI scratch-notes/1234 and return a pathParameters object as follows:
{
"pathParameters": {
"id": "1234"
}
}
Of course, both objects can be combined to invoke our API offline with the Serverless framework which would look like this:
{
"pathParameters": {
"id": "59747e00-8e61-11ea-8cb8-5d9bcedbe6f4"
},
"body": "myBody",
"requestContext": {
"identity": {
"cognitoIdentityId": "USER-SUB-1234"
}
}
}

How to programmatically connect to AWS websocket API Gateway

I am trying to implement messaging between users on my website by leveraging AWS's websocket api gateway. Every guide/documentation that I look at says to use wscat to test the connection to the gateway. I am at the point where I can connect to the api gateway and send messages between clients using wscat but am struggling to get it working programmatically from my ts code.
What I want to do is make an api call to the websocket api gateway once the user logs in so they can send messages at any point. I am using serverless for my backend and Angular 6 for the front end. I read that I need to make a POST request to https://{api-id}.execute-api.us-east-1.amazonaws.com/{stage}/#connections/{connection_id} to send messages through a websocket connection but i'm having trouble using typescript in a service I created to connect/get a connection id.
I am making a second API call after the user successfully logs in to open a connection to the websocket api gateway. I tried calling a function that makes a post request with no body (not sure what I would send in the body of the request since I've only connected to it using the wscat tool) to the URL I get after deploying my serverless code. I also tried making a POST request to the https:// URL I see in the AWS console after manually deploying the API gateway.
base.service.ts
protected getBaseSocketEndpoint(): string {
// 'wss://xxxxxxx.execute-api.us-east-1.amazonaws.com/dev' <-- tried this too
return 'https://xxxxxxxx.execute-api.us-east-1.amazonaws.com/dev/#connections';
}
authentication.service.ts
this.authService.login(username, password).pipe(first()).subscribe(
(response) => {
console.log(response);
this.authService.setCookie('userId', response.idToken.payload.sub);
this.authService.setCookie('jwtToken', response.idToken.jwtToken);
this.authService.setCookie('userEmail', response.idToken.payload.email);
this.authService.setCookie('refreshToken', response.refreshToken.token);
this.toastr.success('Login successful. Redirecting to your dashboard.', 'Success!', {
timeOut: 1500
});
this.authService.connectToWebSocket(response.idToken.payload.sub).pipe(first()).subscribe(
response => {
console.log(response);
}
);
this.routerService.routeToUserDashboard();
},
(error) => {
// const errorMessage = JSON.parse(error._body).message;
this.toastr.error("Incorrect username and password combination.", 'Error!', {
timeOut: 1500
});
}
);
authentication.service.ts extends BaseService
public connectToWebSocket(userId: string): Observable<any> {
const body = {
userId: userId
};
console.log('calling connectToWebSocket()..');
return this.http.post(this.getBaseSocketEndpoint(), body).pipe(map(response => {
console.log(response);
}));
}
serverless.yaml
functions:
connectionHandler:
handler: connectionHandler.connectionHandler
events:
- websocket:
route: $connect
cors: true
- websocket:
route: $disconnect
cors: true
defaultHandler:
handler: connectionHandler.defaultHandler
events:
- websocket:
route: $default
cors: true
sendMessageHandler:
handler: messageHandler.sendMessageHandler
events:
- websocket:
route: sendMessage
cors: true
connectionHandler.js (lambda)
const success = {
statusCode: 200,
headers: { "Access-Control-Allow-Origin": "*" },
body: "everything is alright"
};
module.exports.connectionHandler = (event, context, callback) => {
var connectionId = event.requestContext.connectionId;
if (event.requestContext.eventType === "CONNECT") {
addConnection(
connectionId,
"b72656eb-db8e-4f32-a6b5-bde4943109ef",
callback
)
.then(() => {
console.log("Connected!");
callback(null, success);
})
.catch(err => {
callback(null, JSON.stringify(err));
});
} else if (event.requestContext.eventType === "DISCONNECT") {
deleteConnection(
connectionId,
"b72656eb-db8e-4f32-a6b5-bde4943109ef",
callback
)
.then(() => {
console.log("Disconnected!");
callback(null, success);
})
.catch(err => {
callback(null, {
statusCode: 500,
body: "Failed to connect: " + JSON.stringify(err)
});
});
}
};
// THIS ONE DOESNT DO ANYHTING
module.exports.defaultHandler = (event, context, callback) => {
callback(null, {
statusCode: 200,
body: "default handler was called."
});
};
const addConnection = (connectionId, userId, callback) => {
const params = {
TableName: CHATCONNECTION_TABLE,
Item: {
connectionId: connectionId,
userId: userId
}
};
var response;
return dynamo
.put(params, function(err, data) {
if (err) {
errorHandler.respond(err, callback);
return;
} else {
response = {
statusCode: 200,
headers: { "Access-Control-Allow-Origin": "*" },
body: JSON.stringify(data)
};
callback(null, response);
}
})
.promise();
};
const deleteConnection = (connectionId, userId, callback) => {
const params = {
TableName: CHATCONNECTION_TABLE,
Key: {
connectionId: connectionId,
userId: userId
}
};
var response;
return dynamo
.delete(params, function(err, data) {
if (err) {
errorHandler.respond(err, callback);
return;
} else {
response = {
statusCode: 200,
headers: { "Access-Control-Allow-Origin": "*" },
body: JSON.stringify(data)
};
callback(null, response);
}
})
.promise();
};
Expected: trigger POST api call and open a persistent connection with the Websocket API Gateway.
Actual: unable to connect via API call above. I get a 403 in the console with the message:
Access to XMLHttpRequest at 'https://xxxxxxx.execute-api.us-east-1.amazonaws.com/dev/#connections' from origin 'http://localhost:4200' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Not sure why im getting a CORS error when I have CORS enabled in my serverless file.

I had a similar problem. You can use all the socket.io-client goodness. But you have to set the option transports to :
let socket = io(url, {
reconnectionDelayMax: 1000,
transports: ["websocket"],
});
The default is
transports: ["polling", "websocket"],
So your app will kick off polling with the resulting in a CORS error. It's not that clear in the docs, but here is a useful link.
Look under "Available options for the underlying Engine.IO client:".

I had the same problem and finally figured out, that normally there should not be such a CORS error message with websockets:
Why is there no same-origin policy for WebSockets? Why can I connect to ws://localhost?
Skipping the client library "socket.io" and using "vanilla websockets" helped me out.
In your case I would check the libraries behind "connectToWebSocket".

I used python lambda handler, so it might be helpful to many.
I have implemented a websocket which sends user some message 5 times with a gap
serverless.yml
service: realtime-websocket-test
provider:
name: aws
stage: ${opt:stage, 'dev'}
runtime: python3.8
region: ${opt:region, 'ap-south-1'}
memorySize: 128
timeout: 300
functions:
connect:
handler: handler.lambda_handler
events:
- websocket:
route: $connect
- websocket:
route: $disconnect
- websocket:
route: $default
handler.py
import time
import json
import boto3
def lambda_handler(event, context):
print(event)
event_type = event["requestContext"]["eventType"]
if event_type == "CONNECT" or event_type == "DISCONNECT":
response = {'statusCode': 200}
return response
elif event_type == "MESSAGE":
connection_id = event["requestContext"]["connectionId"]
domain_name = event["requestContext"]["domainName"]
stage = event["requestContext"]["stage"]
message = f'{domain_name}: {connection_id}'.encode('utf-8')
api_client = boto3.client('apigatewaymanagementapi', endpoint_url = f"https://{domain_name}/{stage}")
for _ in range(5):
api_client.post_to_connection(Data=message,
ConnectionId=connection_id)
time.sleep(5)
response = {'statusCode': 200}
return response

Cycle.js - Driver - PhoenixJS (Websockets)

We currently have a VueJS application and I am looking at migrating it to Cycle.js (first major project).
I understand in Cycle.JS we have SI and SO for drivers (using adapt()); naturally a WebSocket implementation fits this as it has both read and write effects.
We use Phoenix (Elixir) as our backend using Channels for soft real-time communication. Our client-side WS library is Phoenix herehttps://www.npmjs.com/package/phoenix.
The example on Cycle.js.org is perfect if you know how to connect.
In our case, we authenticate using a REST endpoint which returns a token (JWT) which is used to initialize the WebSocket (token parameter). This token cannot simply be passed into the driver, as the driver is initialized when the Cycle.js application runs.
An example (not actual code) of what we have now (in our VueJS application):
// Code ommited for brevity
socketHandler = new vueInstance.$phoenix.Socket(FQDN, {
_token: token
});
socketHandler.onOpen(() => VueBus.$emit('SOCKET_OPEN'));
//...... Vue component (example)
VueBus.$on('SOCKET_OPEN', function () {
let chan = VueStore.socketHandler.channel('PRIV_CHANNEL', {
_token: token
});
chan.join()
.receive('ok', () => {
//... code
})
})
The above is an example, we have a Vuex store for a global state (socket etc), centralized message bus (Vue app) for communicating between components and channel setups which come from the instantiated Phoenix Socket.
Our channel setup relies on an authenticated Socket connection which needs authentication itself to join that particular channel.
The question is, is this even possible with Cycle.js?
Initialize WebSocket connection with token parameters from a REST call (JWT Token response) - we have implemented this partially
Create channels based off that socket and token (channel streams off a driver?)
Accessing multiple channel streams (I am assuming it may work like sources.HTTP.select(CATEGORY))
We have a 1: N dependency here which I am not sure is possible with drivers.
Thank you in advance,
Update# 17/12/2018
Essentially what I am trying to imitate is the following (from Cycle.js.org):
The driver takes a sink in, in order to perform write effects (sending messages on a specific channels) but also may return a source; this means there are two streams which are async? Which means that creating the socket at runtime may lead to one stream accessing the "socket" before it is instanitated; please see comments in the snippet below.
import {adapt} from '#cycle/run/lib/adapt';
function makeSockDriver(peerId) {
// This socket may be created at an unknown period
//let socket = new Sock(peerId);
let socket = undefined;
// Sending is perfect
function sockDriver(sink$) {
sink$.addListener({
next: listener => {
sink$.addListener({
next: ({ channel, data }) => {
if(channel === 'OPEN_SOCKET' && socket === null) {
token = data;
// Initialising the socket
socket = new phoenix.Socket(FQDN, { token });
socketHandler.onOpen(() => listener.next({
channel: 'SOCKET_OPEN'
}));
} else {
if(channels[channel] === undefined) {
channels[channel] = new Channel(channel, { token });
}
channels[channel].join()
.receive('ok', () => {
sendData(data);
});
}
}
});
},
error: () => {},
complete: () => {},
});
const source$ = xs.create({
start: listener => {
sock.onReceive(function (msg) {
// There is no guarantee that "socket" is defined here, as this may fire before the socket is actually created
socket.on('some_event'); // undefined
// This works however because a call has been placed back onto the browser stack which probably gives the other blocking thread chance to write to the local stack variable "socket". But this is far from ideal
setTimeout(() => socket.on('some_event'));
});
},
stop: () => {},
});
return adapt(source$);
}
return sockDriver;
}
Jan van Brügge, the soluton you provided is perfect (thank you) except I am having trouble with the response part. Please see above example.
For example, what I am trying to achieve is something like this:
// login component
return {
DOM: ...
WS: xs.of({
channel: "OPEN_CHANNEL",
data: {
_token: 'Bearer 123'
}
})
}
//////////////////////////////////////
// Some authenticated component
// Intent
const intent$ = sources.WS.select(CHANNEL_NAME).startWith(null)
// Model
const model$ = intent$.map(resp => {
if (resp.some_response !== undefined) {
return {...}; // some model
}
return resp;
})
return {
DOM: model$.map(resp => {
// Use response from websocket to create UI of some sort
})
}

first of all, yes this is possible with a driver, and my suggestion will result in a driver that feels quite like the HTTP driver.
First of all to have some rough pseudo code that where I can explain everything, I might have misunderstood parts of your question so this might be wrong.
interface WebsocketMessage {
channel: string;
data: any;
}
function makeWebSocketDriver() {
let socket = null;
let token = null;
let channels = {}
return function websocketDriver(sink$: Stream<WebsocketMessage> {
return xs.create({
start: listener => {
sink$.addListener({
next: ({ channel, data }) => {
if(channel === 'OPEN_SOCKET' && socket === null) {
token = data;
socket = new phoenix.Socket(FQDN, { token });
socketHandler.onOpen(() => listener.next({
channel: 'SOCKET_OPEN'
}));
} else {
if(channels[channel] === undefined) {
channels[channel] = new Channel(channel, { token });
}
channels[channel].join()
.receive('ok', () => {
sendData(data);
});
}
}
});
}
});
};
}
This would be the rough structure of such a driver. You see it waits for a message with the token and then opens the socket. It also keeps track of the open channels and sends/receives in those based on the category of the message. This method just requires that all channels have unique names, I am not sure how your channel protocol works in that regard or what you want in particular.
I hope this enough to get you started, if you clarify the API of the channel send/receive and the socket, I might be able to help more. You are also always welcome to ask questions in our gitter channel