Overview
I am using the Node.js library actions-on-google client to build a smarthome action for the Garage Door device type. This action is deployed as a Cloud Function in GCP. I can confirm that the following works perfectly so far:
Accounting linking with our OAuth flow
Responding to sync intents (ie. onSync() in the client)
Responding to execute intents (ie. onExecute() in the client)
Problem
Despite that other callbacks (onSync() and onExecute()) work fine, we do not see any evidence of onQuery() being called no matter what. There aren't any errors showing in Stackdriver nor are their any logs being generated in Stackdriver under the "Google Actions" filter either.
We expect onQuery() to run when we ask Google Assistant things like is the garage door open? and is Matt's Door closed?
We tried removing async to see if the call was hanging
We tried removing headers in the lambda
We tried removing all other code and redploying to isolate onQuery()
Code
The following code shows the simple onQuery() callback. onExecute() and onSync code has been removed for clarity.
'use strict';
const functions = require('firebase-functions');
const {smarthome} = require('actions-on-google');
const app = smarthome({
jwt: require('./XXXXXXXXX-XXXXXXXXXX.json'),
debug: true,
});
//
// Note: Removed onSync() and onExecute() for clarity
//
app.onQuery(async (body, headers) => {
// Expecting to see these logging statements in
// Stackdriver like we do for onExecute() and
// onSync() ... but nothing ever shows up.
console.info('=== onQuery.body', body);
console.info('=== onQuery.headers', headers);
// We have hardcoded the following ID and a "closed"
// state. It matches a valid device ID to the testing
// account we are using.
return {
requestId: body.requestId,
payload: {
devices: {
'2489e4a92799728292b8d5a8b1c9d177': {
on: true,
online: true,
openPercent: 0,
}
}
}
};
});
exports.smarthome = functions.https.onRequest(app);
We considered the possibility that the JSON returned in the call to onSync() might be missing a trait or something that prevents it from responding to a query intent properly but we have not been able to identify anything that might be incorrect or missing. Here is the JSON payload returned from onSync():
{
"requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
"payload": {
"agentUserId": "1836.15267389",
"devices": [{
"id": "1234",
"type": "action.devices.types.GARAGE",
"traits": [
"action.devices.traits.OpenClose"
],
"name": {
"defaultNames": ["Smart Garage Door"],
"name": "Matt's Door",
"nicknames": ["Matt's Door"]
},
"willReportState": true,
"attributes": {
"openDirection": [
"UP",
"DOWN"
]
},
"deviceInfo": {
"manufacturer": "ABC Corp",
"model": "test",
"hwVersion": "1.0",
"swVersion": "1.0"
}
}]
}
}
Expected Result
We expect Google Assistant to respond with "Garage door is closed" or some other equivalent. Instead, we receive "Sorry, I can't reach Matt's Door right now. Please try again."
Answering my own question here in case anyone else has this trouble. The reason why my action was handling the SYNC and EXECUTE intents but not the QUERY intent came down to what default/user names I assigned each device in the SYNC response.
Ultimately, I started using the following and my action began responding to QUERY intents as expected again:
...
name: {
defaultNames: ['Garage Door'],
name: door.name,
nicknames: [door.name, 'Garage Door']
},
...
where door.name is a name which is set by the user and is returned by an API call.
Related
I have an application in Electron that does facial recognition of people to then decide whether or not they can enter the place and for that I'm using Amazon Rekognition.
Everything was working fine (for a few months) until, two days ago, a customer reported to me that the app was behaving strangely, like it wasn't responding to requests for facial recognition.
After several tests, I discovered that what is happening with it is a timeout error, which occurs in all API calls, whether they are looking for faces (SearchFacesByImage) or registering new faces (IndexFaces).
The error says:
{
"message": "connect ETIMEDOUT 3.226.60.54:443",
"errno": -4039,
"code": "TimeoutError",
"syscall": "connect",
"address": "3.226.60.54",
"port": 443,
"time": "2022-12-14T13:50:10.909Z",
"region": "us-east-1",
"hostname": "rekognition.us-east-1.amazonaws.com",
"retryable": true
}
What intrigued me was the fact that everything was working fine, until this behavior just started happening (and I didn't make any code changes/updates to the app running on my client's computer).
And what makes me even more intrigued is that this behavior occurs completely randomly and only on the machine of that client in question. Sometimes the API calls work correctly (returning whether the person was recognized or not), but most of the time, the calls take about 90 seconds to return the timeout error. When executing the same code on my machine (same methods and same CollectionId) everything runs normally and there was no timeout error at any time - while at the exact same moment on my client's machine the behavior continues.
I was using aws-sdk and then switched to #aws-sdk/client-rekognition (thinking that could solve the problem) but the code only worked on a few of the first calls to the API and a few minutes later it got the timeout errors again.
The code I'm using to configure and make calls to Rekognition is basically this:
const { RekognitionClient, IndexFacesCommand, SearchFacesByImageCommand } = require('#aws-sdk/client-rekognition')
const rekognitionClient = new RekognitionClient({
credentials: {
accessKeyId: 'accessKeyId',
secretAccessKey: 'secretAccessKey'
},
region: 'us-east-1'
})
const registerFaceOnRekognition = async (bytes, userId) => {
const params = {
CollectionId: 'collectionId',
Image: { Bytes: bytes },
ExternalImageId: userId,
MaxFaces: 1,
QualityFilter: 'HIGH'
}
const command = new IndexFacesCommand(params)
try {
const { FaceRecords } = await rekognitionClient.send(command)
if (!FaceRecords.length) {
console.log('No faces detected.')
return
}
console.log('Face created:')
console.log(FaceRecords[0].Face.FaceId)
} catch (error) {
console.error(error) // timeout error
}
}
const searchFaceByImageOnRekognition = async (bytes) => {
const params = {
CollectionId: 'collectionId',
Image: { Bytes: bytes },
MaxFaces: 1,
FaceMatchThreshold: 99,
QualityFilter: 'HIGH'
}
const command = new SearchFacesByImageCommand(params)
try {
const { FaceMatches } = await rekognitionClient.send(command)
if (!FaceMatches.length) {
console.log('This face has not been registered yet')
return
}
console.log('Face found:')
console.log(FaceMatches[0].Face.ExternalImageId)
} catch (error) {
console.error(error) // timeout error
}
}
// Method called through the renderer process that has a canvas where the webcam view is reproduced
const onTakePicture = (event, data) => {
const bytes = Buffer.from(data.dataURL.replace('data:image/jpeg;base64,', ''), 'base64')
// If there is a userId, register the face in the image
if (data.userId) {
registerFaceOnRekognition(bytes, data.userId)
return
}
// Else, search for the face in the image
searchFaceByImageOnRekognition(bytes)
}
Just remembering that: during all tests on my client's computer the internet connection was stable and working properly.
What is the best way to investigate and resolve this issue?
UPDATE:
I enabled Rekognition debug logs and they can be found at: https://gist.github.com/IgorSamer/4e58e09f3fa615401f85ca325b794245
In it, the first three requests (2022-12-16T13:48:45.932Z, 2022-12-16T13:53:20.325Z and 2022-12-16T14:19:12.479Z) occur normally. However, all other consecutive requests start to give the timeout error, where, in fact, no data is returned after the [DEBUG] App: endpoints Resolved endpoint: step.
As previously mentioned the internet connection is working fine. I could also managing to reproduce the error via remote access, that is, the machine internet was ok at the time of error.
Is there a possibility that there is a block made by my client's firewall/network that prevents requests from being sent by the SDK after a few successful requests? If yes, what is the best way to investigate this?
Exploration
This is what I would do initially to gather some info:
Verify if this is happening ALL the time with that specific client.
Verify if this is happening ONLY with one client, or more.
Verify if this is happening in one or multiple regions (i.e us-east-1).
Verify if Amazon Recognition has had/or has issues in the affected region during the time window of interest.
Check Recognition's status in the Health dashboard in your AWS console: link
Use AWS Recognition Guidelines and Quotas as a reference to determine if your app/service usage of Recognition is under the set limits.
Note there's a limit on TPS per resource (i.e SearchFacesByImage, IndexFaces) per account.
Possible approaches
Verify if there was a change in the client network/firewall. Just ask.
Replicate your app's API call with AWS CLI and study logs.
Access remotely to your client's device.
Setup temporal AWS credentials (remember to remove access after the test)
Send an API call to the Recognition endpoint. Note that even a 4XX error will be good news, as you got at least some response.
Set up proper logging for your app (as CloudWatch logs may not be enough to troubleshoot).
Check Splunk's APM and NewRelic's APM
I hope this may be of help to at least create a troubleshooting strategy
I am working with react and I have to give my user login through Azure B2C, So I am trying to do that but I am not able to find out how to do that and what is the.
What I have tried
I got this example from Microsoft site which is done using plain JavaScript (vanilla), I have no idea how I will implement this in my react code.
So I tried to move with some react library, I google around and found This library
I have followed the same code they have written, but when I hit login button it takes me to login page of azure, So in my app.js I am doing console.log(authentication.getAccessToken()); after login it throws null, I don't know why
My code
authentication.initialize({
// optional, will default to this
instance: 'https://login.microsoftonline.com/tfp',
// My B2C tenant
tenant: 'mytenant.onmicrosoft.com',
// the policy to use to sign in, can also be a sign up or sign in policy
signInPolicy: 'B2c_signupsignin',
// the the B2C application you want to authenticate with (that's just a random GUID - get yours from the portal)
clientId: 'fdfsds5-5222-ss522-a659-ada22',
// where MSAL will store state - localStorage or sessionStorage
cacheLocation: 'sessionStorage',
// the scopes you want included in the access token
scopes: ['https://mytenant.onmicrosoft.com/api/test.read'],
// optional, the redirect URI - if not specified MSAL will pick up the location from window.href
redirectUri: 'http://localhost:3000',
});
And then on click of login I am doing this
const Log_in = () => {
authentication.run(() => {});
};
in my app.js I am doing like below
import authentication from 'react-azure-b2c';
function App() {
console.log(authentication.getAccessToken());
}
So initially it is showing null which is fine, but after login also it is throwing error only.
So I was not able to resolve this, that's why I move to the other library which is almost similar to this
This one
So here when I click on login button I am getting error as
The example I got from Microsoft with valina Javascript, I think that is the perfect way to do but How can I imliment that through react I don't know
This the code with vanilla js
I have been stuch here from long time i don't know what to do now, not able to find good example on google to implement it with react
PS: I am using react hooks functional component to write my code, please guide me through this
I just want to implement this using react in a proper way, I know out tehre so many peoples who are already using this, so I just want to see a good example.
Edit / update
I tried doing like this
b2cauth.initialize({
instance: 'https://mylogin.b2clogin.com/tfp',
tenant: 'mylogin.b2clogin.com',
signInPolicy: 'B2C_1_SigninSignupUsername',
clientId: 'fc3081ec-504a-4be3-a659-951a9408e248',
cacheLocation: 'sessionStorage',
scopes: ['https://mylogin.b2clogin.com/api/demo.read'],
redirectUri: 'http://localhost:3000',
});
b2cauth.run(() => {
ReactDOM.render(<App />, document.getElementById('root'));
serviceWorker.unregister();
});
I check Microsoft link pasted as answer, and changed instance:
instance: 'https://mylogin.b2clogin.com/tfp',
to
instance:'https://my-tanent-name.b2clogin.com/tenent-id/oauth2/authresp',
but I am getting error as bad request
and I check network tab and I check the url it is hitting and it is hitting below
https://login.microsoftonline.com/common/discovery/instance?api-version=1.0&authorization_endpoint=https://my-tenatnt-name.b2clogin.com/tenant-id/oauth2/authrespmy-tenant-name.b2clogin.com/b2c_1_signinsignupusername/oauth2/v2.0/authorize
I tried removing https from instance and hit it like this
//mytenant.b2clogin.com/tenant-id/oauth2/authresp
it throws error as Uncaught AuthorityUriInsecure
I think it is going to wrong place
Your coordinates for the b2c instance are not correct (see note). You can find more details: https://learn.microsoft.com/en-us/azure/active-directory-b2c/b2clogin
If you like you can use this sample, which shows how to use B2C policy from React application using oidc-client.js library. By default it is configured to use PKCE but you can configure it to use implicit flow instead if needed (not recommended).
Complete instructions provided in the git repo but here is the high level overview.
You need to first create application in b2c along with the policy (not shown). You should add two redirect uris -- https://localhost:3000 and https://localhost:3000/callback.html
You can also add permissions in case you like to receive an access_token in addition to the id_token.
Your manifest should look similar to:
{
"id": "443ca8db-7bd1-4ebd-9671-ce94e006a18a",
"acceptMappedClaims": null,
"accessTokenAcceptedVersion": 2,
"addIns": [],
"allowPublicClient": null,
"appId": "50d2c416-a5ad-4c5c-b36a-0d1ac5b48167",
"appRoles": [],
"oauth2AllowUrlPathMatching": false,
"createdDateTime": "2020-09-02T00:11:35Z",
"groupMembershipClaims": null,
"identifierUris": [],
"informationalUrls": {
"termsOfService": null,
"support": null,
"privacy": null,
"marketing": null
},
"keyCredentials": [],
"knownClientApplications": [],
"logoUrl": null,
"logoutUrl": null,
"name": "OIDC-Test-APP",
"oauth2AllowIdTokenImplicitFlow": false,
"oauth2AllowImplicitFlow": false,
"oauth2Permissions": [],
"oauth2RequirePostResponse": false,
"optionalClaims": null,
"orgRestrictions": [],
"parentalControlSettings": {
"countriesBlockedForMinors": [],
"legalAgeGroupRule": "Allow"
},
"passwordCredentials": [],
"preAuthorizedApplications": [],
"publisherDomain": "contoso.onmicrosoft.com",
"replyUrlsWithType": [
{
"url": "http://localhost:3000/signin-callback.html",
"type": "Spa"
},
{
"url": "http://localhost:3000/",
"type": "Spa"
}
],
"requiredResourceAccess": [
{
"resourceAppId": "00000003-0000-0000-c000-000000000000",
"resourceAccess": [
{
"id": "37f7f235-527c-4136-accd-4a02d197296e",
"type": "Scope"
},
{
"id": "7427e0e9-2fba-42fe-b0c0-848c9e6a8182",
"type": "Scope"
}
]
},
{
"resourceAppId": "18ac2afe-2c1f-4ea8-8d63-14dd50ee4f85",
"resourceAccess": [
{
"id": "d5515006-5646-4398-ad59-fffc357f3423",
"type": "Scope"
}
]
}
],
"samlMetadataUrl": null,
"signInUrl": null,
"signInAudience": "AzureADandPersonalMicrosoftAccount",
"tags": [],
"tokenEncryptionKeyId": null
}
Clone the repo and update the settings present inside AuthSettings.ts to match your tenant. You must update client_id and contoso which is the tenant name.
const settings = {
// This is the metadata endpoint
authority: 'https://contoso.b2clogin.com/contoso.onmicrosoft.com/v2.0/.well-known/openid-configuration?p=B2C_1A_signup_signin',
// Turn off calls to user info since CORS will block it
loadUserInfo: false,
// The URL where the Web UI receives the login result
redirect_uri: 'http://localhost:3000/signin-callback.html',
// The no longer recommended implicit flow must be used if CORS is disabled
// If you want to use impicit flow use id_token instead of code for the return type.
response_type: 'code',
// Other OAuth settings
client_id: '18ac2afe-2c1f-4ea8-8d63-14dd50ee4f85',
// openid is required, remove https://contoso.onmicrosoft.com/test/Read if access_token is not required.
scope: 'openid https://contoso.onmicrosoft.com/test/Read',
// Supply these details explicitly. Directly copied from azure ad b2c policy metadata endpoint.
metadata: {
issuer: 'https://contoso.b2clogin.com/9859cd0c-9d99-4683-abcc-87462f67a0bc/v2.0/',
authorization_endpoint: 'https://contoso.b2clogin.com/contoso.onmicrosoft.com/oauth2/v2.0/authorize?p=b2c_1a_signup_signin',
token_endpoint: 'https://contoso.b2clogin.com/contoso.onmicrosoft.com/oauth2/v2.0/token?p=b2c_1a_signup_signin',
jwks_uri : 'https://contoso.b2clogin.com/contoso.onmicrosoft.com/discovery/v2.0/keys?p=b2c_1a_signup_signin',
end_session_endpoint: "https://contoso.b2clogin.com/contoso.onmicrosoft.com/oauth2/v2.0/logout?p=b2c_1a_signup_signin&post_logout_redirect_uri=http%3A%2F%2Flocalhost:3000%2F"
},
} as UserManagerSettings;
Build and run the app using yarn or npm
Application will launch by default on http://localhost:3000
Click Login and it should take you to the b2c policy to complete the journey.
After you complete the journey in the b2c policy you will be redirected back to the application.
I'm working with the js/node api of getstream and I'm trying to add a realtime feature to the comments on the activities, but I'm receiving a 403 error, displaying I dont have permission.
I've tried using targetFeeds: '[timeline:userid]' but it wrecks the application.
Also I tried to use the notification feed as in the documents is being used, and I can set targetFeeds like this: '[notification:userid]' which obviously is not the desired thing to do because this will cause that every message on different activities of this user will be shown on the callback.
client.reactions.add("comment", activityId, {
"text": newComment,
"profileImage": 'https://i.pravatar.cc/300',
"timestamp": date,
"from": userId,
"id": foreignId,
},
{targetFeeds: [`CommentsFeed:${activityId}`]});
And the response of the 403 is the next one:
{
code: 17
detail: "You don't have permission to do this"
duration: "0.18ms"
exception: "NotAllowedException"
status_code: 403
}
The expected result is not having the 403, that will trigger the callback I implemented.
The default permission settings allow users to only write activities to their own feeds; in this case you are adding an activity to CommentsFeed:${activityId}.
You can request support (support#getstream.io) to whitelist this for you app(s). Just make sure to mention this case and include your applications.
I’m having trouble getting basic conversation callbacks to be fired. Can anyone point me to a working sample with botkit 4 of getting responses from a conversation in Slack? I set up the SlackAdapter and used the SlackEventMiddleware and SlackMessageTypeMiddleware, but my callbacks aren’t getting called.
I took this basic code from the botkit docs and am calling it after a /slash command. The question is written, but no matter what I write, none of the callbacks are fired. I see the events coming to my server, but not to these call backs.
Here’s the code I’m testing with:
let convo = new BotkitConversation('cheese', controller)
await bot.startPrivateConversation(message.user)
// create a path for when a user says YES
convo.addMessage('You said yes! How wonderful.', 'yes_thread')
// create a path for when a user says NO
convo.addMessage('You said no, that is too bad.', 'no_thread')
// create a path where neither option was matched
// this message has an action field, which directs botkit to go back to the `default` thread after sending this message.
convo.addMessage('Sorry I did not understand.', 'bad_response')
// Create a yes/no question in the default thread...
convo.addQuestion(
'Do you like cheese?',
[
{
pattern: 'yes',
handler: async (response, convo, bot) => {
await convo.gotoThread('yes_thread')
}
},
{
pattern: 'no',
handler: async (response, convo, bot) => {
await convo.gotoThread('no_thread')
}
},
{
default: true,
handler: async (response, convo, bot) => {
await convo.gotoThread('bad_response')
}
}
],
'likes_cheese',
'default'
)
controller.addDialog(convo)
await bot.beginDialog(`cheese`)
Any help much appreciated!
So turns out something in my custom storage provider was interfering. I haven't gotten to the bottom of it yet, but when I commented out my storage provider, I was able to get conversation call backs.
e.g.
const controller: Botkit = new Botkit({
adapter: getSlackAdapter()
// TODO: This is causing conversation call backs to NOT fire, let's revisit this later. We may not actually need this.
//storage: services.storageService
})
I'm using the Google Calendar API and am trying to receive push notifications when a calendar event is triggered https://developers.google.com/calendar/v3/push
I think everything is setup correctly...
gapi.client.calendar.events.watch({
calendarId: 'primary',
resource: {
id: uuid,
type: 'web_hook',
address: window.location.href,
},
}, (err, response) => {
if (err) {
console.log('err:', err);
} else {
console.log('response:', response);
}
}).then((res) => {
console.log('res:', res);
});
But I guess not. I get a 200 response when I call the above code
{
"kind": "api#channel",
"id": "...",
"resourceId": "...",
"resourceUri": "https://content.googleapis.com/calendar/v3/calendars/primary/events?alt=json&maxResults=250&alt=json",
"expiration": "1554203159000"
}
I believe I should also be receiving a sync message, but I am not (https://developers.google.com/calendar/v3/push#sync)
To test things I am modifying an event within the calendar itself (changing the title, date, deleting, etc), and I expect something to happen in my browser, but nothing.
I'm not familiar with Push notifications in general, so not exactly sure what to expect.
I'm already authenticated and displaying events as per https://developers.google.com/calendar/quickstart/js
What am I missing? Any help is really appreciated!
Thanks in advance!
I suspect you are miss understanding exactly what push notifications is.
There are two primary ways to track when a resource has changed. You can poll that resource often and check for any changes in the resource.
For example Your application could run every five minutes and make a request to Google asking to have the event returned to you. When that event is returned you will then check if there are any changes in the event created by the user. This method of checking for changes is very time consuming and requires resources to constantly poll the server looking for changes. A better way of doing it is using Push notifications
Push notifications notify your application when a change has been made.
This document describes how to use push notifications that inform your application when a resource changes.
Its set up by enabling a watch
POST https://www.googleapis.com/calendar/v3/calendars/my_calendar#gmail.com/events/watch
Authorization: Bearer auth_token_for_current_user
Content-Type: application/json
{
"id": "01234567-89ab-cdef-0123456789ab", // Your channel ID.
"type": "web_hook",
"address": "https:/examle.com/notifications", // Your receiving URL.
...
"token": "target=myApp-myCalendarChannelDest", // (Optional) Your channel token.
"expiration": 1426325213000 // (Optional) Your requested channel expiration time.
}
This tells Googles servers that you would like to be notified when ever someone makes a change to the event. This sets up a web hook to https:/examle.com/notifications which will be notified as soon as there is a change to the event.
Name of the event, date time are normally changes i dont think you will get a push notification if someone else is added to the event.
What this is not
The server is NOT going to send you a message 10 minutes before the event is due. Thats user notification and something completely different and not part of the Google Calendar api.