Develop Reference

Query MariaDB from NodeJS without Caching from MariaDB

I try to do a query from MariaDB with my NodeJS without using the MariaDB caching. I use Sequelize to query the database.
Everything works fine but I recognized that there is definitly some issue with caching. The select query I use always reports a long time that a new user is not available. But another MariaDB insert generated the data before.
To check the user I use:
isUserExists: async function ( inUser, inChat, inLevel ) {
const iskunde = await kunden.findAll({
where: {
kunde: inUser,
chat: inChat,
level: inLevel,
created_at: {
[Op.gte]: Sequelize.literal("DATE_SUB(NOW(), INTERVAL 15 MINUTE)"),
},
},
raw: true,
});
//var currentdate = new Date();
//console.log(currentdate);
//console.log(iskunde);
if(Array.isArray(iskunde) && iskunde.length) {
return true;
}else{
return false;
}
},
This becomes a big problem with my app. So I try to find a way to get the "realtime" data from database. From what I read there is a SELECT possible with SQL_NO_CACHE but I do not find any information how to manage this with Sequelize. Any idea?

Evaporate.js upload file with x-amz-security-token: SignatureDoesNotMatch

I want upload a file with evaporate.js and crypto-js using x-amz-security-token:
import * as Evaporate from 'evaporate';
import * as crypto from "crypto-js";
Evaporate.create({
aws_key: <ACCESS_KEY>,
bucket: 'my-bucket',
awsRegion: 'eu-west',
computeContentMd5: true,
cryptoMd5Method: data => crypto.algo.MD5.create().update(String.fromCharCode.apply(null, new Uint32Array(data))).finalize().toString(crypto.enc.Base64),
cryptoHexEncodedHash256: (data) => crypto.algo.SHA256.create().update(data as string).finalize().toString(crypto.enc.Hex),
logging: true,
maxConcurrentParts: 5,
customAuthMethod: (signParams: object, signHeaders: object, stringToSign: string, signatureDateTime: string, canonicalRequest: string): Promise<string> => {
const stringToSignDecoded = decodeURIComponent(stringToSign)
const requestScope = stringToSignDecoded.split("\n")[2];
const [date, region, service, signatureType] = requestScope.split("/");
const round1 = crypto.HmacSHA256(`AWS4${signParams['secret_key']}`, date);
const round2 = crypto.HmacSHA256(round1, region);
const round3 = crypto.HmacSHA256(round2, service);
const round4 = crypto.HmacSHA256(round3, signatureType);
const final = crypto.HmacSHA256(round4, stringToSignDecoded);
return Promise.resolve(final.toString(crypto.enc.Hex));
},
signParams: { secretKey: <SECRET_KEY> },
partSize: 1024 * 1024 * 6
}).then((evaporate) => {
evaporate.add({
name: 'my-key',
file: file, // file from <input type="file" />
xAmzHeadersCommon: { 'x-amz-security-token': <SECURITY_TOKEN> },
xAmzHeadersAtInitiate: { 'x-amz-security-token': <SECURITY_TOKEN> },
}).then(() => console.log('complete'));
},
(error) => console.error(error)
);
but it produce this output
AWS Code: SignatureDoesNotMatch, Message:The request signature we calculated does not match the signature you provided. Check your key and signing method.status:403
What I'm doing wrong
SIDE NOTE
This is the versione I'm using on browser side:
{
"crypto-js": "^4.1.1",
"evaporate": "^2.1.4"
}

You have your crypto.HmacSHA256 parameters reversed. They should all be the other way around. I've been bashing my head against a wall trying to get evaporate 2.x to work for the last week, it's been very frustrating.
I tried your code above and looked over all the docs and forum posts related to this, and I think using Cognito for this auth just doesn't work or isn't obvious how it's supposed to work, even though the AWS docs suggest it's possible.
In the end I went with using Lambda authentication and finally got it working after seeing much misinformation about how to use various crypto libraries to sign this stuff. I got it working last night after rigorously examining every bit of what was going on. Reading the docs also helped get me on the right path as to how the crypto needs to work, it gives example inputs and outputs so you can test for sure that your crypto methods are working as AWS expects them to work:
https://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
Tasks 1, 2 and 3 are especially important to read and understand.

Testing Cross Browser Extension With Jest, How To Mock Chrome Storage API?

After putting off testing for a while now due to Cypress not allowing visiting chrome:// urls, I decided to finally understand how to unit/integration test my extension - TabMerger. This comes after the many times that I had to manually test the ever growing functionality and in some cases forgot to check a thing or two. Having automated testing will certainly speed up the process and help me be more at peace when adding new functionality.
To do this, I chose Jest since my extension was made with React (CRA). I also used React Testing Library (#testing-library/react) to render all React components for testing.
As I recently made TabMerger open source, the full testing script can be found here
Here is the test case that I want to focus on for this question:
import React from "react";
import { render, fireEvent } from "#testing-library/react";
import * as TabFunc from "../src/Tab/Tab_functions";
import Tab from "../src/Tab/Tab";
var init_groups = {
"group-0": {
color: "#d6ffe0",
created: "11/12/2020 # 22:13:24",
tabs: [
{
title:
"Stack Overflow - Where Developers Learn, Share, & Build Careersaaaaaaaaaaaaaaaaaaaaaa",
url: "https://stackoverflow.com/",
},
{
title: "lichess.org • Free Online Chess",
url: "https://lichess.org/",
},
{
title: "Chess.com - Play Chess Online - Free Games",
url: "https://www.chess.com/",
},
],
title: "Chess",
},
"group-1": {
color: "#c7eeff",
created: "11/12/2020 # 22:15:11",
tabs: [
{
title: "Twitch",
url: "https://www.twitch.tv/",
},
{
title: "reddit: the front page of the internet",
url: "https://www.reddit.com/",
},
],
title: "Social",
},
};
describe("removeTab", () => {
it("correctly adjusts groups and counts when a tab is removed", () => {
var tabs = init_groups["group-0"].tabs;
const { container } = render(<Tab init_tabs={tabs} />);
expect(container.getElementsByClassName("draggable").length).toEqual(3);
var removeTabSpy = jest.spyOn(TabFunc, "removeTab");
fireEvent.click(container.querySelector(".close-tab"));
expect(removeTabSpy).toHaveBeenCalledTimes(1);
expect(container.getElementsByClassName("draggable").length).toEqual(2); // fails (does not remove the tab for some reason)
});
});
I mocked the Chrome API according to my needs, but feel that something is missing. To mock the Chrome API I followed this post (along with many others, even for other test runners like Jasmine): testing chrome.storage.local.set with jest.
Even though the Chrome storage API is mocked, I think the issue lies in this function which gets called upon initial render. That is, I think the chrome.storage.local.get is not actually being executed, but am not sure why.
// ./src/Tab/Tab_functions.js
/**
* Sets the initial tabs based on Chrome's local storage upon initial render.
* If Chrome's local storage is empty, this is set to an empty array.
* #param {function} setTabs For re-rendering the group's tabs
* #param {string} id Used to get the correct group tabs
*/
export function setInitTabs(setTabs, id) {
chrome.storage.local.get("groups", (local) => {
var groups = local.groups;
setTabs((groups && groups[id] && groups[id].tabs) || []);
});
}
The reason I think the mocked Chrome storage API is not working properly is because when I manually set it in my tests, the number of tabs does not increase from 0. Which forced me to pass a prop (props.init_tabs) to my Tab component for testing purposes (https://github.com/lbragile/TabMerger/blob/f78a2694786d11e8270454521f92e679d182b577/src/Tab/Tab.js#L33-L35) - something I want to avoid if possible via setting local storage.
Can someone point me in the right direction? I would like to avoid using libraries like jest-chrome since they abstract too much and make it harder for me to understand what is going on in my tests.

I think I have a solution for this now, so I will share with others.
I made proper mocks for my chrome storage API to use localStorage:
// __mocks__/chromeMock.js
...
storage: {
local: {
...,
get: function (key, cb) {
const item = JSON.parse(localStorage.getItem(key));
cb({ [key]: item });
},
...,
set: function (obj, cb) {
const key = Object.keys(obj)[0];
localStorage.setItem(key, JSON.stringify(obj[key]));
cb();
},
},
...
},
...
Also, to simulate the tab settings on initial render, I have a beforeEach hook which sets my localStorage using the above mock:
// __tests__/Tab.spec.js
var init_ls_entry, init_tabs, mockSet;
beforeEach(() => {
chrome.storage.local.set({ groups: init_groups }, () => {});
init_ls_entry = JSON.parse(localStorage.getItem("groups"));
init_tabs = init_ls_entry["group-0"].tabs;
mockSet = jest.fn(); // mock for setState hooks
});
AND most importantly, when I render(<Tab/>), I noticed that I wasn't supplying the id prop which caused nothing to render (in terms of tabs from localStorage), so now I have this:
// __tests__/Tab.spec.js
describe("removeTab", () => {
it("correctly adjusts storage when a tab is removed", async () => {
const { container } = render(
<Tab id="group-0" setTabTotal={mockSet} setGroups={mockSet} />
);
var removeTabSpy = jest.spyOn(TabFunc, "removeTab");
var chromeSetSpy = jest.spyOn(chrome.storage.local, "set");
fireEvent.click(container.querySelector(".close-tab"));
await waitFor(() => {
expect(chromeSetSpy).toHaveBeenCalled();
});
chrome.storage.local.get("groups", (local) => {
expect(init_tabs.length).toEqual(3);
expect(local.groups["group-0"].tabs.length).toEqual(2);
expect(removeTabSpy).toHaveBeenCalledTimes(1);
});
expect.assertions(4);
});
});
Which passes!!
Now on to drag and drop testing 😊

Google Speech API - Server Unavailable Error for long audio files

I'm using Google's nodejs-speech package to use the longRunningRecognize endpoint/function in Google's Speech API.
I've used both v1 and v1p1beta, and run into an error with longer files. (48 mins is as long as I've tried, and 15 mins causes the same problem, though 3 mins does not). I've tried both the promise pattern and separating the request into two parts -- one to start the longRunningRecognize process, and the other to check on results after waiting. The error is shown below the code samples for both.
Example promise version of request:
import speech from '#google-cloud/speech';
const client = new speech.v1p1beta1.SpeechClient();
const audio = {
uri: 'gs://my-bucket/file.m4a'
};
const config = {
encoding: 'AMR_WB',
sampleRateHertz: 16000,
languageCode: 'en-US',
enableWordTimeOffsets: true,
enableSpeakerDiarization: true
};
const request = {
audio,
config
};
client.longRunningRecognize(request)
.then(data => {
const operation = data[0];
return operation.promise();
})
.then(data => {
const response = data[0];
const results = response.results;
const transcription = results
.filter(result => result.alternatives)
.map(result => result.alternatives[0].transcript)
.join('\n');
console.log(transcription);
})
.catch(error => {
console.error(error);
});
(I've since closed the tab with the results, but I think this returned an error object that just said { error: { code: 13 } }, which matches the below, more descriptive error).
Separately, I've tried a version where instead of chaining promises to get the final transcription result, I collect the name from the operation, and make a separate request to get the result.
Here's that request code:
... // Skipping setup
client.longRunningRecognize(request)
.then(data => {
const operation = data[0];
console.log(operation.latestResponse.name);
})
.catch(error => {
console.error(error);
});
When I hit the relevant endpoint (https://speech.googleapis.com/v1p1beta1/operations/81703347042341321989?key=ABCD12345) before it's had time to process, I get this:
{
"name": "81703347042341321989",
"metadata": {
"#type": "type.googleapis.com/google.cloud.speech.v1p1beta1.LongRunningRecognizeMetadata",
"startTime": "2018-08-16T19:33:26.166942Z",
"lastUpdateTime": "2018-08-16T19:41:31.456861Z"
}
}
Once it's fully processed, though, I've been running into this:
{
"name": "81703347042341321989",
"metadata": {
"#type": "type.googleapis.com/google.cloud.speech.v1p1beta1.LongRunningRecognizeMetadata",
"progressPercent": 100,
"startTime": "2018-08-16T17:20:28.772208Z",
"lastUpdateTime": "2018-08-16T17:44:40.868144Z"
},
"done": true,
"error": {
"code": 13,
"message": "Server unavailable, please try again later."
}
}
I've tried with shorter audio files (3 mins, same format and encoding), and the above processes both worked.
Any idea what's going on?

A possible workaround is changing the audio format to FLAC, which is the recommended encoding type for Cloud Speech-to-text API due to its lossless compression.
For reference, this can be done using sox, through the following command:
sox file.m4a --rate 16k --bits 16 --channels 1 file.flac
Additionally, this error may also happen when there is a long period of silence at the beginning. In this case, the audio files can be trimmed by specifying after trim the amount of seconds the audio should skip at the beginning and at the end of the file:
sox input.m4a --rate 16k --bits 16 --channels 1 output.flac trim 20 5

How to monitor HTTP calls using browsermob-proxy and nightwatch.js?

I am writing testcases using Nightwatch.js framework for SPA application. A requirement came in here we have to monitor HTTP calls and get the performance results for the site. As this could be easily achieved using JMeter.
Using automation testing tool, we can do it by using browsermob-proxy and selenium.
Is it possible to do the same using Nightwatch.js and browsermob-proxy?
Also what are the steps to do to the same.

For using Nightwatchjs and browsermob-proxy together, check out this repo, which includes info on the NodeJS bindings for browsermob-proxy and programmatically generating HAR (HTTP Archive) files.
If you're content with just using Nightwatchjs, this repo has code in the tests directory for the following:
Custom command to get the requests made so far
Custom assertion for checking if a request, given a filter and query string params, exists.
You might have to brush up on how to add custom commands and assertions to your Nightwatch project, but after that you should be set to go!

You can use browsermob-proxy-api
just simply download browsermob-proxy server then
install by npm command: npm install browsermob-proxy-api --save-dev
configure you night watch like this in desiredCapabilites:
'test_settings': {
'default': {
'launch_url': 'http://localhost:3000',
'screenshots': {
'enabled': true, // if you want to keep screenshots
'path': './screenshots' // save screenshots here
},
'globals': {
'waitForConditionTimeout': 30000 // sometimes internet is slow so wait.
},
'desiredCapabilities': { // use Chrome as the default browser for tests
'browserName': 'chrome',
'proxy': {
'proxyType': 'manual',
'httpProxy': 'localhost:10800'
},
'acceptSslCerts': true,
'javascriptEnabled': true, // turn off to test progressive enhancement
}
},
then download index.js from here:
https://github.com/jmangs/node-browsermob-proxy-api
and add code from example to your step_definitions if you use gherkin or describe step

Bit late into dance. I managed to integrate browsermob to nightwatch. Here are the detailed steps
Download browsermob proxy https://bmp.lightbody.net/
Open your cmd and go to bin folder and then start browsermob using "browsermob-proxy".
I am assuming you have basic nightwatch setup. You also need mobproxy. Install it from "npm i browsermob-proxy-api"
Create a global hook in nightwatch. Say 'globalmodule.js' and give this file path in globals_path in nightwatch.json
In globalmodule, create global hooks as described in http://nightwatchjs.org/guide#external-globals
In beforeEach hook, add below code: //if you are not under corporate proxy and you dont need to chain to upstream proxy
var MobProxy = require('browsermob-proxy-api');
var proxyObj = new MobProxy({'host': 'localhost', 'port': '8080'});
//assuming you started browsermob in 8080 port. That is in step 2.
//if you are working under corporate proxy, you might have to chain your request. This needs editing in browsermob-proxy-api package. Follow steps given at end of this section.
Start proxy on new port
proxyObj.startPort(port, function (err, data) {
if (err) {
console.log(err);
} else {
console.log('New port started')
}
})
Once we have new port, we have to start our chrome browser in above port so that all browser request are proxied through browsermob.
proxyObj.startPort(port, function (err, data) {
if (err) {
console.log(err);
} else {
console.log('New port started')
var dataInJson = JSON.parse(data);
//Step 8:
this.test_settings.desiredCapabilities = {
"browserName": "chrome",
"proxyObj": proxyObj, //for future use
"proxyport": dataInJson.port, //for future use
"proxy": {
"proxyType": "manual",
"httpProxy": "127.0.0.1:" + dataInJson.port,
"sslProxy": "127.0.0.1:" + dataInJson.port //important is you have https site
},
"javascriptEnabled": true,
"acceptSslCerts": true,
"loggingPrefs": {
"browser": "ALL"
}
}
}
})
Try to run with above setting, you can check if cmd [created in step2 to confirm request are going via above port. There will be some activiy]
For creating HAR and getting created HAR, browsermob-proxy-api gives excellent api.
add createHAR.js in any path and mention that path in nightwatch.json[custom_commands section]
exports.command = function (callback) {
var self = this;
if (!self.options.desiredCapabilities.proxyObj) {
console.error('No proxy setup - did you call setupProxy() ?');
}
this.options.desiredCapabilities.proxyObj.createHAR(this.options.desiredCapabilities.proxyport, {
'captureHeaders': 'true',
'captureContent': 'true',
'captureBinaryContent': 'true',
'initialPageRef': 'homepage'
}, function (err, result){
if(err){
console.log(err)
}else{
console.log(result)
if (typeof callback === "function") {
console.log(this.options.desiredCapabilities.proxyObj);
console.log(this.options.desiredCapabilities.proxyport);
// console.log(result);
callback.call(self, result);
}
}
});
return this;
};
then to getHAR, add getHAR.js, add below code.
var parsedData;
exports.command = function(callback) {
var self = this;
if (!self.options.desiredCapabilities.proxy) {
console.error('No proxy setup - did you call setupProxy() ?');
}
self.options.desiredCapabilities.proxyObj.getHAR(self.options.desiredCapabilities.proxyport, function (err, data) {
console.log(self.options.desiredCapabilities.proxyObj);
console.log(self.options.desiredCapabilities.proxyport);
//console.log(result);
if(err){
console.log(err)
}else{
parsedData = JSON.parse(data)
console.log(parsedData.log.entries)
}
if (typeof callback === "function") {
console.log(self.options.desiredCapabilities.proxyObj);
console.log(self.options.desiredCapabilities.proxyport);
callback.call(self, parsedData);
}
});
return this;
};
At start of test, createHAR will not have proxyObj, So this step should be executed sync. Wrap that step with browser.perform()
browser.perform(function(){
browser.createHAR()
})
////some navigation
browser.perform(function(){
browser.getHAR()
})
Note: If you are working behind corporate proxy, You might have to use chain proxy piece which browsermob offers.
According to browsermob proxy documentation, get down to api section, -> /proxy can have request parameters "proxyUsername" and "proxyPassword"
In node_modules->browsermob-proxy-api->index.js
add below line after line 22:
this.proxyUsername = cfg.proxyUsername || '';
this.proxyPassword = cfg.proxyPassword || '';
this.queryString = cfg.queryString || 'httpProxy=yourupstreamProxy:8080'; //you will get this from pac file
then at line 177, where package is making request '/proxy' to browser.
replace
path: url
to
path: url + '?proxyUsername=' +this.proxyUsername + '&proxyPassword=' + this.proxyPassword + '&' + this.queryString