Develop Reference

API request not updating variable in the first request. Second request is able to output the correct information

I am building an application to collate all the indicators that I use while investing in the stock market, as a personal project. I am currently making use of the yahoo-finance API. Given below is my code for one of the data loading functions, that will later be used with Tulind package.
var yahooFinance = require("yahoo-finance");
var returnValue = [];
const loadData = (symbol, from, to, freq) => {
let open = [];
let close = [];
let high = [];
let low = [];
let volume = [];
let updateValues = () => {
returnValue.map((e) => {
open.push(e.open);
close.push(e.adjClose);
high.push(e.high);
low.push(e.low);
volume.push(e.volume);
});
};
const data = yahooFinance.historical(
{
symbol: symbol,
from: from,
to: to,
freq: freq,
},
function (error, quotes) {
if (error)
console.log("Error in server/indicatorCalc/parent_calc.js", err);
else {
returnValue = quotes;
}
console.log("Data loaded in parent_calc.js");
}
);
updateValues();
return {
open,
close,
high,
low,
volume,
};
};
// console.log(tulind.indicators);
module.exports = { loadData };
Given above is the function, which has an API endpoint at localhost:5000/indicatorParent/
For testing purposes, the API is being called using the thunderclient extension on VSCode.
The first request sent with the required parameters is outputting an empty value (or the previous value if this is not the first request). When I click the send button for the second time, it gets updated with the correct/expected values. I want to know how to rectify this apparent "lag" in the updating of the returnValue variable. I am open to suggestions in changes in the flow of code as well.

Getting a function to execute on a node server when data recieved

I'll try to keep this simple. I'm working on a paging app.
What I have: A terminal without internet, running a desktop app made from node.js using express,ejs, ect. I also have a physical server with internet access and a mssql database on this machine I have a node.js server that interfaces with the database, collects the data and send sends it over to terminal via websocket when the app is launched, the node.js app gets this data and is rendered to the web interface using ejs. This data is displayed in a form with a button that when clicked fires the app.post route. In the app.post that data is packaged in an array, and sent back to the server using a second websocket connection. I can then take that form data (a name, phone#, and a radio button value) and form a SMS message using and using twilio send a message to that number (with addition info)
Yes, I have two websocket connections one Server > client serves data to web app
and another client > server serves form data to the server.
What's working: I get data from the boh server/database > webserver/client and data is displayed correctly when I hit the button data is packaged and send back to the boh server.
On the boh server i have a function that get the data array and parses it out, and sends an SMS message using twilio
My issue: I have to restart the server app to get it to process the data and send the message, if I hit the page button, it does all the stuff in the background it should, packages the array and sends the info to the server. The server is waiting for the data to be sent and client has sent the data, however it will only send off the text message is a stop and restart the node.js server, if i do that, the server starts and runs through the initial process of getting the sql data, and sets it up to be called when the app launches, then continues on to read that data was sent from the client, received and it will parse the data from the SMS message through a function send the message, wait a few seconds and then grab the response and confirm delivery of message. I am quite sure I am missing something basic here, but I have different functions but nothing I do will seem to get it to fire when the data is received.
I'm new at node and not very advanced in js, but I understand on some level why its not firing as is right, script is running, data has not been sent, so it stops when it gets here, then i hit the button, and it does nothing, because the script is stopped, and it doesn't has know way of knowing that data was sent, but when i rerun the script, the data that was sent is still sitting there wait to be received, so it recognizes the open websocket, and the sent data and work appropriately, I feel like a am missing something on the server side that tell it to wait for the data send but have not been able to make it work
twilws.onopen = async() => {//when the page button is presses, it starts a websocket server on the client, if that makes sense
twilws.send('test')
if (twilws.readyState === WebSocket.OPEN) {
logger.info('Twilio Sockpuppet Connected')
} else {
logger.error('Twilio Sockpuppet Fail')
}
twilws.onmessage = async (e) => {//after i hit the page button the server is on and the array is sent, i am trying to get this to be waiting for data and when it arrives go, but it will only do that when i restart the script
try {
data = JSON.parse(e.data);//parse the data
logger.info('got the edata ' + e.data)
//main(data) this will fire but not when the button is pressed.
} catch (er) {
logger.error('socket parse error: ' + e.data);
}
}
}
twilws.onclose = () => {//close the connection, purges that data so that the websocket can be recreated and an array with new data sent.
logger.info('Web Socket Connection Closed');
twilws.close(1000, 'all done');
};
Here is the whole server
I believe i may be blocking something, sorry about the bad formatting i have been changing and trying different things for a couple weeks off and on now and have not had a chance to clean thing up.
const express = require("express");
const cors = require("cors");
const app = express();
const db = require("./app/models");
const twilioconfig = require('./configs/twilioConfig.js');
const path = require('path');
const WebSocket = require('ws');
const sql = require('mssql');
const WebSocketServer = require('websocket').server;
const WebSocketClient = require('websocket').client;
const WebSocketFrame = require('websocket').frame;
/*const WebSocketRouter = require('websocket').router;*/
const W3CWebSocket = require('websocket').w3cwebsocket;
const http = require('http');
const https = require('https');
const twilio = require('twilio');
const { Console } = require("console");
app.disable('view cache');
const pino = require('pino')
const SonicBoom = require('sonic-boom')
const logger = require('pino')()
const transport = pino.transport({
target: 'pino/file',
options: { destination: './logs/logs.txt', level: 'info', mkdir: true, append: true }
})
pino(transport)
/*const dbConfig2 = require("./app/config/db.config.js");*/
/*const config = require("./app/config/config.js");*/
const { client } = require("websocket");
const { err } = require("./node_modules/pino-std-serializers/index");
const { setInterval } = require("node:timers/promises");
app.use(pino)
//const webserver = app.listen(8080, function () {
// console.log('Node WEb Server is running..');
//});
var params = {
autoReconnect: false, //Enable/Disable reconnect when the server closes connection (boolean)
autoReconnectInterval: 1000, //Milliseconds to wait between reconnect attempts (number)
autoReconnectMaxRetries: 600, //Max number of reconnect attempts to allow (number)
requestTimeout: 30000, //Milliseconds to wait for a response before resending the request (number)
requestRetryInterval: 5000, //Milliseconds between request retry checks. This garbage collects the retry queue (number)
requestRetryQueueMaxLength: 10 //Max queue length of retry queue before old messages start getting dropped (number)
}
var wss = new WebSocket.Server({ port: 8081 })
sql.connect(config, function (err) {
if (err)
logger.error(err);
const sqlRequest = new sql.Request();
const sqlQuery = "SELECT TOP 5 guest_name,guest_phone_number,CONVERT(varchar,creation_time, 126) AS creation_time,CONVERT(varchar,last_modified_timestamp, 126) AS last_modified_timestamp,party_size from dbo.WaitList where status = '4' AND CAST(creation_time as date) = CAST( GETDATE() AS Date ) ORDER BY creation_time ASC";
logger.info("query passes preflight.....lets get data")
sqlRequest.query(sqlQuery, function (err, data) {
if (err) {
logger.error(err)
} else {
logger.info("We do preliminary query now.")
}
//console.table(data.recordset);
//logger.info('rows affected ' + data.rowsAffected);
//console.log(data.recordset[0]);
var array = [];
for (let i = 0; i < data.rowsAffected; i++) {
var a = data.recordset[i];
array.push(a);
}
wss.on('connection', ws => {
logger.info('Client connection established')
ws.on('message', function () {
sqlRequest.query(sqlQuery, function (err, data) {
if (err) {
logger.error(err)
}
//console.table(data.recordset);
//logger.info(data.rowsAffected);
//console.log(data.recordset[0]);
var array = [];
for (let i = 0; i < data.rowsAffected; i++) {
var a = data.recordset[i];
array.push(a);
}
})
wss.clients
.forEach(client => {
logger.info('sending data')
client.send(JSON.stringify(array))
})
})
})
})
});
const twilws = new W3CWebSocket('ws://172.16.0.101:8082', params);
twilws.onopen = () => {//when the page button is presses, it starts a websocket server on the client, if that makes sense
twilws.send('test')
if (twilws.readyState === WebSocket.OPEN) {
logger.info('Twilio Sockpuppet Connected')
} else {
logger.error('Twilio Sockpuppet Fail')
}
twilws.onmessage = async (e) => {//after i hit the page button the server is on and the array is sent, i am trying to get this to be waiting for data and when it arrives go, but it will only do that when i restart the script
try {
data = JSON.parse(e.data);//parse the data
logger.info('got the edata ' + e.data)
//main(data) this will fire but not when the button is pressed.
} catch (er) {
logger.error('socket parse error: ');
}
}
}
twilws.onclose = () => {//close the connection, purges that data so that the websocket can be recreated and an array with new data sent.
logger.info('Web Socket Connection Closed');
twilws.close(1000, 'all done');
};
//async function main(s) {
// /* these settings are loaded from configs/twilioConfig.js, go here to set the store account info and edit message body.*/
// const TWILIO_ACCOUNT_SID = twilioconfig.twilioOptions.TWILIO_ACCOUNT_SID;
// const TWILIO_AUTH_TOKEN = twilioconfig.twilioOptions.TWILIO_AUTH_TOKEN;
// const STORE_TWILIO_NUMBER = twilioconfig.twilioOptions.STORE_TWILIO_NUMBER;
// const TEXT_TWILIO_BODY = twilioconfig.twilioOptions.TEXT_TWILIO_BODY;
// var messarray = new Array([s])
// var num = JSON.stringify(messarray[0][0][0])
// var state = JSON.stringify(messarray[0][0][1])
// logger.info(num + ', ' + state)
// var readyin = state
// var custnum = num
// /*logger.info(TWILIO_ACCOUNT_SID + ' ' + TWILIO_AUTH_TOKEN)*/
// var customer = new twilio(TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN);//these are our twilio account sid and token, set in twilioConfig, get from oneNote
// if (readyin == 0) {
// waitmsg = TEXT_TWILIO_BODY + "Your Table is Ready!";
// } else if (readyin == 15) {
// waitmsg = TEXT_TWILIO_BODY + "Your Table will be ready in about 15 minutes!";
// }
// logger.info(`Recieved Form data.....Twilio data read successfully....`);//log that we got the data from the gui form
// // send the message to the customer number through twilio, custnum from form, from is the store number, alson in twilioConfig
// customer.messages.create({
// to: custnum,
// from: STORE_TWILIO_NUMBER,
// body: waitmsg
// })
// .then(message => {
// var messageid = message.sid
// logger.info(messageid + ' Hey i found this, we might need it in a sec')
// const msid = JSON.stringify({ "MessageSid": messageid })
// const twilid = messageid
// setTimeout(getStatus, 5000);
// function getStatus() {
// customer.messages(messageid).fetch()
// .then(call => {
// const d = new Date(call.dateCreated).toLocaleString();
// const messageStatus = call.status
// var twil_response_array = [twilid, messageStatus, d]
// logger.info(twil_response_array)
// wss.on('connection', ws => {
// ws.on('message', function () {
// })
// wss.clients
// .forEach(client => {
// client.send(twil_response_array)
// })
// })
// })
// }
// //})
app.listen(8180, function () {
logger.info('Server is running..');
});
});

When making a list by query request in GMail via C#, what's the "source"?

Context: C#, JavaScript, ClearScript
I've written a plugin for my ClearScript-enabled JavaScript that connects to GMail. Everything's fine regarding OAuth2 etc. What's not working is a query against the inbox. If I request the inbox without a query, that gives me data, but I don't want millions of records.
attach(".\\Plugin_GMail.dll")
var tuple = Plugin_GMail.GoogleMail.Mail.Mail_Authenticate(REDACTED)
var service = Plugin_GMail.GoogleMail.Mail.Mail_CreateService(tuple.Item1, "mail")
var request = service.Users.Labels.List("me");
var labels = request.Execute().Labels;
var before = new Date(2022,11,31).valueOf();
var after = new Date(2022,0,1).valueOf();
var res = Plugin_GMail.GoogleMail.Mail.Messages_ListByQuery(service, "me", "before:$b after:$a".replace("$b",before).replace("$a",after),true)
The code for Messages_ListByQuery is
public static string Messages_ListByQuery(GmailService service, string userId, string query, bool debug = false)
{
if (debug) Debugger.Launch();
var msgList = new List<Message>();
var result = new JSONResponse();
var request = service.Users.Messages.List(userId);
request.MaxResults = 500;
request.Q = query;
request.LabelIds = "INBOX";
while (true)
{
try
{
var msgs = request.Execute().Messages;
if (msgs.Count() == 0)
{
break;
}
msgList.AddRange(msgs);
}
catch (Exception e)
{
result.Error = e.Message;
break;
}
}
result.Cargo = msgList;
return JsonConvert.SerializeObject(result);
}
So the response after executing the query-based List is
{"Error":"Value cannot be null.\r\nParameter name: source","Cargo":[],"Crew":null}
I have been through the documentation and can't find anything about "source". And there's nothing in the Playground about "source" either. Where next?

Your code is telling it to just loop forever, getting the same page of results.
while (true) // loop forever
{
try
{
var msgs = request.Execute().Messages; // get first page of results
if (msgs.Count() == 0)
{
break; // will never hit unless user has no messages. in their inbox at all.
}
msgList.AddRange(msgs); // keeps adding first page of messages over and over.
}
catch (Exception e)
{
result.Error = e.Message;
break;
}
}
The line
var msgs = request.Execute().Messages;
Is just going to say run the request and get me messages. Which will always return messages as your running the same request over and over.
The following line is just going to add the messages to your list.
msgList.AddRange(msgs);
So with the while true you are telling your app to just request the same 500 rows again and again forever.
You should instead look at using the next page token and getting the next page of results instead of the same results over and over.

Google cast v3 CAF Receiver application with DRM url

I am trying to use the v3 CAF receiver app using DRM to casting videos, from my IOS app. If I use the basic v3 CAF receiver app (default receiver) it is working fine, but when I using DRM url (dash/.mpd and licenseUrl ) it will throw below error
Error
[ 20.844s] [Error] [INFO] {"type":"LOAD_CANCELLED","requestId":0}
See the below code.
const playerManager = context.getPlayerManager();
const playbackConfig = new cast.framework.PlaybackConfig();
/** Debug Logger **/
const castDebugLogger = cast.debug.CastDebugLogger.getInstance();
var manifestUri = 'https://example.domain.video/prod/drm/1/7e942940-d705-4417-b552-796e8fd25460/Media_1_20_d2aaec7102dc42c09dd54e4f00cbea412019062801270383196000/dash/manifest.mpd';
var licenseServer = 'https://wv.example.domain.com/hms/wv/rights/?ExpressPlayToken=BQALuGDeKZcAJDE2YzAwYTRkLTYwZWYtNGJiZC1hZmEzLTdhMmZhYTY2NzM5OQAAAHCZzHVjRyfs3AEgxFuwPvZsrqMndjiBPzLQ5_VUx6rJOEDD5noQmXJoVP-Va1gQzxfp9eHux15_pEr6g0RxXNZIjlsN6b7SIfpHPyS9iuPQqgvEgq5I_tV9k1lhQvKuqgpBN0Z5BtxCLwHc8xrnLbuUK6fiThcLMR4He_x38reAsumjFYg';
// setting manually licenseUrl from here
playbackConfig.licenseUrl = licenseServer;
playbackConfig.manifestRequestHandler = requestInfo => {
requestInfo.withCredentials = true;
};
playbackConfig.licenseRequestHandler = requestInfo => {
requestInfo.withCredentials = true;
requestInfo.headers = {
// 'Content-type':'application/dash+xml', // trying this also
'Content-type':'application/octet-stream'
}
playbackConfig.licenseUrl = requestInfo.media.customData.licenseUrl;
return playbackConfig;
};
// MessageInterceptor
playerManager.setMessageInterceptor(
cast.framework.messages.MessageType.LOAD,
request => {
const error = new cast.framework.messages.ErrorData(cast.framework.messages.ErrorType.LOAD_CANCELLED);
castDebugLogger.info('Error', error);
if (!request.media) {
error.reason = cast.framework.messages.ErrorReason.INVALID_PARAM;
castDebugLogger.info('reason', error.reason);
return error;
}
if (request.media && request.media.entity) {
request.media.contentId = request.media.entity;
}
return new Promise((resolve, reject) => {
if (!request.media) {
castDebugLogger.error('MyAPP.LOG', 'Content not found');
reject();
} else {
// I have passed manually data (license Url and content Id etc.) from here for testing purpose
const item = new cast.framework.messages.QueueItem();
item.media = new cast.framework.messages.MediaInformation();
item.media.contentId = manifestUri;
item.media.streamType = cast.framework.messages.StreamType.BUFFERED;
// Trying all options of contentType
item.media.contentType = "application/octet-stream";
//request.media.contentType = 'application/x-mpegurl';
//item.media.contentType = "video/mp4";
//request.media.contentType = 'video/mp4';
//request.media.contentType = 'application/dash+xml';
item.media.metadata = new cast.framework.messages.MovieMediaMetadata();
item.media.metadata.title = "Example title";
item.media.metadata.subtitle = "Example subtitle ";
item.media.metadata.images = [new cast.framework.messages.Image("https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/images/BigBuckBunny.jpg")];
request.media = item.media;
playbackConfig.protectionSystem = cast.framework.ContentProtection.WIDEVINE;
resolve(request);
}
});
});
// start
context.start({
playbackConfig: playbackConfig,
touchScreenOptimizedApp: true
});
LA_URL and .mpd url is working fine with another online shaka player.

Did you check in the remote web inspector if the network request is sent to the licenser when the load request is issued for the encoded dash stream? Most probably this will help to find where the problem is.
Possibly you will have to add some inteligence to your licenseRequestHandler to add a token of some sort. Or possibly there's a CORS issue.
Note: Before you post some code to stackoverflow, it might be wize to clean it up a bit: remove dead code, remove confusing commented code, provide proper indentation. You're wasting brain cycles of everybody reading your code and trying to process what you shared with the world!

Postmessage in a loop

I want to bombard the receiver with say 1M messages, but also that the receiver will get each message as soon as possible. The naive way is to loop 1M times and just send a message to the receiver with postmessage. Doesn't work.
What i get is that the whole 1M messages are queued, and only when the code finishes, the receiver starts processing them.
What i need to happen is that the sender will send 1M messages and as he keeps on sending the messages the receiver simultaneously will process them.
For example, what i have now is something like this:
sender : send m1.
sender : send m2.
sender : send m3.
receiver : received m1.
receiver : received m2.
receiver : received m3.
What i want:
sender : send m1.
receiver : received m1.
sender : send m2.
receiver : received m2.
sender : send m3.
receiver : received m3.
How can i achieve this? I can not make the receiver send acks. My goal is to send as many massages as i can the fastest.
Edit: The code i have now:
Sender:
function sendx(x){
console.log("start spam");
for(let i=0; i<200000; i++){
window.opener.postMessage(x, '*');
}
console.log("done");
}
Receiver:
window.addEventListener("message", r_function );
function r_function(event)
{
let index = event.data;
let junk = something(index);
return junk;
}
Where the sender is a new window created by the receiver. What i get in practice is that only when the 'sendx' function ends, the receiver start receiving messages.

What i need to happen is that the sender will send 1M messages and as he keeps on sending the messages the receiver simultaneously will process them.
That's what happens already.
const worker = new Worker(URL.createObjectURL(
new Blob([worker_script.textContent])
));
let logged_first = false;
worker.onmessage = e => {
if(e.data === "spam") {
if(!logged_first) {
console.log('received first message at', new Date().toLocaleString());
logged_first = true; // ignore next messages
}
}
else {
console.log(e.data);
}
}
<script type="text/worker-script" id="worker_script">
const now = performance.now();
postMessage("start spamming at " + new Date().toLocaleString());
while(performance.now() - now < 5000) {
postMessage('spam');
}
postMessage("end spamming at " + new Date().toLocaleString());
</script>
However, for it to work, there is one big condition that needs to be met:
Your two JavaScript instances (sender & receiver) must run on different threads.
That is, if you were doing it using a MessageChannel on the same thread, then it would obviously be unable to treat the messages at the same time it's sending it:
const channel = new MessageChannel();
channel.port1.onmessage = e => {
console.log('received first message at', new Date().toLocaleString());
channel.port1.onmessage = null; // ignore next messages
};
const now = performance.now();
console.log("start spamming at ", new Date().toLocaleString());
while(performance.now() - now < 5000) {
channel.port2.postMessage('spam');
}
console.log("end spamming at ", new Date().toLocaleString());
And if you are dealing with an iframe or an other window, you can not be sure that you'll meet this condition. Browsers all behave differently, here, but they will all run at least some windows on the same process. You have no control as to which process will be used, and hence can't guarantee that you'll run in an other one.
So the best you can do, is to run your loop in a timed-loop, which will let the browser some idle time where it will be able to process other windows event loops correctly.
And the fastest timed-loop we have is actually the one postMessage offers us.
So to do what you wish, the best would be to run each iteration of your loop in the message event of a MessageChannel object.
For this, generator function* introduced in ES6 are quite useful:
/***************************/
/* Make Fake Window part */
/* ONLY for DEMO */
/***************************/
const fake_win = new MessageChannel();
const win = fake_win.port1; // window.open('your_url', '')
const opener = fake_win.port2; // used in Receiver
/********************/
/* Main window part */
/********************/
const messages = [];
win.onmessage = e => {
messages.push(e.data);
};
!function log_msg() {
document.getElementById('log').textContent = messages.length;
requestAnimationFrame(log_msg);
}();
/*******************/
/* Receiver part */
/*******************/
// make our loop a Generator function
function* ourLoopGen(i) {
while(i++ < 1e6) {
opener.postMessage(i);
yield i;
}
}
const ourLoop = ourLoopGen(0);
// here we init our time-loop
const looper = new MessageChannel();
looper.port2.onmessage = e => {
const result = ourLoop.next();
if(!result.done)
looper.port1.postMessage(''); // wait next frame
};
// start our time-loop
looper.port1.postMessage('');
<pre id="log"></pre>
We could also do the same using ES6 async/await syntax, since we can be sure that nothing else in our MessageChannel powered timed-loop will interfere (unlike in a Window's postMessage), we can promisify it:
/***************************/
/* Make Fake Window part */
/* ONLY for DEMO */
/***************************/
const fake_win = new MessageChannel();
const win = fake_win.port1; // window.open('your_url', '')
const opener = fake_win.port2; // used in Receiver
/********************/
/* Main window part */
/********************/
const messages = [];
win.onmessage = e => {
messages.push(e.data);
};
! function log_msg() {
document.getElementById('log').textContent = messages.length;
requestAnimationFrame(log_msg);
}();
/*******************/
/* Receiver part */
/*******************/
const looper = makeLooper();
// our async loop function
async function loop(i) {
while (i++ < 1e6) {
opener.postMessage(i);
await looper.next()
}
}
loop(0);
// here we init our promisified time-loop
function makeLooper() {
const engine = new MessageChannel();
return {
next() {
return new Promise((res) => {
engine.port2.onmessage = e => res();
engine.port1.postMessage('');
});
}
};
};
<pre id="log"></pre>
But it could obviously also be made entirely ES5 style with callbacks and everything:
/***************************/
/* Make Fake Window part */
/* ONLY for DEMO */
/***************************/
var fake_win = new MessageChannel();
var win = fake_win.port1; // window.open('your_url', '')
var opener = fake_win.port2; // used in Receiver
/********************/
/* Main window part */
/********************/
var messages = [];
win.onmessage = function(e) {
messages.push(e.data);
};
!function log_msg() {
document.getElementById('log').textContent = messages.length;
requestAnimationFrame(log_msg);
}();
/*******************/
/* Receiver part */
/*******************/
var i = 0;
var looper = makeLooper(loop);
// our callback loop function
function loop() {
if (i++ < 1e6) {
opener.postMessage(i);
looper.next(loop);
}
}
loop(0);
// here we init our promisified time-loop
function makeLooper(callback) {
var engine = new MessageChannel();
return {
next: function() {
engine.port2.onmessage = function(e) {
callback();
}
engine.port1.postMessage('');
}
};
};
<pre id="log"></pre>
But note that browsers will anyway throttle the pages that are not in focus, so you may have slower results than in these snippets.