Develop Reference

NodeJs API sending blank response while converting larger files

I am currently working on a piece of code that uploads mp4 videos from my localhost onto a server. The basic thing is that if the video is a .mp4 it is directly uploaded, else it is converted to .mp4 and then uploaded. I'm using the video converter using handbrake-js.
All works fine except for a tiny part. When the file isn't that huge, say less than 70-80 Mb, it works like a charm. But the problem is with larger files. Even though I am explicitly calling the res.end / res.send in the .on(end) callback, I receive some blank response in my angular controller, even before the conversion has finished. I have noticed it happens at around 30 to 40% of the conversion. It has a readystate which is equal to XMLHttpRequest.DONE and also status = 200.
Here is the Node side code:
try {
if (fs.existsSync(uploadPath + filename.substring(0, filename.lastIndexOf('.')) + '.mp4')) {
res.end('<b><i>' + filename + '</i></b> already exists in the directory.');
}
else {
const fstream = fs.createWriteStream(path.join(cfg.tempStoragePath, filename));
file.pipe(fstream);
console.log("\nfile_type: " + file_type);
filename = filename.substring(0, filename.lastIndexOf('.'));
// On finish of the copying file to temp location
fstream.on('close', () => {
hbjs.spawn({
input: cfg.tempStoragePath + filename + '.' + file_type,
output: uploadPath + filename + '.mp4'
})
.on('error', err => {
// invalid user input, no video found etc
console.log('error! No input video found at: \n: ' + cfg.tempStoragePath + filename + '.' + file_type);
res.send('Conversion of the file, <b><i>' + filename + '</i></b>, from <b>.' + file_type + '</b>' + ' to <b>.mp4</b> failed because no input video was found at: \n: ' + cfg.tempStoragePath + filename + '.' + file_type);
})
.on('progress', progress => {
progress_percent = (Number(progress.percentComplete) * 2 <= 100) ? Number(progress.percentComplete) * 2 : 100;
eta = progress.eta.split(/[a-zA-Z]/);
minutes = ((+eta[0]) * 60 + (+eta[1])) / 2;
console.log('Percent complete: %d, ETA: %d ///// %s ==> mp4', progress_percent, minutes, file_type);
})
.on('end', end => {
console.log('Conversion from .' + file_type + ' to .mp4 complete.');
//delete the temp file
fs.unlink(cfg.tempStoragePath + filename + '.' + file_type);
let new_path = uploadPath + filename + '.mp4';
let stat = fs.statSync(new_path);
console.log(`Upload of '${filename}' finished`);
if(Number(progress_percent) === Number(100))
res.send('The file, <b><i>' + filename + '</i></b>, has been converted from <b>.' + file_type + '</b>' + ' to <b>.mp4</b> complete.');
})
});
}
}
catch (err) {
res.end(err);
}
Following is part of my angular controller:
request = new XMLHttpRequest();
request.onreadystatechange = function () {
if (request.readyState === XMLHttpRequest.DONE && request.status === 200) {
showConversionModal('<p>' + request.responseText + '</p>', 'done');
}
};
showSomeModal('something');
request.open("POST", client.clientHost + ":" + client.clientPort + "/uploadVideoService");
formData = new FormData();
formData.append("file", files[0], files[0].name);
request.send(formData);
NOTE : All the data which is do a console.log() inside the node is as expected. And even the res.end/send works fine for smaller files which take lesser time. But the problem arises only for those which conversion takes longer than the smaller files.
Also, the if loop which checks for existing file scenario doesn't work as expected for these larger files. I thought at least that should work, because it doesn't even get into the handbrake part. But that is not the case.
And in the browser I get this error:
Failed to load resource: net::ERR_CONNECTION_RESET
which points to the request.send(formData); line, and I have also tried almost all solution from this SO article, with no good effect. But still, the conversion happens fine.
P.S.: Also note that the conversion and upload happens without a problem even for the larger files, it's just the response I'm receiving on the client side that has been my headache.
UPDATE: I tried using the debugger in VS Code, and saw that the breakpoints are rightly hitting the res.end() inside the if loop which checks for existing file scenario, but for some strange reason, the angular controller isn't reacting to it. And this happens only for the larger files.

I figured it out myself after a long time. Turns out there's something like a buffer in Busboy.
Here, I used to link the busboy and express like this:
app.use(busboy());
Just setting the field called highWaterMark to an arbitrary size which would be greater than the file size did the trick.
app.use(busboy({
highWaterMark: 2048 * 1024 * 1024, // Set buffer size in MB
}));
I don't know what might have caused this problem, or what the new field does that solves it, but, it just works is all that I know. It would be helpful if someone could elaborate on this a bit.

Can't send sms with Twilio, Node.js and Sequelize

I have a program which respond automatically a sms when receive an income message.
It works when I don't use Sequelize to save data to the database. But when I add the code below, the twiml.message(msg) is never executed.
The problem is when call twiml.message(msg) inside the then it doesn't work. So how to resolve this problem? Thanks
Info.create(param)
.then(info => {
var msg = 'Numero: ' + info.id +
' Nom:' + info.nom +
' Date: ' + info.datesms);
var twiml = new MessagingResponse();
twiml.message(msg);
});

PhantomJS 2.0.0 - Select: Invalid argument error

The script below contains some URLs in "links" array. The function gatherLinks() is used to gather more URLs from sitemap.xml of the URLs in "links" array. Once the "links" array has enough URLs (decided by variable "limit"), function request() is called for each URL in "links" array to send a request to the server and fetch the response. Time taken for each response is reported. Total time taken by the program is reported when the program ends.
I wrote a PhantomJS program (source below) to send some requests and calculate the time taken (in order to compare the performance of 2.0.0 and 1.9.8). I get links using sitemap.xml file of the sites I hardcode in "links" array.
When run using PhantomJS 2.0.0, after some 65 requests the program (method page.open() of request function) starts outputting the following:
select: Invalid argument
select: Invalid argument
select: Invalid argument
select: Invalid argument
select: Invalid argument
.
.
.
.
When run using PhantomJS 1.9.8, it crashes after about 200 requests with the following error.
"PhantomJS has crashed. Please read the crash reporting guide at https://github.com/ariya/phantomjs/wiki/Crash-Reporting and file a bug report at https://github.com/ariya/phantomjs/issues/new with the crash dump file attached: /tmp/2A011800-3367-4B4A-A945-3B532B4D9B0F.dmp"
I tried to send the crash report but their guide is not very useful for me.
It's not the urls that I use, I have tried using other urls but same results.
Is there something wrong with my program? I am using OSX.
var system = require('system');
var fs = require('fs');
var links = [];
links = [
"http://somesite.com",
"http://someothersite.com",
.
.
.
];
var index = 0, fail = 0, limit = 300;
finalTime = Date.now();
var gatherLinks = function(link){
var page = require('webpage').create();
link = link + "/sitemap.xml";
console.log("Fetching links from " + link);
page.open(link, function(status){
if(status != "success"){
console.log("Sitemap Request FAILED, status: " + status);
fail++;
return;
}
var content = page.content;
parser = new DOMParser();
xmlDoc = parser.parseFromString(content, 'text/xml');
var loc = xmlDoc.getElementsByTagName('loc');
for(var i = 0; i < loc.length; i++){
if(links.length < limit){
links[links.length] = loc[i].textContent;
} else{
console.log(links.length + " Links prepared. Starting requests.\n");
index = 0;
request();
return;
}
}
if(index >= links.length){
index = 0;
console.log(links.length + " Links prepared\n\n");
request();
}
gatherLinks(links[index++]);
});
};
var request = function(){
t = Date.now();
var page = require('webpage').create();
page.open(links[index], function(status) {
console.log('Loading link #' + (index + 1) + ': ' + links[index]);
console.log("Time taken: " + (Date.now() - t) + " msecs");
if(status != "success"){
console.log("Request FAILED, status: " + status);
fail++;
}
if(index >= links.length-1){
console.log("\n\nAll links done, final time taken: " + (Date.now() - finalTime) + " msecs");
console.log("Requests sent: " + links.length + ", Failures: " + fail);
console.log("Success ratio: " + ((links.length - fail)/links.length)*100 + "%");
phantom.exit();
}
index++;
request();
});
}
gatherLinks(links[0]);
After playing around with the program, I couldn't find any particular pattern to the problems I mention below. For 2.0.0, I could only once succeed in sending 300 requests without an error. I have tried all different combinations of URLs, program usually fails between request 50-80. I maintain a log of urls that failed, all of them run fine when I send a single request using another PhantomJS program. For 1.9.8, it's much more stable and the crash I mention below is not very frequent. But again, I couldn't find any pattern to the crashing, it still crashes once in a while.

There are lots of problems with your code. The main one is probably that you're creating a new page for every single request and never close it afterwards. I think you're running out of memory.
I don't see a reason to create a new page for every request, so you can easily reuse a single page for all requests. Simply move the line var page = require('webpage').create(); to the global scope out of gatherLinks() and request(). If you don't want to do that, then you can call page.close() after you're done with it, but keep the asynchronous nature of PhantomJS in mind.
If the reason to use multiple page objects was to prevent cache re-use for later requests, then I have to tell you that this doesn't solve that problem. page objects in a single PhantomJS process can be regarded as tabs or windows and they share cookies and cache. If you want to isolate every request, then you will need to run every request in its own process for example through the use of the Child Process Module.
There is another problem with your code. You probably wanted to write the following in gatherLinks():
if(index >= links.length){
index = 0;
console.log(links.length + " Links prepared\n\n");
request();
return; // ##### THIS #####
}
gatherLinks(links[index++]);

Read email headers in Outlook Web Access (OWA)

I am developing a outlook Web App (Office 365 Developer). Regarding that, is there a way to read the headers of the selected mail which lays on inbox?. I am using Exchange server 2013. I would like to use Jquery or Javascript for write the code.
I tried to add "Message Header Analyzer" from Microsoft ( link:- 'https://store.office.com/message-header-analyzer-WA104005406.aspx?assetid=WA104005406'). Now it is working properly and it can read headers. But I need to implement the same functionality using my own codes.
If anyone can provide a good reference as a start, I would greatly appreciated that. (because, I got a great effort in searching google. But.. still no luck)
thanks in advance.

First of all, I would like to thank all the persons who responded to me to develop a solution to this. Special thanks should go to #FreeAsInBeer and MrPiao. After spending several days I was able to develop the following solution for getting mail headers. I removed all the unnecessary business logic from the code and finally came up with the following code. It can be used for reading the headers of inbox emails using JQuery.
I am making an EWS request outside to get the headers. From its callback method, I can retrieve the expected result. Afterwards, it is better to use jQuery.parseXML to read and manipulate the response (which is not included in the code)
I hope this explanation will help you.
var _mailbox;
var _ItemId1
(function () {
"use strict";
// The Office initialize function must be run each time a new page is loaded
Office.initialize = function (reason) {
$(document).ready(function () {
app.initialize();
_mailbox = Office.context.mailbox;
_ItemId1 = _mailbox.item.itemId;
});
};
})();
function getSelectedEmailHeaders() {
// Wrap an Exchange Web Services request in a SOAP envelope.
var var1 = '<?xml version="1.0" encoding="utf-8"?>';
var var2 = '<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">';
var var3 = ' <soap:Header>';
var var4 = ' <t:RequestServerVersion Version="Exchange2010" />';
var var5 = ' </soap:Header>';
var var6 = ' <soap:Body>';
var var7 = ' <m:GetItem>';
var var8 = ' <m:ItemShape>';
var var9 = ' <t:BaseShape>IdOnly</t:BaseShape>';
var var10 = ' <t:AdditionalProperties>';
var var11 = ' <t:FieldURI FieldURI="item:Subject" />';
var var12 = ' <t:FieldURI FieldURI="item:MimeContent" />';
var var13 = ' </t:AdditionalProperties>';
var var14 = ' </m:ItemShape>';
var var15 = ' <m:ItemIds>';
var var16 = ' <t:ItemId Id="' + _ItemId1 + '" />';
var var17 = ' </m:ItemIds>';
var var18 = ' </m:GetItem>';
var var19 = ' </soap:Body>';
var var20 = '</soap:Envelope>';
var envelopeForHeaders = var1 + var2 + var3 + var4 + var5 + var6 + var7 + var8 + var9 + var10 + var11 + var12 + var13 + var14 + var15 + var16 + var17 + var18 + var19 + var20;
//Calling EWS
_mailbox.makeEwsRequestAsync(envelopeForHeaders, callbackForHeaders);
}
//This Function called when the EWS request is complete.
function callbackForHeaders(asyncResult) {
//Write the content of the asyncResult on console
console.log(asyncResult);
}
Thank You. Kushan Randima

retry sql connection using JavaScript

I know using javascript is not the best way to connect to a SQL server but this is for an in-house application. I connect using the following:
dbNSConnection = new ActiveXObject("ADODB.Connection") ;
var sNSConnectionString="Driver={SQL Server};TrustedConnection=Yes;Server=" + sNSServer + ";Database=" + sNSDatabase + ";UID=" + sNSUID + ";PWD=" + sNSPWD;
dbNSConnection.Open(sNSConnectionString);
How can I make sure connection has gone thru and how do I retry if not connected?

Isn't there any other way you could get this done on the codebehind, and pass the results to the javascript? This may be an example of what I'm talking about originally.
If not, can't you simply do the following?
try
{
dbNSConnection = new ActiveXObject("ADODB.Connection") ;
var sNSConnectionString="Driver={SQL Server};TrustedConnection=Yes;Server=" + sNSServer + ";Database=" + sNSDatabase + ";UID=" + sNSUID + ";PWD=" + sNSPWD;
dbNSConnection.Open(sNSConnectionString);
//Run rest of query here
}
catch(err)
{
var message = err.message;
//Finish building errors message here
}