Develop Reference

Lambda download and forward PDF (PDF proxy)

i'm wondering what am i doing wrong with this lambda function.
Goal:
Send http options to fetch an PDF and forward it to consumer from Lambda service.
Current code:
"use strict";
const http = require("http");
function getPDF(options, event) {
console.log(options);
return new Promise((resolve, reject) => {
let body = "";
let statusCode = 0;
let headers = { };
http
.request(options, (res) => {
statusCode = res.statusCode;
const headersFromReq = res.headers || {};
res.on("data", (chunk) => (body += chunk));
res.on("end", function () {
console.log( statusCode, headers, body);
resolve({
body: Buffer.from(body).toString(),
statusCode,
headers: {
...headersFromReq,
//'Content-type': 'application/pdf',
//'content-disposition': 'attachment; filename=test.pdf'
}
});
})
.on("error", reject)
.end();
});
});
}
exports.handler = async (event) => {
try {
const response = await getPDF(event.options, event);
return response;
} catch (error) {
console.error(error);
return {
statusCode: 500,
body: JSON.stringify(error),
headers: {}
};
}
};
Whatever i've tried, it either times out or does not result in the actually needed response of Base64 encoded PDF.
Params for testing would look something like this:
{
"options": {
"hostname": "www.africau.edu",
"port": 80,
"path": "images/default/sample.pdf",
"method": "GET",
"headers": {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36",
"Accept": "application/pdf",
"Accept-encoding": "gzip, deflate, br"
}
}
}
Current logs -
Function Logs
START RequestId: 8d6be86c-788d-4f49-8305-8caf377cd32e Version: $LATEST
2021-09-28T09:01:21.507Z 8d6be86c-788d-4f49-8305-8caf377cd32e INFO {
hostname: 'www.africau.edu',
port: 80,
path: 'images/default/sample.pdf',
method: 'GET',
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36',
Accept: 'application/pdf',
'Accept-encoding': 'gzip, deflate, br'
}
}
END RequestId: 8d6be86c-788d-4f49-8305-8caf377cd32e
REPORT RequestId: 8d6be86c-788d-4f49-8305-8caf377cd32e Duration: 11011.54 ms Billed Duration: 11000 ms Memory Size: 128 MB Max Memory Used: 54 MB Init Duration: 152.43 ms
2021-09-28T09:01:32.494Z 8d6be86c-788d-4f49-8305-8caf377cd32e Task timed out after 11.01 seconds

Your approach has an underlying conceptual problem - it may take time to execute. The time that you don't have when you run things in lambda. Your lambda "technically" has the maximum of 15 minutes to finish the execution (although you explicitly have to configure it. I think by default it's 10s), but if you trigger it from AWS API Gateway, that goes down to 30 seconds and this is not a limit you can configure. It's the total max. Moreover your lambda response cannot be larger than 6MB and it is normally supposed to be JSON, so you would have to convert your file to Base64, but again, if you serve that file via api gateway even that limit goes down once again... What you're trying just cannot be done reliably with lambda in this way. There is a different way however that would actually be recommended by AWS.
You send a request to API Gateway that triggers a lambda
Lambda looks up if the requested file already exists in S3
If it doesn't exist:
The lambda downloads a file and puts it into S3. Note that you can now set up S3 bucket policy so that file stays in S3 only for certain amount of time. You probably don't want to keep it there forever, but it's nice to keep it cached for a while in case the user tries to re-download your PDF. This way they will be able to get the response much faster
The lambda then generates a pre-signed S3 URL to the freshly downloaded file (a special URL that you can request from S3 that will be valid for another few minutes only) and returns it in the response
If it already exists:
the lambda just generates the pre-signed S3 URL and returns it in the response
Your client (UI application I presume) has to generate a consecutive request to the pre-signed url received in the response (so it talks directly to S3). This way, even if your user has slow internet connection and they need 20 minutes to download the file, you don't get any timeouts... well you will still get some if the file is really large and the lambda cannot download it quickly enough, but that would require a longer discussion. In this case I'm assuming your file is under 15MB.

Express Node.js response to AJAX call from client does not send fully

im using an ajax get request to get data from the server (a spotify playlist structure, to be displayed for the user. But I'm having some trouble with transmitting. This issue is only prevelent when using the external ip (and not with localhost)
The data being received correctly on localhost:
with headers:
Request URL: http://www.localhost:8080/spotify/ajax/requestSongs
Request Method: GET
Status Code: 200 OK
Remote Address: [::1]:8080
Referrer Policy: no-referrer-when-downgrade
Content-Length: 821946
Content-Type: application/json; charset=utf-8
Date: Tue, 14 Apr 2020 19:18:51 GMT
ETag: W/"c8aba-7NdUbhReuN0XXnw3DPVSyrNAS38"
X-Powered-By: Express
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7
Connection: keep-alive
Cookie: connect.sid=s%3A0yZXFKy7MgsfbY_OX5yanp5HKlvdSGKY.8BaJrUmEGn9z9ohyRTtkXBCTj6wxiDDoaLNFS8vEdl8
Host: www.localhost:8080
If-None-Match: W/"c8aba-7NdUbhReuN0XXnw3DPVSyrNAS38"
Referer: http://www.localhost:8080/spotify/edit?playlist=5LDRNl7lGrxm7mx2RVggVC
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36
The data being incorrectly received on external ip:
with headers:
Request URL: http://86.90.233.152:8080/spotify/ajax/requestSongs
Request Method: GET
Status Code: 200 OK
Remote Address: MYIP:8080
Referrer Policy: no-referrer-when-downgrade
Connection: keep-alive
Content-Length: 821946
Content-Type: application/json; charset=utf-8
Date: Tue, 14 Apr 2020 19:03:00 GMT
ETag: W/"c8aba-7NdUbhReuN0XXnw3DPVSyrNAS38"
X-Powered-By: Express
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7
Connection: keep-alive
Cookie: connect.sid=s%3AlGbD5iWpq9sJlLCFvBsj20RkR9_gZrEr.D9rOpXr5M6FfHnUe7DUx0xzLybQhsGsLIuYsys0FrP4
Host: 86.90.233.152:8080
Referer: http://MYIP:8080/spotify/edit?playlist=5LDRNl7lGrxm7mx2RVggVC
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36
The ajax response doesn't change after the second XMLhttp reply.
Furthermore, to try and find the issue I log every response coming into ajax and the console is:
clientscript.js:8 XMLHttpRequest {readyState: 1, timeout: 0, withCredentials: false, upload: XMLHttpRequestUpload, onreadystatechange: ƒ, …}
clientscript.js:8 XMLHttpRequest {readyState: 2, timeout: 0, withCredentials: false, upload: XMLHttpRequestUpload, onreadystatechange: ƒ, …}
clientscript.js:8 XMLHttpRequest {readyState: 3, timeout: 0, withCredentials: false, upload: XMLHttpRequestUpload, onreadystatechange: ƒ, …}
clientscript.js:8 XMLHttpRequest {readyState: 3, timeout: 0, withCredentials: false, upload: XMLHttpRequestUpload, onreadystatechange: ƒ, …}
SOURCE:
Server side replier
router.get("/ajax/requestSongs", function(req,res, next)
{
if(req.session.lastPlaylist)
{
localPlaylists[res.locals.sessionId].getSortedGenres((sortedGenres) => localPlaylists[res.locals.sessionId].getPlaylist( (playlist) =>
{
var data={sortedGenres : sortedGenres, playlist : playlist}
console.log(data);
res.send(data);
}));
}
});
Client side receiver:
function getData(suburl, callback)
{
var xhttp = new XMLHttpRequest();
xhttp.onreadystatechange = function() {
//console.log(this.responseText);
console.log(this)
if (this.readyState == 4 && this.status == 200)
{
console.log(this.responseText);
callback(this.responseText);
}
};
xhttp.open("GET", pathPrefix+"/ajax/"+suburl, true);
xhttp.send();
}
An example of how data is cut off (the ....... is correct data):
........"art rock","blues rock","classic rock","hard rock","metal","psychedelic rock","rock","soft rock"]},{"artists":[{"external_urls":{"spotify":"https://open.s

Because it only makes a difference when you're on an external link and the data is large (not huge, but not small either), I suspect that there's some sort of problem with the transmission, perhaps some sort of internal buffer overflow problem. I was under the impression that res.send() is supposed to just "handle that for you", but apparently that is not the case.
I would first add an error handler on the output stream and see if it logs anything:
router.get("/ajax/requestSongs", function(req, res, next)
{
if(req.session.lastPlaylist)
{
localPlaylists[res.locals.sessionId].getSortedGenres((sortedGenres) => localPlaylists[res.locals.sessionId].getPlaylist( (playlist) =>
{
var data={sortedGenres : sortedGenres, playlist : playlist}
console.log(data);
// log any errors on the response stream
res.on('error', function(err) {
console.log(err);
});
res.send(data);
}));
}
});
Then, my theory of some sort of buffer overflow would suggest that you need to implement flow control upon sending of the data. I thought res.send() was supposed to do that for you, but you can implement it yourself like this and see if it solves the problem:
router.get("/ajax/requestSongs", function(req, res, next) {
if(req.session.lastPlaylist) {
localPlaylists[res.locals.sessionId].getSortedGenres((sortedGenres) => localPlaylists[res.locals.sessionId].getPlaylist( (playlist) => {
res.type("application/json");
res.status(200);
const data = JSON.stringify({sortedGenres : sortedGenres, playlist : playlist});
const chunkSize = 1024 * 10; // send in 10k chunks
let pos = 0, chunk;
function sendNextChunk() {
if (pos < data.length) {
chunk = data.substr(pos, chunkSize);
pos += chunkSize;
let flushed = res.write(chunk);
if (!flushed) {
// need to wait for drain event before sending more
res.once('drain', sendNextChunk);
} else {
// send more without stack build-up
setImmediate(sendNextChunk);
}
} else {
// finish the response
res.end();
}
}
// log any write errors on the response
res.on('error', function(err) {
console.log(err);
});
// start the sending sequence
sendNextChunk();
}));
} else {
// have to send something here when no playlist
res.send("No previously established playlist");
}
});

Upload file into Spark API endpoint

I'm trying to upload a file to a Spark Api with and enpoint like the following:
after((Filter) (request, response) -> {
response.header("Access-Control-Allow-Origin", "*");
response.header("Access-Control-Allow-Methods", "POST");
});
post("/scene", (request, response) -> {
Path tempFile = Files.createTempFile(uploadDir.toPath(), "", "");
request.attribute("org.eclipse.jetty.multipartConfig", new MultipartConfigElement("/temp"));
try (InputStream is = request.raw().getPart("uploaded_file").getInputStream()) {
Files.copy(is, tempFile, StandardCopyOption.REPLACE_EXISTING);
DAOIntf dao = new DAO(tempFile.toAbsolutePath().toString(),"map.xml");
Scene scene = dao.getXmlScene();
return new Gson().toJson(scene);
}catch (Exception e){
return e;
}
});
By using fetch from a React app like this:
handleFileChange(e) {
e.preventDefault();
let fileInput = document.getElementsByName("uploaded_file")[0];
fetch(apiAddress, {
method: 'POST',
body: fileInput.files[0]
}).then( (res) => {
return res.json()
}).then( (res) => {
this.props.onFileChange(JSON.stringify(res));
});
}
I know the API is working because i tried it with a simple form in plain html. So I'm guessing the problem is related with how I build the payload.
The way it is I'm getting a 404 and this message at the server logs:
INFO spark.http.matching.MatcherFilter - The requested route [/scene] has not been mapped in Spark
This is the package I'm sending
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: es-ES,es;q=0.9,en;q=0.8,en-GB;q=0.7
Access-Control-Request-Headers: content-type
Access-Control-Request-Method: POST
Connection: keep-alive
Host: localhost:4567
Origin: http://localhost:3000
Referer: http://localhost:3000/
--UPDATE--
Im using this component to activate the call:
render(){ return (
<form>
<input type='file' name='uploaded_file' ref={input => {
this.fileInput = input;
}}/>
<button onClick={this.handleFileChange}>Upload</button> </form> ) }
If I use it with out selecting a file, I get a OK 200 and a javax.servlet.ServletException: Content-Type != multipart/form-data exception in the server. So the communication is effective. Still don't now what IS the problem tho.

Requesting font awesome file from nodeJs is sending back wrong data/file

my application: When you send a request from a browser to my node server, my node server will request an origin website, download all of its static files (including code) and server them back to the user. Next time you visit my node server it will server all the content back from node instead of requesting the origin.
When i make a request for a font awesome file from node
http://example.com/modules/megamenu/fonts/fontawesome-webfont.woff?v=4.2.0
The file's content is different from when i request the same url with cUrl.
This is causing this error in the browser when i return the file from node back to the browser:
Failed to decode downloaded font: http://nodeDomain.test/modules/megamenu/fonts/fontawesome-webfont.woff?v=4.2.0
If i copy and paste the content from the file i requested via curl into the file stored on my node server, the error disappears and all the font awesome stuff works.
Here are the headers I am sending with the request to the origin server from node.
{
connection: 'keep-alive',
pragma: 'no-cache',
'cache-control': 'no-cache',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36',
accept: '*/*',
referer: 'http://example.com/modules/megamenu/css/font-awesome.min.css',
'accept-language': 'en-US,en;q=0.8',
cookie: 'PrestaShop-a30a9934ef476d11b.....'
}
I tried to see what headers where being sent when doing the curl request from command line but i cannot figure out how to do it.
______Node code used to fetch file_______
Url: in options is the one stated above
headers: are the browsers request headers
var options = {
url: originRequestPath,
headers: requestHeaders
}
var originPage = rquest(options);
var responseBody = '';
var resHeads = '';
originPage.on('response', function(res)
{
//store response headers locally
}
originPage.on('data', function(chunk)
{
responseBody += chunk;
});
originPage.on('end', function()
{
storeData.storePageData(storeFilePath, responseBody);
});
__________Store Function below________________
exp.storePageData = function(storePath, pageContent)
{
fs.outputFile(storePath, pageContent, function(err) {
if(err){ console.log(err)}
});
}

I believe the problem with your code is you are converting your buffer output to utf8 string. since you are adding buffer with empty string responseBody += chunk; that buffer is converted to utf-8 string. Thus you are losing some data for binary files. Try this way:
var originPage = rquest(options);
var chunks = []
originPage.on('response', function(res)
{
//store response headers locally
}
originPage.on('data', function(chunk)
{
chunks.push(chunk)
});
originPage.on('end', function()
{
var data = Buffer.concat(chunks)
//send data to browser and store content locally
});

Node.js / Socket io video streaming does not seem to work while any other file type works

Okay so i have the following route for when a file is being uploaded:
router.route('/moduleUpload')
.post(function (request, response) {
request.files.file.originalname = request.files.file.originalname.replace(/ +?/g, '');
var media = new Media(request.files.file, './user_resources/module/' + request.body.module_id + '/');
if (!fs.existsSync(media.targetDir)) {
fs.mkdirSync(media.targetDir, 0777, function (err) {
if (err) {
console.log(err);
response.send("ERROR! Can't make the directory! \n"); // echo the result back
}
});
}
moveFile(media);
var token = jwt.encode({
mediaObject: media
}, require('../secret')());
response.status(200).json(token);
});
As you can see this uses the support of a function called moveFile(media);
So here is that function:
function moveFile(media) {
var source = fs.createReadStream(media.file.path);
var dest = fs.createWriteStream(media.targetDir + media.file.originalname);
source.pipe(dest);
source.on('end', function () { /* copied */
console.log('file has been moved!')
});
source.on('error', function (err) { /* error */
console.log('error while moving file')
});
}
This works and the console prints out:
file has been moved!
Now after this i attempt to download / stream the video file i use the following route:
router.route('/resource/:encodedString')
.all(function (req, res) {
var decoded = jwt.decode(req.params.encodedString, require('../secret')());
var mediaObject = decoded.mediaObject;
res.header('content-disposition', 'filename=' + mediaObject.file.originalname, "Content-Length: " + mediaObject.file.size, "mimeType:" + mediaObject.file.mimetype);
var stream = fs.createReadStream(mediaObject.targetDir + mediaObject.file.originalname);
stream.pipe(res);
stream.on('end', function () {
console.log("Reading operation completed.");
res.end();
});
});
Using the encoded string i first decode it to find the media object and then stream it back.
This works WITH EVERY file execpt mp4 / video files.
Here is the complete console print out of when i upload a video:
listening on *:8105
file has been moved!
Reading operation completed.
And here are my browser request headers:
Remote Address:***
Request ****
Request Method:GET
Status Code:200 OK
Response Headers
view source
Access-Control-Allow-Credentials:true
Access-Control-Allow-Origin:http://localhost
Connection:keep-alive
Content-Length:5
Content-Type:application/octet-stream
Date:Sun, 21 Jun 2015 17:13:24 GMT
Set-Cookie:io=1T7R1vmJ3xV-k0ApAABH
Request Headers
view source
Accept:*/*
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Cookie:io=1T7R1vmJ3xV-k0ApAABH; lb_login_id=117
Host:angular.learningbankapp.com:8105
Origin:http://localhost
Referer:http://localhost/learningbankapp/
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36
So can someone tell me what im doing wrong?