I am trying to parse an API response from a RETS server that is image data in a multipart format. I am trying to manually parse the data using the code below and write the image data to files, but when I go to open the image file I get a message saying that the file type is unsupported. Any help I can get is greatly appreciated.
Code to make the API call and parse the data:
let image_res = await axios.get(url,
{
// responseType: 'arraybuffer',
headers: {
'Accept': '*/*',
'Authorization': 'Bearer ' + token,
'Connection': 'keep-alive',
'Content-Type': 'application/x-www-form-urlencoded',
'Accept-Encoding': 'gzip, deflate, br'
}
}
)
var boundary = image_res.headers['content-type'].split(';')[1].split('=')[1];
let parts = image_res.data.split(boundary)
for (var i = 0; i < parts.length; i++) {
var dirty_img = parts[i].split('Content-Type: image/jpeg')[1]
if (dirty_img) {
var img = dirty_img.trim();
img = img.replace(/--/g, '');
console.log("Image: ", img)
fs.writeFile(`./LongLeaf/images/556354089-${i}.jpg`, img, { encoding: 'binary', flag: 'w+' }, err => {
if (err) {
console.log(err)
return
}
})
}
}
Screenshot of the data returned from the API call:
This pattern continues throughout the data where each image will have a separator similar to --TRM44afa5fc8b3d4651b3205064a794c38a and then Content-ID, Object-ID, Order-Hint, Content-Type, a blank line, and then the image data.
I tried using the parse-multipart-data npm package but it was returning an empty array each time. I initially thought the issue was that there was no filename for the images but I wrote code to add in a filename and still get an empty array.
Related
recently I got an issue with uploading documents functionality in Expo CLI - React native, the issue is the file is not delivering to the backend like FormData() it arrives in a weird array, not like the website at all, I got the same endpoint connected to React JS project and it works fine and the browser delivers the documents correctly,
But not sure what is wrong with FromData here,
I also tried react-native-fs but it's not compatible with Expo
and here is my code
let formData = new FormData();
formData.append('file', doc);
let token = this.props.user.token;
let header = { headers: { 'Accept': '*', 'Authorization': 'Bearer '+token, 'Content-Type': 'multipart/form-data' } };
let res = await axios.post(BackendURL+'/porta/files/upload/'+document.projectDocumentId, formData, header);
if(res.data.success === true){
alert('File uploaded successfully');
this.props.ReloadData(true);
}
console.log(doc);
Issue has been solved,
the problem where in the header, i removed the content type & accept like this
let formData = new FormData();
formData.append('file', {
name: doc.name,
uri: doc.uri,
type: "application/pdf"
});
this.props.loadingStatus(true);
let token = this.props.user.token;
let header = { headers: { 'Authorization': 'Bearer '+token } };
let res = await axios.post(BackendURL+'/portal/files/upload/'+document.projectDocumentId, formData, header);
I am trying to upload a file from a react front end to a C# backend. I am using drop zone to get the file and then I call an api helper to post the file but I am getting different errors when I try different things. I am unsure exactly what the headers should be and exactly what I should send but I get two distinct errors. If I do not set the content-type I get 415 (Unsupported Media Type) error. If I do specify content type as multipart/form-data I get a 500 internal server error. I get the same error when the content-type is application/json. The url is being past in and I am certain it is correct. I am unsure if the file should be appended as file[0][0] as I have done or as file[0] as it is an array but I believe it should be the first. Any suggestions welcome :) Here is my api post helper code:
export const uploadAdminFile = (file, path, method = 'POST', resource =
config.defaultResource) => {
const url = createUrl(resource, path);
const data = new FormData();
data.append('file', file[0][0]);
data.append('filename', file[0][0].name);
const request = accessToken =>
fetch(
url,
{
method,
mode: 'cors',
withCredentials: true,
processData: false,
headers: {
Accept: 'application/json',
'Content-Type': 'application/json', //'multipart/form-data',
Authorization: `Bearer ${accessToken}`,
},
body: data,
})
.then(res => res.json())
.then(success => console.log('API HELPER: file upload success: ', success)
.catch(err => console.log('API HELPER: error during file upload: ', err)));
return sendRequest(request, resource);
};
Thanks for the help and suggestions, it turned out to be a backend issue but even still I learned a lot in the process. I will post my working code here in case anyone comes across this and finds it useful.
export const uploadAdminFile = (file, path, resource=config.defaultResource) => {
const url = createUrl(resource, path);
const formData = new FormData();
formData.append('file', file[0][0]);
formData.append('filename', file[0][0].name);
const request = accessToken =>
fetch(url,
{
method: 'POST',
headers: {
Accept: 'application/json',
Authorization: `Bearer ${accessToken}`,
},
body: formData,
});
return sendRequest(request, resource);
};
As mentioned, the file name does not need to be sent separately and count be omitted. I am indexing the file this way because I get it from dropzone as an array and I only want a single file (the first one in the array). I hope this helps someone else out and here is a link to the mdn fetch docs (good information) and a good article on using fetch and formData.
I'm trying to upload an image via Slack using node.js and the request package, but not having much luck. Either I receive invalid_array_arg or no_file_data errors from the API.
Here is my request:
var options = { method: 'POST',
url: 'https://slack.com/api/files.upload',
headers:
{ 'cache-control': 'no-cache',
'content-type': 'application/x-www-form-urlencoded' },
form:
{ token: SLACK_TOKEN,
channels: SLACK_CHANNEL,
file: fs.createReadStream(filepath)
} };
request(options, function (error, response, body) {
if (error) throw new Error(error);
console.log(body);
});
I had a look at a few relevant posts:
Can I upload an image as attachment with Slack API?
Slack API (files.upload) using NodeJS
fix files.upload from Buffer with formData options #307
The only thing that worked was using the curl command directly, but using cygwin (CommandPrompt failed: curl: (1) Protocol https not supported or disabled in libcurl). The issue calling curl from node (using child_process) but that silently fails in Command Prompt and still returns no_file_data using cygwin (passing an absolute path to the file):
stdout: {"ok":false,"error":"no_file_data"}
stderr: % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 469 100 35 100 434 359 4461 --:--:-- --:--:-- --:--:-- 6112
I'm using node v6.9.1 on Windows.
What am I missing ? How can I upload an image to slack via node.js on Windows ?
The Slack API error invalid_array_arg means that there is a problem with the format of the arguments passed to Slack. (see here)
When using the file property for files.upload, Slack excepts the data as multipart/form-data, not as application/x-www-form-urlencoded. So instead of form, you need to use formData in your request object. I also removed the incorrect part in the header.
This works:
var fs = require('fs');
var request = require('request');
var SLACK_TOKEN = "xoxp-xxx";
var SLACK_CHANNEL = "general";
var filepath = "file.txt";
var options = { method: 'POST',
url: 'https://slack.com/api/files.upload',
headers:
{ 'cache-control': 'no-cache' },
formData:
{ token: SLACK_TOKEN,
channels: SLACK_CHANNEL,
file: fs.createReadStream(filepath)
} };
request(options, function (error, response, body) {
if (error) throw new Error(error);
console.log(body);
});
In this sample script, it supposes to upload a binary file (zip file). When you use this, please modify for your environment. When files are uploaded to Slack, multipart/form-data is used. In my environment, there were the situations that files couldn't be uploaded by some libraries. So I created this. If this is useful for your environment, I'm glad.
Users can upload the binary file by converting byte array as follows.
At first, it builds form-data.
Adds the zip file converted to byte array and boundary using Buffer.concat().
This is used as body in request.
The sample script is as follows.
Sample script :
var fs = require('fs');
var request = require('request');
var upfile = 'sample.zip';
fs.readFile(upfile, function(err, content){
if(err){
console.error(err);
}
var metadata = {
token: "### access token ###",
channels: "sample",
filename: "samplefilename",
title: "sampletitle",
};
var url = "https://slack.com/api/files.upload";
var boundary = "xxxxxxxxxx";
var data = "";
for(var i in metadata) {
if ({}.hasOwnProperty.call(metadata, i)) {
data += "--" + boundary + "\r\n";
data += "Content-Disposition: form-data; name=\"" + i + "\"; \r\n\r\n" + metadata[i] + "\r\n";
}
};
data += "--" + boundary + "\r\n";
data += "Content-Disposition: form-data; name=\"file\"; filename=\"" + upfile + "\"\r\n";
data += "Content-Type:application/octet-stream\r\n\r\n";
var payload = Buffer.concat([
Buffer.from(data, "utf8"),
new Buffer(content, 'binary'),
Buffer.from("\r\n--" + boundary + "\r\n", "utf8"),
]);
var options = {
method: 'post',
url: url,
headers: {"Content-Type": "multipart/form-data; boundary=" + boundary},
body: payload,
};
request(options, function(error, response, body) {
console.log(body);
});
});
I'm trying to scrape a web-page for some data and I managed to post a request and got the right data. The problem is that I get something like :
"Kannst du bitte noch einmal ... erzýhlen, wie du wýhrend der Safari einen Lýwen verjagt hast?"
normally erzählen - während, so Ä,Ö,ß,Ü are not showing correctly.
here is my code:
var querystring = require('querystring');
var iconv = require('iconv-lite')
var request = require('request');
var fs = require('fs');
var writer = fs.createWriteStream('outputBodyutf8String.html');
var form = {
id:'2974',
opt1:'',
opt2:'30',
ref:'A1',
tid:'157',
tid2:'',
fnum:'2'
};
var formData = querystring.stringify(form);
var contentLength = formData.length;
request({
headers: {
'Content-Length': contentLength,
'Content-Type': 'application/x-www-form-urlencoded'
},
uri: 'xxxxxx.php',
body: formData,
method: 'POST'
}, function (err, res, body) {
var utf8String = iconv.decode(body,"ISO-8859-1");
console.log(utf8String);
writer.write(utf8String);
});
how to get the HTML body in with the correct letters?
How do I find out the correct encoding of a response?
I went to the website you are attempting to scrape, and found this:
And another character encoding declaration here:
This website defined two different charater encodings! Which do I use?
Well, this doesn't apply to you.
When reading an HTML file from a local machine, then the charset or content-type defined in the meta tags will be used for encoding.
Since you are retrieving this document, over HTTP, the files will be encoded according to the response header.
Here's the reponse header I received after visiting the website.
As you can see, they don't have a defined character set. It should be located in the Content-Type property. Like this:
Since they don't have any indicated charset in the response header, then, according to this post, it should use the meta declaration.
But wait, there was two meta charset declarations.
Since the compiler reads the file top to bottom, the second declared charset should be used.
Conclusion: They use UTF-8
Also, I don't think you need the conversion. I may be wrong, but you should just be able to access the response.
request({
headers: {
'Content-Length': contentLength,
'Content-Type': 'application/x-www-form-urlencoded'
},
uri: 'xxxxxx.php',
body: formData,
method: 'POST'
}, function (err, res, body) {
console.log(body);
writer.write(body);
});
Edit: I don't believe the error is on their side. I believe it's on your side. Give this a try:
Remove the writer:
var writer = fs.createWriteStream('outputBodyutf8String.html');
And in the request callback, replace everything with this:
function (err, res, body) {
console.log(body);
fs.writeFile('outputBodyutf8String.html', body, 'utf8', function(error) {
if(error)
console.log('Error Occured', error);
);
}
All the code should look like this:
var querystring = require('querystring');
var iconv = require('iconv-lite')
var request = require('request');
var fs = require('fs');
var form = {
id:'2974',
opt1:'',
opt2:'30',
ref:'A1',
tid:'157',
tid2:'',
fnum:'2'
};
var formData = querystring.stringify(form);
var contentLength = formData.length;
request({
headers: {
'Content-Length': contentLength,
'Content-Type': 'application/x-www-form-urlencoded'
},
uri: 'xxxxxxx.php',
body: formData,
method: 'POST'
}, function (err, res, body) {
console.log(body);
fs.writeFile('outputBodyutf8String.html', body, 'utf8', function(error) {
if(error)
console.log('Error Occured', error);
);
}
I am trying to upload files to Google Drive in Angular 2. So far I am able to upload files, but without title and they are "Untitled"
Here is code to do that:
gDriveUploader(file): Promise<any> {
let authToken = tokenHere;
const url = `https://www.googleapis.com/upload/drive/v2/files/`
let formData:FormData = new FormData();
formData.append('title', file, file.name);
let headers = new Headers({
'Authorization': 'Bearer ' + authToken
});
headers.append('Accept', file.type);
let options = new RequestOptions ({
headers: headers,
});
console.log('OPTIONS: ', options)
return this.http.post(`${url}`, formData, options)
.toPromise()
.then(response => response.json())
.catch(e=>console.log(e));
}
I know, that in order to send metadata with file, I have to add this metadata to Request body and use at multipart or resumable upload types. But here I faced issue after issue and just can't make it properly.
I completely messed up. Here is on of my approaches with resumable upload type:
gDriveUploader(file): Promise<any> {
let authToken = token;
const url = `https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable`
console.log('FILE TO UPLOAD: ', file)
let formData:FormData = new FormData();
formData.append('name', file, file.name);
let headers = new Headers({
'Authorization': 'Bearer ' + authToken,
'Content-Type': 'application/json; charset=UTF-8', //if remove "Bad Content" Error
//'Content-Length': file.size, not sure if this one right?
});
let options = new RequestOptions ({
headers: headers,
});
return this.http.post(`${url}`, formData, options)
.toPromise()
.then(response => response.json())
.catch(e=>console.log(e));
}
that's not only two of my approaches...
According to Drive API for resumable upload:
POST https://www.googleapis.com/drive/v3/files?uploadType=resumable
HTTP/1.1
Authorization: Bearer [YOUR_AUTH_TOKEN]
Content-Length: 38
Content-Type: application/json; charset=UTF-8
X-Upload-Content-Type: image/jpeg
X-Upload-Content-Length: 2000000
What is Content-Length: 38? it's file length and I can just use file.size?
With multipart I can't figure out how to add those boundary separator in the request.
I saw some Q and A, that multipart were not supported by Angular, but that was 1-2 year ago. What about now?
Can I somehow use resumable upload to GDrive with additional file metadata using standard Angular features?
So. A bit more research on how API works. I came up with the following solution for resumable file upload. Main Idea, that first time I have to make a request and "set metadata" for my file and get response with the link, where to upload the file. And this link came as one of the response header called location.
Here is fully working code. Just pass File object to the first function.
I just quickly made 2 functions for this. First one will set metadata (just name) and call second function to upload just binary data.
gDriveUploader(file): Promise<any> {
let authToken = token
const url = `https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable`
let headers = new Headers({
'Authorization': 'Bearer ' + authToken,
'Content-Type': 'application/json; charset=UTF-8',
});
let options = new RequestOptions ({
headers: headers,
});
return this.http.post(`${url}`, {name: file.fullName}, options) //just set the name
.toPromise()
.then(response => this.gDriveUploadFile(file, response.headers.get('location'))) //call second function to upload `file` to proper URI from response
.then(response => {
let id = response.json().id //parse id of uploaded file
let resp = {fileName: file.fullName, fileType: file.fileType, fileSize: file.size, fileId: id} //create an object with file file properties, if you need that
return resp // return object back to function that called this service
})
.catch(e=>console.log(e));
}
Second function to upload data:
gDriveUploadFile(file, url): Promise<any> { //file and url we got from first func
let authToken = token
let headers = new Headers({
'Authorization': 'Bearer ' + authToken,
'Content-Type': 'application/json; charset=UTF-8',
'X-Upload-Content-Type': file.type
});
let options = new RequestOptions ({
headers: headers,
});
return this.http.post(`${url}`, file, options) //call proper resumable upload endpoint and pass just file as body
.toPromise()
}
Maybe solution not ideal, so far I do not handle errors here and do not use resumable features like uploading by chunks, just upload file at once. But hopefully if someone else stuck with GDrive uploading can get an idea.