I'm using VueJS and Cypress. I have a modal where I submit a FormData with different filled and a file:
var formData = new FormData();
formData.append("document_file", this.documentFile);
formData.append("comments", this.comments.value);
...
They way I upload it:
this.$http.post('http://localhost:8081/api/upload',formData,{emulateJSON: true},{
header:{ "Content-Type":"multipart/form-data" },
}).then(function(response) {
// ... Code
}).catch((err) => {
// .. Code
});
}
I want to use Cypress to test the params. Now, when I don't use the document file in the formData, I have the following methods to parse the multipart/form-data; boundary=---:
function parse(request) {
console.log("request");
console.log(request);
const headers = request.headers;
const body = request.body;
const content_type = headers['content-type'];
expect(content_type, 'boundary').to.match(/^multipart\/form-data; boundary=/);
const boundary = content_type.split('boundary=')[1];
const values = p(boundary, body);
return values;
}
function p(boundary, body) {
expect(boundary, 'boundary').to.be.a('string');
expect(body, 'body').to.be.a('string');
const parts = body.split(`--${boundary}`).map((s) => s.trim()).filter((s) => s.startsWith('Content-Disposition: form-data;'));
const result = {};
parts.forEach((part) => {
const lines = part.split(/\r?\n/g);
const key = lines[0].match(/name="(.+)"/)[1];
result[key] = (lines.length >= 2) ? lines[2].trim() : "";
});
return result;
}
Which work great. But when I upload the file, I get a different request.body:
They way I'm trying to test:
cy.get('#createDocument').then((interception) => {
assert.isNotNull(interception);
const values = parse(interception.request);
assert.isTrue(values.comments === "");
});
How can I handle this?
Related
I'm trying to upload a base64 image from Angular to ExpressJS. I'm using html2canvas to create the base64 image. If I try and upload imageData in it's current form I get
imageData.replace is not a function. If I try stringifying it like this in angular service
const image = JSON.stringify(imageData);
const data = image.replace(/^data:image\/\w+;base64,/, '');
then I get Buffer is not a constructor
How can I get it to upload to ExpressJS server successfully? I appreciate any help!
component
ngAfterViewInit() {
sleep(5000).then(() => {
const table = document.getElementById('table');
this.dataUrl = html2canvas(table).then(function (canvas) {
const res = canvas.toDataURL();
return res;
});
this.angularService.uploadImage('png', this.dataUrl);
});
}
service
uploadImage(contentType, imageData) {
console.log("imageData", imageData)
const headers = new HttpHeaders();
if (contentType === 'jpeg') {
headers.set('Content-Type', 'image/jpeg;');
} else if (contentType === 'png') {
headers.set('Content-Type', 'image/jpeg;');
}
const data = imageData.replace(/^data:image\/\w+;base64,/, '');
const buff = new Buffer(imageData, 'base64');
return this.http.put(
environment.slsLocal + '/update-image-url',
buff.buffer,
{ headers: headers }
);
}
console.log('imageData', imageData) in service looks like this
You have 2 issues.
html2canvas(table).then returns promise. Not res. You have to call this.angularService inside html2canvas(table).then.
I am not sure what you are trying to do with const buff = new Buffer(imageData, 'base64');. If you want to upload the base64 contents, just put data.
Also please note that you have to subscribe the return of this.http.put because HTTP request will not thrown to your server unless it is subscribed.
Put it together.
component
ngAfterViewInit() {
sleep(5000).then(() => {
const table = document.getElementById('table');
html2canvas(table).then(function (canvas) {
const res = canvas.toDataURL();
this.dataUrl = res;
this.angularService.uploadImage('png', this.dataUrl)
.subscribe( result => {
// your logic
} );
});
});
}
service
uploadImage(contentType, imageData) {
console.log("imageData", imageData)
const headers = new HttpHeaders();
if (contentType === 'jpeg') {
headers.set('Content-Type', 'image/jpeg;');
} else if (contentType === 'png') {
headers.set('Content-Type', 'image/jpeg;');
}
const data = imageData.replace(/^data:image\/\w+;base64,/, '');
return this.http.put(
environment.slsLocal + '/update-image-url',
data,
{ headers: headers }
);
}
Background
Javascript library for Microsoft Office add-ins allows you to get raw content of the DOCX file through getFileAsync() api, which returns a slice of up to 4MB in one go. You keep calling the function using a sliding window approach till you have reed entire content. I need to upload these slices to the server and the join them back to recreate the original DOCX file.
My attempt
I'm using axios on the client-side and busboy-based express-chunked-file-upload middleware on my node server. As I call getFileAsync recursively, I get a raw array of bytes that I then convert to a Blob and append to FormData before posting it to the node server. The entire thing works and I get the slice on the server. However, the chunk that gets written to the disk on the server is much larger than the blob I uploaded, normally of the order of 3 times, so it is obviously not getting what I sent.
My suspicion is that this may have to do with stream encoding, but the node middleware does not expose any options to set encoding.
Here is the current state of code:
Client-side
public sendActiveDocument(uploadAs: string, sliceSize: number): Promise<boolean> {
return new Promise<boolean>((resolve) => {
Office.context.document.getFileAsync(Office.FileType.Compressed,
{ sliceSize: sliceSize },
async (result) => {
if (result.status == Office.AsyncResultStatus.Succeeded) {
// Get the File object from the result.
const myFile = result.value;
const state = {
file: myFile,
filename: uploadAs,
counter: 0,
sliceCount: myFile.sliceCount,
chunkSize: sliceSize
} as getFileState;
console.log("Getting file of " + myFile.size + " bytes");
const hash = makeId(12)
this.getSlice(state, hash).then(resolve(true))
} else {
resolve(false)
}
})
})
}
private async getSlice(state: getFileState, fileHash: string): Promise<boolean> {
const result = await this.getSliceAsyncPromise(state.file, state.counter)
if (result.status == Office.AsyncResultStatus.Succeeded) {
const data = result.value.data;
if (data) {
const formData = new FormData();
formData.append("file", new Blob([data]), state.filename);
const boundary = makeId(12);
const start = state.counter * state.chunkSize
const end = (state.counter + 1) * state.chunkSize
const total = state.file.size
return await Axios.post('/upload', formData, {
headers: {
"Content-Type": `multipart/form-data; boundary=${boundary}`,
"file-chunk-id": fileHash,
"file-chunk-size": state.chunkSize,
"Content-Range": 'bytes ' + start + '-' + end + '/' + total,
},
}).then(async res => {
if (res.status === 200) {
state.counter++;
if (state.counter < state.sliceCount) {
return await this.getSlice(state, fileHash);
}
else {
this.closeFile(state);
return true
}
}
else {
return false
}
}).catch(err => {
console.log(err)
this.closeFile(state)
return false
})
} else {
return false
}
}
else {
console.log(result.status);
return false
}
}
private getSliceAsyncPromise(file: Office.File, sliceNumber: number): Promise<Office.AsyncResult<Office.Slice>> {
return new Promise(function (resolve) {
file.getSliceAsync(sliceNumber, result => resolve(result))
})
}
Server-side
This code is totally from the npm package (link above), so I'm not supposed to change anything in here, but still for reference:
makeMiddleware = () => {
return (req, res, next) => {
const busboy = new Busboy({ headers: req.headers });
busboy.on('file', (fieldName, file, filename, _0, _1) => {
if (this.fileField !== fieldName) { // Current field is not handled.
return next();
}
const chunkSize = req.headers[this.chunkSizeHeader] || 500000; // Default: 500Kb.
const chunkId = req.headers[this.chunkIdHeader] || 'unique-file-id'; // If not specified, will reuse same chunk id.
// NOTE: Using the same chunk id for multiple file uploads in parallel will corrupt the result.
const contentRangeHeader = req.headers['content-range'];
let contentRange;
const errorMessage = util.format(
'Invalid Content-Range header: %s', contentRangeHeader
);
try {
contentRange = parse(contentRangeHeader);
} catch (err) {
return next(new Error(errorMessage));
}
if (!contentRange) {
return next(new Error(errorMessage));
}
const part = contentRange.start / chunkSize;
const partFilename = util.format('%i.part', part);
const tmpDir = util.format('/tmp/%s', chunkId);
this._makeSureDirExists(tmpDir);
const partPath = path.join(tmpDir, partFilename);
const writableStream = fs.createWriteStream(partPath);
file.pipe(writableStream);
file.on('end', () => {
req.filePart = part;
if (this._isLastPart(contentRange)) {
req.isLastPart = true;
this._buildOriginalFile(chunkId, chunkSize, contentRange, filename).then(() => {
next();
}).catch(_ => {
const errorMessage = 'Failed merging parts.';
next(new Error(errorMessage));
});
} else {
req.isLastPart = false;
next();
}
});
});
req.pipe(busboy);
};
}
Update
So it looks like I have found the problem at least. busboy appears to be writing my array of bytes as text in the output file. I get 80,75,3,4,20,0,6,0,8,0,0,0,33,0,44,25 (as text) when I upload the array of bytes [80,75,3,4,20,0,6,0,8,0,0,0,33,0,44,25]. Now need to figure out how to force it to write it as a binary stream.
Figured out. Just in case it helps anyone, there was no problem with busboy or office.js or axios. I just had to convert the incoming chunk of data to Uint8Array before creating a blob from it. So instead of:
formData.append("file", new Blob([data]), state.filename);
like this:
const blob = new Blob([ new Uint8Array(data) ])
formData.append("file", blob, state.filename);
And it worked like a charm.
Being new to java script and node.js, I ran into an error.
I'm attempting to scrape newspaper articles from a french newspaper using password and serchterms.
This is the code I'm using:
const fs = require("fs");
const request = require("request");
const cheerio = require("cheerio");
const jsonTab = []; // We create our table
function writeFile() {
// Will write the json file
fs.writeFile("output.json", JSON.stringify(jsonTab, null, 4), (err) => {
console.log("File successfully written!");
});
}
// The URL of the advanced search feature with our keywords
const url = 'http://www.lemonde.fr/recherche/?keywords="Rap+"+"hip-hop"+"hip%20hop"+"rappeur"+"rappeurs"+"raps"+"rappers"&page_num=1&operator=or&exclude_keywords=&qt=recherche_texte_title&author=&period=custom_date&start_day=01&start_month=01&start_year=1970&end_day=20&end_month=09&end_year=2017&sort=asc';
/* The first request call, our goal here is to get the number of results and then
to calculate the number of pages */
request(url, (error, response, html) => {
const $ = cheerio.load(html);
// All the variables we will use later
let number;
let description;
let date;
let title;
let link;
if (!error) {
$(".bg_gris_clair").filter(() => {
/* We want to select all the HTML
elements with the class ".bg_gris_clair" (and we already know there is
only one) */
const data = $(this);
const str = data.children("strong").text().trim();
number = parseInt(str.substring(0, str.indexOf("e")).replace(/\s/g, ""), 10);
});
}
let count = 1;
for (let i = 1; i <= number / 10; i++) {
const urlPerPage = 'http://www.lemonde.fr/recherche/?keywords="Rap+"+"hip-hop"+"hip%20hop"+"rappeur"+"rappeurs"+"raps"+"rappers"&page_num=' + i + "&operator=or&exclude_keywords=&qt=recherche_texte_title&author=&period=custom_date&start_day=01&start_month=01&start_year=1970&end_day=20&end_month=09&end_year=2017&sort=asc";
request(urlPerPage, (err, response2, html2) => {
if (!err) {
const $ = cheerio.load(html2);
$(".grid_11.omega.resultat").filter(() => {
const json = {
date: "",
title: "",
description: "",
url: ""
};
const data = $(this);
title = data.children("h3").children("a").text().trim();
link = "http://lemonde.fr" + data.children("h3").children("a").attr("href").trim();
description = data.children("p").text().trim();
const dateStr = data.children("span").text();
date = dateStr.replace(/.+?(?=\d)/, "");
json.title = title;
json.url = link;
json.description = description;
json.date = date;
jsonTab.push(json);
});
} else if (err) {
console.log(err);
}
count += 1;
// Write the file once we iterated through all the pages.
if (count === parseInt(number / 10, 10)) {
writeFile();
}
});
}
});
const fs = require("fs");
const request = require("request");
const cheerio = require("cheerio");
// Prepare all the variables needed later
let count = 0;
let timeout = 0;
const id = "myusername";
const mdp = "mypassword";
let obj;
// The URLs we will scrape from
const connexionUrl = "https://secure.lemonde.fr/sfuser/connexion";
// Will write an "output.json" file
function writeFile() {
fs.writeFile("output.json", JSON.stringify(obj, null, 4), (err) => {
console.log(
"File successfully written! - Check your project directory for the output.json file"
);
});
}
// creating a clean jar to store the cookies
const j = request.jar();
// First Get Request Call
request(
{
url: connexionUrl,
jar: j
},
(err, httpResponse, html) => {
const $ = cheerio.load(html);
// We use Cheerio to load the HTML and be able to find the connection__token
const token = $("#connection__token")[0].attribs.value; // here is the connection__token
// Construction of the form required in the POST request to login
const form = {
"connection[mail]": id,
"connection[password]": mdp,
"connection[stay_connected]": 1,
"connection[save]": "",
"connection[_token]": token
};
// POST REQUEST to Log IN. Same url with "request headers" and the complete form.
request.post(
{
url: connexionUrl,
jar: j,
headers: {
Accept:
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4",
"Cache-Control": "no-cache",
"Content-Type": "application/x-www-form-urlencoded",
Origin: "http://secure.lemonde.fr/",
Host: "secure.lemonde.fr",
"Upgrade-Insecure-Requests": 1,
"User-Agents":
"Mozilla/5.0 (Macintosh; Intel Mac OS X x.y; rv:42.0) Gecko/20100101 Firefox/42.0",
Connection: "keep-alive",
Pragma: "no-cache",
Referer: "https://secure.lemonde.fr/sfuser/connexion"
},
form: form
},
(error, response, body) => {
// WE ARE CONNECTED :D
/* Second GET request call : this time, we use the response of the POST
request to request the right URL */
request(
{
url: response.headers.location,
jar: j
},
(err, httpResponse, html2) => {
const json = fs.readFileSync("./firstStep.json"); // Load the JSON created in step one
obj = JSON.parse(json); // We create our JSON in a usable javascript object
// forEach loop to iterate through all the object and request each link
obj.forEach((e) => {
let articleUrl = e.url;
/* We use a setTimeout to be sure that all the requests are performed
one by one and not all at the same time */
setTimeout(() => {
request(
{
url: articleUrl,
jar: j
},
(error1, httpResponse, html3) => {
if (!error1) {
const $ = cheerio.load(html3); // load the HTML of the article page
$(".contenu_article.js_article_body").filter(() => {
const data = $(this);
// get the content, remove all the new lines (better for Excel)
let text = data
.text()
.trim()
.replace(/\n/g, "\t");
e.text = text; // push the content in the table
});
$(".txt3.description-article").filter(() => {
const data = $(this);
const description = data
.text()
.trim()
.replace(/\n/g, "\t");
e.description = description;
});
}
}
);
count += 1;
// Write a new JSON file once we get the content of all the articles
if (count === obj.length) {
writeFile();
}
}, timeout);
timeout += 50; // increase the timeout length each time
});
}
);
}
);
}
);
Running step3 it throws an error:
Error: ENOENT: no such file or directory, open './firstStep.json'
The tutorial can be found here: https://www.freecodecamp.org/news/how-i-scraped-7000-articles-from-a-newspaper-website-using-node-1309133a5070/
I am more used to using R, this is a new playing field for me.
I suspect the problem is straight forward, and related to the directory.
Thank you
I have these methods for choosing and uploading the image, but I am not sure about where the error is, if it is in the code or in the API
const [imagens, setImagens] = useState([])
function choseImage(event) {
imagens.push(URL.createObjectURL(event.target.files[0]))
}
const upload = () => {
const fd = new FormData();
// imagens is the array of images
fd.append('imagem', imagens)
fetch('http://localhost:5000/api/Upload', fd)
.then(res => {
console.log(res)
});
}
here is the UploadController, the messege is:
System.NullReferenceException: 'Object reference not set to an instance of an object.' file was null.
[HttpPost]
public async Task Post(IFormFile file)
{
var nomeArquivo = (file.FileName);
var uploads = Path.Combine(Directory.GetCurrentDirectory(), "wwwRoot\\Upload", nomeArquivo);
if (!Directory.Exists(uploads)) Directory.CreateDirectory(uploads);
if (file.Length > 0)
{
using (var fileStream = new FileStream(Path.Combine(uploads, file.FileName), FileMode.Create))
{
await file.CopyToAsync(fileStream);
}
}
}
this is my first freelance project, that is the last thing missing, Thanks in advance
Your code is nearly complete with a couple of changes commented in the code below. In your React component you can do something like the following.
export const ImageFileUpload = () => {
const [images, setImages] = useState([]);
const onFileChange = (files) => {
setImages(f => [...f, ...files]);
};
const handleClick = (e) => {
e.preventDefault();
const formData = new FormData();
for (let i = 0; i < images.length; i++) {
// 'images' name of the formData values must match the action method param on your controller
formData.append(`images`, images[i]);
}
// Make sure you api is running on the same endpoint or you need to set up CORS
fetch('https://localhost:5001/fileupload', {
body: formData,
method: "POST" // Make sure you add the method POST to your request when API is only accepting POST
}).then(res => console.log(res));
};
return (
<form>
<input type="file" multiple={true} onChange={e => onFileChange(e.target.files)} />
<button onClick={handleClick}>Upload</button>
</form>
)
};
Your controller should look like this.
[ApiController]
[Route("[controller]")]
public class FileUploadController : ControllerBase
{
[HttpPost]
// 1. The name of the parameter must match the name of the `formData`
// 2. This method should take `List<IFormFile>`
public async Task Upload(List<IFormFile> images)
{
foreach (var image in images)
{
var nomeArquivo = (image.FileName);
var uploads = Path.Combine(Directory.GetCurrentDirectory(), "wwwRoot\\Upload\\", nomeArquivo);
if (!Directory.Exists(uploads)) Directory.CreateDirectory(uploads);
using (var fileStream = new FileStream(Path.Combine(uploads, image.FileName), FileMode.Create))
{
await image.CopyToAsync(fileStream);
}
}
}
}
I am trying to upload an array of images using my custom API (Node JavaScript), here I just want to save the file name to my database, and uploading of the file works just fine (the middleware which I created for uploading works just fine because I have tried using the same middleware in the postman).
When it comes to the actual JavaScript file it is not working and shows the following error:
Cast to string failed for value "{ '0': {}, '1': {} }" at path "images", images: Cast to Array failed for value "[ { '0': {}, '1': {} } ]" at path "images"
Here is my code
const createProperty = ['city', 'propertyType', 'propertyName', 'images'];
const newProperty = new Array();
for (var i = 0; i < myids.length; i++) {
newProperty[i] = $(myids[i]).val();
}
newProperty[myids.length] = [document.getElementById('PropertyPhotos').files];
const data = {};
createProperty.forEach((id, index) => {
data[createProperty[index]] = newProperty[index]
})
await createData(data);
// here is the CreateDate function
export const createData = async (data) => {
try {
const res = await axios({
method: 'POST',
url: '/api/v1/properties',
data
});
if (res.data.status === 'success') {
showAlert('success', 'Property Created Successfully');
}
console.log(res.data);
} catch (err) {
showAlert('error', err.response.data.message);
console.log(err.response.data.message);
}
}
It looks like you are using Mongo. Look here and here.
And to send files you need to use form-data, but I don't see it in your code.
const formdata = new FormData();
formdata.append("file", fileInput.files[0], "/filepath.pdf");