Develop Reference

Attempting to download a .xlsx file and upload to Azure Blob Storage using Playwright with Javascript produces malformed .xlsx file with a bad header

I am trying to download an excel file and then upload it to Azure Blob Storage for use in Azure Data Factory. I have a playwright javascript that worked when the file was a .csv but when I try the same code with an excel file, it will not open in Excel. It says,
"We found a problem with some content in 'order_copy.xlsx'. Do you want us to try to recover as much as we can?:
After clicking yes, it says,
"Excel cannot open the file 'order_copy.xlsx' because the file format or file extension is not valid. Verify that the file has not been corrupted and that the file extension matches the format of the file."
Any ideas on how to use the createReadStream more effectively to do this and preserve the .xlsx format?
I don't think the saveAs method will work since this code is being executed in an Azure Function with no access to a local known path.
My first thought was the content type was not right, so I set that, but it still did not work. I tried a UTF-8 encoder but that also did not work.
//const data = await streamToText(download_csv.createReadStream())
const download_reader = await download_csv.createReadStream();
let data = '';
for await (const chunk of download_reader) {
data += chunk; //---I suspect I need to do something different here
}
// const data_utf8 = utf8.encode(data) //unescape( encodeURIComponent(data) );
const AZURE_STORAGE_CONNECTION_STRING = "..." //---Removed string here
// Create the BlobServiceClient object which will be used to create a container client
const blob_service_client = BlobServiceClient.fromConnectionString(AZURE_STORAGE_CONNECTION_STRING);
// Get a reference to a container
const container_client = blob_service_client.getContainerClient('order');
const blob_name = 'order_copy.xlsx';
// Get a block blob client
const block_blob_client = container_client.getBlockBlobClient(blob_name);
const contentType = 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
const blobOptions = { blobHTTPHeaders: { blobContentType: contentType } };
//const uploadBlobResponse = await block_blob_client.upload(data_utf8, data_utf8.length, blobOptions);
const uploadBlobResponse = await block_blob_client.upload(data, data.length, blobOptions);
console.log("Blob was uploaded successfully. requestId: ", uploadBlobResponse.requestId);
Any guidance would be appreciated. Thank you in advance for your help!
-Chad

Thanks #Gaurav for the suggestion on not setting the data to a string. The following code worked after I changed to using a array of the chunks and concatenated it using the Buffer similar to your suggested code.
let chunks = []
for await (const chunk of download_reader) {
chunks.push(chunk)
}
const fileBuffer = Buffer.concat(chunks)
...
const uploadBlobResponse = await block_blob_client.upload(fileBuffer, fileBuffer.length, blobOptions);
Thanks everyone!

Is it possible to get an image from a message and add it to a folder in Discord.js?

Like is it possible to do something like !image add [image file] and then add the attachment to a folder? I think i can do that with fs, but i'm not sure how

You can use the fs function fs.writeFile() or fs.writeFileSync(). This function accepts the absolute path to a file to write to, and the data to write. In your case, it should be a buffer or stream.
// const fs = require('fs');
fs.writeFileSync('./some_dir/some_file_name.extension', data);
To get the data in question, you should access Message#attachments(), a collection of all attachments on the message. Assuming you only want the first, you can use Collection#first() to narrow down the results.
const attachment = message.attachments.first();
if (!attachment) {
// maybe place in some error handling
}
Unfortunately, the MessageAttachment class doesn't actually hold a buffer/stream representing the attachment, only the URL leading to it. This means you'll need a third-party library such as axios or node-fetch.
// const fetch = require('node-fetch');
fetch(attachment.url)
.then(res => res.buffer())
.then(buffer => {
fs.writeFileSync(`./images/${attachment.name}`, buffer);
});
Make sure to validate that URL to make sure it's an image!
if(!/\.(png|jpe?g|svg)$/.test(attachment.url)) {
// this attachment isn't an image!
// we don't want to be downloading .exe files now, do we?
}
Finally, you should also be weary that if two files are named the same, such as image.png, trying to write the second one will overwrite the first. One way to overcome that issue is to add numerical suffixes to duplicates, such as image.png, image-1.png, image-2.png, etc. That could work out a little like this:
fetch(attachment.url)
.then(res => res.buffer())
.then(buffer => {
let path = `./images/${attachment.name}`;
// increment the suffix every iteration until a file
// by the same name cannot be found
for (let count = 1; fs.existsSync(path); count++) {
path = `./images/${attachment.name}-${count}`;
}
fs.writeFileSync(path, buffer);
});

Node js path file

My csv file is in drive 'E' folder 'awais' and i am new to coding i cant access the file anybody can help me with relative path absolute path concept and how can i access this file
i am getting this error : "Unhandled rejection Error: File does not exist. Check to make sure the file path to your csv is correct."
const fs = require('fs')
const path = require('path')
const csv = require('csvtojson');
const csvFile = ('/E/Awais/customer-data.csv');
csv()
.fromFile(csvFile).then ( (jsonObj) =>{
console.log(jsonObj)
fs.writeFile(path.join(__dirname,'customer-data.json'),JSON.stringify(jsonObj,null,1),()=>{
console.log(jsonObj)
});
})
```

You might wanna try:
const csvFile = ('E:/Awais/customer-data.csv');
That would be an absolute path. If you want to use a relative path (meaning that the csv is located in a folder relative to where your code file is stored) you would need to let us know in which folder your code resides. Alternatively (since you already imported path) you could also do:
const csvFile = path.normalize('E:\\Awais\\customer-data.csv');

send file from server to client on bokeh

I have made a user interface to fetch data from a MySQL table and visualize it. It is running on a bokeh server. My users connect remotely to the server using their browser (firefox). This works perfectly fine: I simply import the table into a pandas dataframe.
My users also need to download the table as excel. This means I cannot use the export_csv example which is pure javascript.
I have no experience with JavaScript. All I want is to transfer a file from the directory where my main.py is to the client side.
The technique I have tried so far is to join a normal on_click callback to a button, export the information I need to 'output.xls', then change a parameter from a dummy glyph which in turn runs a Javascript code. I got the idea from Bokeh widgets call CustomJS and Python callback for single event? . Note I haven't set the alpha to 0, so that I can see if the circle is really growing upon clicking the download button.
At the bottom of my message you can find my code. You can see I have tried with both XMLHttpRequest and with Fetch directly. In the former case, nothing happens. In the latter case I obtain a file named "mydata.xlsx" as expected, however it contains only this raw text: <html><title>404: Not Found</title><body>404: Not Found</body></html>.
Code:
p = figure(title='mydata')
#download button
download_b = Button(label="Download", button_type="success")
download_b.on_click(download)
#dummy idea from https://stackoverflow.com/questions/44212250/bokeh-widgets-call-customjs-and-python-callback-for-single-event
dummy = p.circle([1], [1],name='dummy')
JScode_xhr = """
var filename = p.title.text;
filename = filename.concat('.xlsx');
alert(filename);
var xhr = new XMLHttpRequest();
xhr.open('GET', '/output.xlsx', true);
xhr.responseType = 'blob';
xhr.onload = function(e) {
if (this.status == 200) {
var blob = this.response;
alert('seems to work...');
if (navigator.msSaveBlob) {
navigator.msSaveBlob(blob, filename);
}
else {
var link = document.createElement("a");
link = document.createElement('a');
link.href = URL.createObjectURL(blob);
window.open(link.href, '_blank');
link.download = filename;
link.target = "_blank";
link.style.visibility = 'hidden';
link.dispatchEvent(new MouseEvent('click'));
URL.revokeObjectURL(url);
}
}
else {
alert('Ain't working!');
}
};
"""
JScode_fetch = """
var filename = p.title.text;
filename = filename.concat('.xlsx');
alert(filename);
fetch('/output.xlsx').then(response => response.blob())
.then(blob => {
alert(filename);
//addresses IE
if (navigator.msSaveBlob) {
navigator.msSaveBlob(blob, filename);
}
else {
var link = document.createElement("a");
link = document.createElement('a')
link.href = URL.createObjectURL(blob);
window.open(link.href, '_blank');
link.download = filename
link.target = "_blank";
link.style.visibility = 'hidden';
link.dispatchEvent(new MouseEvent('click'))
URL.revokeObjectURL(url);
}
return response.text();
});
"""
dummy.glyph.js_on_change('size', CustomJS(args=dict(p=p),
code=JScode_fetch))
plot_tab = Panel(child=row(download_b,p),
title="Plot",
closable=True,
name=str(self.test))
def download():
writer = pd.ExcelWriter('output.xlsx')
data.to_excel(writer,'data')
infos.to_excel(writer,'info')
dummy = p.select(name='dummy')[0]
dummy.glyph.size = dummy.glyph.size +1

Trying out Eugene Pakhomov's answer, I found what was the issue.
The javascript code I named JScode_fetch is almost correct, however I get a 404 because it is not pointing correctly to the right path.
I made my application in the directory format: I changed my .py file to main.py, placed it into a folder called app, and changed this one line of code in JScode_fetch:
fetch('/app/static/output.xlsx', {cache: "no-store"}).then(response => response.blob())
[...]
You can see the problem was that it was trying to access localhost:5006/output.xlsx, instead of localhost:5006/app/output.xlsx. As it is in directory format, the right link is now localhost:5006/app/static/output.xlsx to count for the static directory.
I also changed a few lines in the download function:
def download():
dirpath = os.path.join(os.path.dirname(__file__),'static')
writer = pd.ExcelWriter(os.path.join(dirpath,'output.xlsx'))
writer = pd.ExcelWriter('output.xlsx')
data.to_excel(writer,'data')
infos.to_excel(writer,'info')
dummy = p.select(name='dummy')[0]
dummy.glyph.size = dummy.glyph.size +1
Now it is working flawlessly!
edit: I have added , {cache: "no-store"} within the fetch() function. Otherwise the browser thinks the file is the same if you have to download a different dataframe excel while using the same output.xlsx filename. More info here.

bokeh serve creates just a few predefined handlers to serve some static files and a WebSocket connection - by default, it doesn't have anything to serve files from the root of the project.
Instead of using the one-file format, you can try using the directory format, save your files to static directory and download them from /static/.
One downside of this approach is that you still have to write that convoluted code to just make your backend create the file before a user downloads it.
The best solution would be to go one step further and embed Bokeh Server as a library into your main application. Since you don't have any non-Bokeh code, the simplest way would be to go with Tornado (an example).
bokeh.server.server.Server accepts extra_patterns argument - you can add a handler there to dynamically create Excel files and serve them from, say, /data/. After all that, the only thing that you need in your front-end is a single link to the Excel file.

Convert ICO icon file to PNG image file using plain Javascript

I want to..
.. convert an ICO file (e.g. http://www.google.com/favicon.ico ) to a PNG file after I downloaded it.
.. preserve transparency.
.. apply the solution in a node.js application.
I don't want to and already tried to ..
.. use native tools such as imagemagick (that's what I currently use in my application, but it's really bad for maintaining platform independency).
.. use tools that internally use native tools (e.g. gm.js).
.. rely on webservices such as http://www.google.com/s2/favicons?domain=www.google.de that don't allow configuring the resulting size or require payments or logins.
Therefore I'd love a Javascript-only solution. I used Jimp in another application, but it does not support ICO files.
Any help is appreciated. Thanks!

Use a FileReader() . Convert the Base64 to a data/png. Done.
const inputFile = __dirname + "/favicon.ico";
const outputFile = __dirname + "/favicon.png";
(function( inputFile, outputFile ) {
const fileApi = require("file-api");
const fs = require("fs");
const File = fileApi.File;
var fileReader = new fileApi.FileReader();
fileReader.readAsDataURL(new File(inputFile));
fileReader.addEventListener("load", function (ev) {
var rawdata = ev.target.result;
rawdata = rawdata.replace(/.*base64,/, "");
fs.writeFileSync(outputFile, rawdata, "base64");
});
})(inputFile, outputFile);

I am not familiar with Node environment but I wrote this ES6 module PNG2ICOjs using purely Javascript ArrayBuffer or Blob and can 100% run on client-side browsers (I assume Node file should act like a Blob).
import { PngIcoConverter } from "../src/png2icojs.js";
// ...
const inputs = [...files].map(file => ({
png: file
}));
// Result is a Blob
const resultBlob1 = await converter.convertToBlobAsync(inputs); // Default mime type is image/x-icon
const resultBlob2 = await converter.convertToBlobAsync(inputs, "image/your-own-mime");
// Result is an Uint8Array
const resultArr = await converter.convertAsync(inputs);