I'd like to generate a CID (Content identifier) for a file in javascript without having access to an IPFS node or the internet. I've tried using js-multihashing-async to first hash the file and js-cid to generate a CID from the hash but I get a different CID than if I just add the file to ipfs-desktop. It looks like the problem is an IPFS node chunks data and the CID is for the DAG that links the files' chunks. I've tried this library but it doesn't produce the same CID as ipfs-desktop does for the same file. This question is essentially the same as mine but none of the answers give a CID that matches the ipfs-desktop-generated CID.
ipfs-only-hash is the right module to use to create an IPFS CID from a file or a Buffer, without needing to start an IPFS daemon. For the same input file and the same options, it should produce the same CID.
This example is from the ipfs-only-hash tests, where it verifies that it hashes the same buffer to the same CID as a js-ipfs node does.
test('should produce the same hash as IPFS', async t => {
const data = Buffer.from('TEST' + Date.now())
const ipfs = new Ipfs({ repo: path.join(os.tmpdir(), `${Date.now()}`) })
await new Promise((resolve, reject) => {
ipfs.on('ready', resolve).on('error', reject)
})
const files = await ipfs.add(data)
const hash = await Hash.of(data)
t.is(files[0].hash, hash)
})
https://github.com/alanshaw/ipfs-only-hash/blob/dbb72ccfff45ffca5fbea6a7b1704222f6aa4354/test.js#L21-L33
I'm one of the maintainers of IPFS Desktop, and under the hood, that app calls ipfs.add on http api for the local IPFS daemon here
When adding or hashing a file manually via the api, there are options to alter how files are chunked into blocks, how those blocks are linked together, and how the blocks are hashed. If any option values differ then the resulting hash and the CID that contains it will be different, even if the input file is the same.
You can experiment with those options and see a visualisation of the resulting DAG (Directed Acyclic Graph) structure here: https://dag.ipfs.io/
For a deep dive on how IPFS chunks and hashes files you can see the author of the ipfs-only-hash and maintainer of js-ipfs explain it here https://www.youtube.com/watch?v=Z5zNPwMDYGg
For the sake of posterity, here is how to match an image's CID downloaded via fetch to the CID generated from ipfs-desktop for the same image (added as a file from the local drive). You have to remove the prefix data:*/*;base64, that is prepended to the image's base64string and decode the string into a buffer array. Then you get the matching CID.
async testHashes() {
const url = "https://raw.githubusercontent.com/IanPhilips/jst-cids-test/master/src/23196210.jpg";
fetch(url)
.then(response => response.blob())
.then(blob => new Promise((resolve, reject) => {
const reader = new FileReader();
reader.onloadend = () => resolve(reader.result);
reader.readAsDataURL(blob)
})).then(async dataUrl =>{
const strData = dataUrl as string;
// remove "data:*/*;base64," from dataUrl
const endOfPrefix = strData.indexOf(",");
const cleanStrData = strData.slice(endOfPrefix+1);
const data = Buffer.from(cleanStrData, "base64");
const hash = await Hash.of(data);
console.log("fetch data CID: " + hash); // QmYHzA8euDgUpNy3fh7JRwpPwt6jCgF35YTutYkyGGyr8f
});
console.log("ipfs-desktop CID: QmYHzA8euDgUpNy3fh7JRwpPwt6jCgF35YTutYkyGGyr8f");
}
Related
I am using the #aws-sdk/client-s3 to read a json file from S3, take the contents and dump it into dynamodb. This all currently works fine using:
const data = await (await new S3Client(region).send(new GetObjectCommand(bucketParams)));
And then deserialising the response body etc.
However, I'm looking to migrate to use jsonlines format, effectiely csv, in the sense it needs to be streamed in line by line or in chunks of lines and processed. I can't seem to find a way of doing this that doesnt load the entire file into memory (using response.text() etc).
Ideally, I would like to pipe the response into a createReadStream, and go from there.
I found this example with createReadStream() form module fs in node.js:
import fs from 'fs';
function read() {
let data = '';
const readStream = fs.createReadStream('business_data.csv', 'utf-8');
readStream.on('error', (error) => console.log(error.message));
readStream.on('data', (chunk) => data += chunk);
readStream.on('end', () => console.log('Reading complete'));
};
read();
You can modify it for your use. Hope this helps.
Connection to your S3 you can do by:
var s3 = new AWS.S3({apiVersion: '2006-03-01'});
var params = {Bucket: 'myBucket', Key: 'myImageFile.jpg'};
var file = require('fs').createWriteStream('/path/to/file.jpg');
s3.getObject(params).createReadStream().pipe(file);
see here
Currently trying to download image from GitHub locally. Everything seems to work, the fetch goes through with a 200 OK response, however, I don't understand how to store image itself:
const rawGitLink = "https://raw.githubusercontent.com/cardano-foundation/CIPs/master/CIP-0001/CIP_Flow.png"
const folder = "/Folder"
const imageName = "/Test"
const imageResponse = await axios.get(rawGitLink)
fs.writeFileSync(___dirname + folder + imageName, imageResponse, (err) => {
//Error handling
}
)
Four problems had to be fixed:
Image name must include png format for this case
The response must be in the correct format as a buffer for an image
You must write the response data and not the object itself
__dirname only needs two underscores
const rawGitLink = "https://raw.githubusercontent.com/cardano-foundation/CIPs/master/CIP-0001/CIP_Flow.png"
const folder = "/Folder"
const imageName = "/Test.png"
const imageResponse = await axios.get(rawGitLink, { responseType: 'arraybuffer' });
fs.writeFileSync(__dirname + folder + imageName, imageResponse.data)
Axios returns a special object: https://github.com/axios/axios#response-schema
let {data} = await axios.get(...)
await fs.writeFile(filename, data) // you can use fs.promises instead of sync
As #Leau said you should include the extension on the filename
Another sugestion is to use the path module to create the filename:
filename = path.join(__dirname, "/Folder", "Test.png")
I am working on reactjs/typescript applications. I am trying to download some files from azure storage v2. Below is the sample path I am supposed to download files. In this path, enrichment is the container name, and the rest all are folders. I am trying to download the last modified file from reportdocument folder.
enrichment/report/SAR-1234-56/reportdocument/file1.docs
I tried something below.
#action
public async reportDownload(sarNumber: string) {
let storage = globals.getGlobals('StorageAccount03');
console.log(storage);
let containerName = globals.getGlobals('StorageAccount03ContainerName');
let marker = undefined;
let allUploadPromise: Array<Promise<unknown>> = [];
const config = {
path: `/Storage/getsastoken/?storageName=${storage}&containerName=${containerName}`,
method: "GET",
success: (url: any) => {
const containerURL: ContainerURL = new ContainerURL(
url,
StorageURL.newPipeline(new AnonymousCredential()));
const listBlobsResponse = containerURL.listBlobFlatSegment(
Aborter.none,
marker,
);
}
};
await handleRequest(config);
}
From here I am struggling to download the latest modified file from the above path.
can someone help me to fix this? Any help would be greatly appreciated. Thank you
It's better to use #azure/storage-blob library and then the code would be something like below instead of directly trying to call blob REST API like you were trying in your code which seems unnecessary reinventing the wheel. The library already does it for you. Refer this for details.
const { BlobServiceClient } = require("#azure/storage-blob");
const account = "<account name>";
const sas = "<service Shared Access Signature Token>";
const containerName = "<container name>";
const blobName = "<blob name>";
const blobServiceClient = new BlobServiceClient(`https://${account}.blob.core.windows.net${sas}`);
async function download() {
const containerClient = blobServiceClient.getContainerClient(containerName);
const blobClient = containerClient.getBlobClient(blobName);
// Get blob content from position 0 to the end
// In browsers, get downloaded data by accessing downloadBlockBlobResponse.blobBody
const downloadBlockBlobResponse = await blobClient.download();
const downloaded = await blobToString(await downloadBlockBlobResponse.blobBody);
console.log("Downloaded blob content", downloaded);
// [Browsers only] A helper method used to convert a browser Blob into string.
async function blobToString(blob) {
const fileReader = new FileReader();
return new Promise((resolve, reject) => {
fileReader.onloadend = (ev) => {
resolve(ev.target.result);
};
fileReader.onerror = reject;
fileReader.readAsText(blob);
});
}
}
The SAS token expiry bothers me.You cannot have a static SAS token that expires sooner unless we can set long expiry (user-delegation SAS token is short lived). Do we really have the capability to create the SAS token dynamically in javascript runtime? I think it's only possible in NodeJS runtime.
actually there are many answers for this question. But my problem is,
i want to generate pdf dynamically with 5 external(URL) images. Im using PDFmake node module.
it supports only two ways local and base64 format. But i don't want to store images locally.
so my requirement is one function which takes url as parameter and returns base64.
so that i can store in global variable and create pdfs
thanks in advance
function urlToBase(URL){
return base64;
}
var img = urlToBase('https://unsplash.com/photos/MVx3Y17umaE');
var dd = {
content: [
{
text: 'fjfajhal'
},
{
image: img,
}
]
};
var writeStream = fs.createWriteStream('myPdf.pdf');
var pdfDoc = printer.createPdfKitDocument(dd);
pdfDoc.pipe(writeStream);
pdfDoc.end();
im using PDFmake module from npm
The contents of the remote image can first be fetched with an HTTP request, for example using the ubiquitous request npm module. The image string contents can then be transformed to a buffer and finally converted to a base64 string. To complete the transformation, add the proper data-url prefix, for example, data:image/png,base64, to the beginning of the base64 string.
Here is a rough example for a PNG image:
const request = require('request-promise-native');
let jpgDataUrlPrefix = 'data:image/png;base64,';
let imageUrl = 'https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png';
request({
url: imageUrl,
method: 'GET',
encoding: null // This is actually important, or the image string will be encoded to the default encoding
})
.then(result => {
let imageBuffer = Buffer.from(result);
let imageBase64 = imageBuffer.toString('base64');
let imageDataUrl = jpgDataUrlPrefix+imageBase64;
console.log(imageDataUrl);
});
I have an API method that when called and passed an array of file keys, downloads them from S3. I'd like to stream them, rather than download to disk, followed by zipping the files and returning that to the client.
This is what my current code looks like:
reports.get('/xxx/:filenames ', async (req, res) => {
var AWS = require('aws-sdk');
var s3 = new AWS.S3();
var str_array = filenames.split(',');
for (var i = 0; i < str_array.length; i++) {
var filename = str_array[i].trim();
localFileName = './' + filename;
var params = {
Bucket: config.reportBucket,
Key: filename
}
s3.getObject(params, (err, data) => {
if (err) console.error(err)
var file = require('fs').createWriteStream(localFileName);
s3.getObject(params).createReadStream().pipe(file);
console.log(file);
})
}
});
How would I stream the files rather than downloading them to disk and how would I zip them to return that to the client?
Main problem is to zip multiple files.
More specifically, download them from AWS S3 in bulk.
I've searched through AWS SDK and didn't find bulk s3 operations.
Which brings us to one possible solution:
Load files one by one and store them to folder
Zip folder (with some package like this)
Send zipped folder
This is raw and untested example, but it might give you the idea:
// Always import packages at the beginning of the file.
const AWS = require('aws-sdk');
const fs = require('fs');
const zipFolder = require('zip-folder');
const s3 = new AWS.S3();
reports.get('/xxx/:filenames ', async (req, res) => {
const filesArray = filenames.split(',');
for (const fileName of filesArray) {
const localFileName = './' + filename.trim();
const params = {
Bucket: config.reportBucket,
Key: filename
}
// Probably you'll need here some Promise logic, to handle stream operation end.
const fileStream = fs.createWriteStream(localFileName);
s3.getObject(params).createReadStream().pipe(fileStream);
}
// After that all required files would be in some target folder.
// Now you need to compress the folder and send it back to user.
// We cover callback function in promise, to make code looks "sync" way.
await new Promise(resolve => zipFolder('/path/to/the/folder', '/path/to/archive.zip', (err) => {resolve()});
// And now you can send zipped folder to user (also using streams).
fs.createReadStream('/path/to/archive.zip').pipe(res);
});
Info about streams link and link
Attention: You'll probably could have some problems with async behaviour, according to streams nature, so, please, first of all, check if all files are stored in folder before zipping.
Just a mention, I've not tested this code. So if any questions appear, let's debug together