I have a utility module that creates an instance of a multer-gridfs storage engine for uploading files to my Mongo database. I use this module inside of any API route that requires the need to upload files.
I need to be able to update the metadata property value with a unique identifier. More than likely this will be the mongoose _id of the user uploading the file, but for now I am not concerned with that aspect of it. I really just want to know if I can change the metadata property dynamically.
Here is the storage engine gridFs_upload_engine.js:
const mongoose = require('mongoose');
const path = require('path');
const crypto = require('crypto');
const multer = require('multer');
const GridFsStorage = require('multer-gridfs-storage');
const Grid = require('gridfs-stream');
//Init Upload Engine
let gfs;
//Global instance of the DB connection
const database = mongoose.connection;
const mongoDb = process.env.MONGODB_URI || process.env.MLAB_URL;
database.once('open', () => {
//Init Stream
gfs = Grid(database.db, mongoose.mongo);
gfs.collection('uploads');
});
//Create Storage Engine
const storage = new GridFsStorage({
url: mongoDb,
file: (res, file) => {
return new Promise((resolve, reject) => {
crypto.randomBytes(16, (err, buf) => {
if (err) {
return reject(err);
}
const filename = buf.toString('hex') + path.extname(file.originalname);
const fileInfo = {
filename: filename,
bucketName: 'uploads',
metadata: 'NEED TO UPDATE THIS'
};
resolve(fileInfo);
});
});
}
});
const uploadEngine = multer({ storage });
module.exports = {
uploadEngine,
gfs
};
Above you can see the metadata property that I need to be able to dynamically change with some undetermined unique identifier. Is it possible to do that with an exported file?
Here is how I am utilizing it inside of an API route:
const express = require('express');
const router = express.Router();
//Controllers
const upload_controller = require('../../controllers/uploader');
//Utilities
const upload = require('../../utils/gridFs_upload_engine');
const { uploadEngine } = upload;
//Upload Single File
router.post(
'/single',
uploadEngine.single('file'),
upload_controller.upload_single_file
);
//Upload Multiple Files
//Max file uploads at once set to 30
router.post(
'/multiple',
uploadEngine.array('file', 30),
upload_controller.upload_multiple_files
);
module.exports = router;
I pass the uploadEngine into the API route here, so that the route controller can use it, and that works with no issue. I am just having quite a time trying to figure out how to update metatdata dynamically and I am leaning towards my current implementation not allowing for that.
I don't know much about node and have now idea what multer-gridfs is but I can answer How can I pass options into an imported module?
You can export an function that returns another function. And you would import it like
const configFunction = require('nameoffile')
// this returns a functions with the config you want
const doSomethingDependingOnTheConfig = configFunction({...someConfig})
And in the file you are importing you would have a function returning another function like
const configFunction = ({...someConfig}) => (your, func) => {
// do what you want deppending on the config
}
module.exports = configFunction
I know this doesn't answer your question the way you want, but answer you question title and I hope this give you a better understanding of how to do what you want to do.
If this doesn't help, just let me know.
You would need to pass a parameter to the module gridFs_upload_engine.js and do the magic there.
An example could be:
In gridFs_upload_engine.js file:
function uploadEngine (id, file) {
// update what you want
}
module.exports = {
...
uploadEngine: uploadEngine
}
In your router:
const upload = require('../../utils/gridFs_upload_engine')
...
router.post('/single/:id', function(req, res, next) {
...
upload.uploadEngine(req.params.id, file)
next()
}, upload_controller.upload_single_file)
In other words, when you are exposing gfs and uploadEngine inside your module, you could instead expose a function that would receive the arguments needed to perform the upload.
Related
I have uploaded the file in my backend filesystem using multer
My server is node and client is react.
I'm having trouble downloading and displaying the saved file on the client react
Whenever I do res.download(file) it just throws an error as connection refused on client side.
My code is as follows:
UserToUploadMapping.js
const mongoose = require("mongoose");
const UserToUploadMapping = new mongoose.Schema({
userId: {
type:String,
required:true
},
file: {
type: Object,
required: true,
},
date: {
type: Date,
default: Date.now,
},
});
module.exports = mongoose.model("UserToUploadMapping", UserToUploadMapping);
uploadVideo.js
const router = require("express").Router();
const multer = require('multer');
const UserToUploadMapping = require('../models/UserToUploadMapping')
let nameFile = ''
const storage = multer.diskStorage({
destination:'./Videos',
filename:(req,file,cb) => {
console.log(file)
nameFile = file.originalname + " "+ Date.now()
cb(null, nameFile)
}
})
const upload = multer({storage:storage})
router.post('/upload', upload.single('video'), async (req,res,next) => {
console.log("object")
const saveMapping = new UserToUploadMapping({
userId:'123',
file:req.file,
})
await saveMapping.save()
res.send("Video uploaded")
})
router.get('/download', async(req,res,next) => {
const x = await UserToUploadMapping.find()
// res.send(x)
res.download(x[0].path)
})
module.exports = router;
CLIENT
const fetchVideo = async () => {
const resp = await axios.get(
"http://localhost:5000/api/user/video/download"
);
console.log(resp)
};
return (
<>
<NavContainer />
<div className={classes.Post}>
<Input
type="file"
onChange={(e) => uploadVideos(e.target.files)}
accept="video/mp4"
/>
{/* <Button onClick={(e) => submitHandler(e)}>Upload</Button> */}
<video></video>
</div>
</>
);
Error
There is a few problems within the uploadVideo.js file :
to get the path from the data, you need to use x[0].file.path
(based on how you save the file in the database)
const saveMapping = new UserToUploadMapping({
userId:'123',
file:req.file,
})
to avoid problems about where the file uploadVideo.js is and where we run the application, you should use an absolute path when saving files in the system.
(small problem) your filename function will give filenames like this video.mp4 1622180824748. I think this is better "video-1622181268053.mp4" (we have the correct file extension)
You can refer to this code
const router = require("express").Router();
const multer = require('multer');
const UserToUploadMapping = require('../models/UserToUploadMapping')
const path = require('path');
const uploadFolder = path.join(__dirname, "Videos"); // use a variable to hold the value of upload folder
const storage = multer.diskStorage({
destination: uploadFolder, // use it when upload
filename: (req, file, cb) => {
// nameFile = file.originalname + " "+ Date.now() // --> give "video.mp4 1622180824748"
let [filename, extension] = file.originalname.split('.');
let nameFile = filename + "-" + Date.now() + "." + extension; // --> give "video-1622181268053.mp4"
cb(null, nameFile)
}
})
const upload = multer({ storage: storage })
router.post('/upload', upload.single('video'), async (req, res, next) => {
const saveMapping = new UserToUploadMapping({
userId: '123',
file: req.file,
})
await saveMapping.save()
res.send("Video uploaded")
})
router.get('/download', async (req, res, next) => {
const video = await UserToUploadMapping.find({});
res.download(video[0].file.path); // video[0].file.path is the absolute path to the file
})
module.exports = router;
Your code indicates you are handling large files (videos). I would strongly recommend looking at separation of concerns, handling this as part of your other business logic is not recommended based on my experience. This can e.g. complicate firewall rules and DDOS protection when that is needed in the future.
As a minimum, move upload and download into its own server, e.g. 'files.yourappnamehere.com' so that you can handle the specifics separately from your business logic api.
If you run in the public cloud, I would strongly recommend looking at reusing blob upload/download functionality, letting your clients upload directly to blob storage and also handling downloads directly from blob storage, e.g. in Azure, AWS or GCP.
This will save you a lot of the implementation details of handling (very) large files, and also give "free" extensibility options such as events on file upload completion.
You are running 2 apps Frontend and Backend with difference ports (3000, 5000) so browsers block cross domain requests. On Express you must enable CORS to allow request from FrontEnd Url (http://localhost:3000).
For the download route, try using window.location functions instead of using Axios.
It looks like you might have a typo in your get handler... you're referencing an element called 'path', but that's not declared in your schema
router.get('/download', async(req,res,next) => {
const x = await UserToUploadMapping.find()
// res.send(x)
res.download(x[0].path)//<-Path Doesn't seem to be in the schema
})
Since you don't have a try/catch in that function, the resulting error could be bringing down your server, making it unavailable
You might also want to take a look at this for more detail on How to download files using axios
I'm trying to dynamically export modules. I'm close but can't figure out how to fix my syntax.
Hard coded:
// index.js inside folder 'models'
const { User } = require('./User');
const { Token } = require('./Token');
const { Session } = require('./Session');
module.exports = {
User,
Token,
Session,
};
Dynamically coded (doesn't work):
// index.js inside folder 'models'
const fs = require('fs');
const path = require('path');
module.exports = () => {
fs.readdirSync(__dirname).forEach((file) => {
if (file === 'index.js') return false;
const fullName = path.join(__dirname, file);
if (file.toLowerCase().indexOf('.js')) {
// I think this somehow needs to be destructured like
// `return {require(fullName)}` or
// `require(fullName)[fullName]` I think
require(fullName);
}
});
};
Elsewhere in my code, I initialize it based on the folder name:
// server.js
require('./models')();
Your dynamic export will not work because you are not returning anything to the exported function.
Try this code as your dynamic model export file
// index.js inside folder 'models'
const fs = require('fs')
const path = require('path')
const models = {}
fs.readdirSync(__dirname)
.filter(file => file !== 'index.js')
.forEach(file => {
const fullName = path.join(__dirname, file)
if (file.toLowerCase().endsWith('.js')) {
// Removes '.js' from the property name in 'models' object
const [filename] = file.split('.')
models[filename] = require(fullName)[filename]
}
})
module.exports = models
This approach no longer exports a function so your require in server.js should now look like this
// server.js
require('./models');
Hello sorry for my english, i have a little problem. i try to upload many images but in back side i have just one image, (i use React express formidable cloudinary) here is my code front :
const [arrayFiles, setArrayFiles] = useState([]);
const handleFiles = (e) => {
let arrayUpload = [...arrayFiles];
arrayUpload.push(e.target.files[0]);
setArrayFiles(arrayUpload);
};
const handleSubmit = async (e) => {
arrayFiles.forEach((file) => {
formData.append("image", file);
});
const response = await axios.post(
"http://localhost:3100/offer/publish",
formData
);
here is my code back but req.files => just one image
my page route :
router.post("/offer/publish", async (req, res) => {
console.log(req.files);
const result = await cloudinary.uploader.upload(req.files.image.path, {
folder: `api/leboncoin/offers/${newOffer._id}`, // _id vient de la création du newOffer au dessus
public_id: "preview",
cloud_name: process.env.CLOUDINARY_NAME,
});
my page index.js:
page index.js :
const express = require("express");
const formidable = require("express-formidable");
const mongoose = require("mongoose");
const cloudinary = require("cloudinary").v2;
const cors = require("cors");
const app = express();
app.use(cors());
app.use(formidable({ multiples: true }));
You only get one file in req.file as you've set your multer.single
Using multer
There are 3 ways you can handle multiple file upload, each with a slightly different taste.
Assume you have a base multer
const storage = multer.diskStorage({
destination: "public/data/",
filename: function(req, file, cb){
// You may change this to however you want, only affect your file name
crypto.randomBytes(20, (err, buf) => {
cb(null, buf.toString("hex") + path.extname(file.originalname))
})
}
});
const upload = multer({ storage: storage });
Use .any()
Accepts all files that comes over the wire. An array of files will be stored in req.files.
WARNING: Make sure that you always handle the files that a user uploads. Never add multer as a global middleware since a malicious user could upload files to a route that you didn't anticipate. Only use this function on routes where you are handling the uploaded files.
router.post("/offer/publish",upload.any(), async (req, res) => {
console.log(req.files); // Should give you an array of files
// Do anything else
});
Use .array(fieldname[, maxCount])
Accept an array of files, all with the name fieldname. Optionally error out if more than maxCount files are uploaded. The array of files will be stored in req.files.
router.post("/offer/publish",upload.array('someFieldName', 10), async (req, res) => {
console.log(req.files); // Should give you an array of files
// Do anything else
});
Use .fields(fields)
Accept a mix of files, specified by fields. An object with arrays of files will be stored in req.files.
fields should be an array of objects with name and optionally a maxCount. Example:
router.post(
"/offer/publish",
upload.fields([
{
name: "image",
maxCount: 1,
},
{
name: "audio",
maxCount: 1,
},
]),
async (req, res) => {
console.log(req.files.image[0]);
console.log(req.files.audio[0]);
// Do anything else
}
);
For your case, I would recommend going with Option 2.
This is a totally reduced example to better explain the issue! So when I use the resolver Query getAllUsers, the MongoDB Collection Users is not available in the external resolver file user.js. So when I send that query I get:
ReferenceError: Users is not defined
That's a correct behaviour. But I do not want to include all the resolvers in my index.js, because I have a better modularization in this way. So I have all my typedefs and resolvers in external files like this.
Current file structure
index.js
/graphql
/typdef
user.graphql
/resolver
user.js
The user.graphql schema is correctly working. It is just the user.js that is producing the error when I execute the query with the not available Users variable, as already said.
Here the index.js and user.js.
index.js
import express from 'express'
import cors from 'cors'
const app = express()
app.use(cors())
import bodyParser from 'body-parser'
import {graphqlExpress, graphiqlExpress} from 'graphql-server-express'
import {makeExecutableSchema} from 'graphql-tools'
import {fileLoader, mergeTypes, mergeResolvers} from 'merge-graphql-schemas';
import {writeFileSync} from 'fs'
const typeDefs = mergeTypes(fileLoader(`${__dirname}/graphql/typedef/*.graphql`), { all: true })
writeFileSync(`${__dirname}/graphql/typedef.graphql`, typeDefs)
export const start = async () => {
try {
const MONGO_URL = 'mongodb://localhost:27017'
const MongoClient = require('mongodb').MongoClient;
MongoClient.connect(MONGO_URL, function(err, client) {
console.log("Connected successfully to server");
const db = client.db('project');
const Users = db.collection('user')
});
const URL = 'http://localhost'
const homePath = '/graphql'
const PORT = 3001
app.use(
homePath,
bodyParser.json(),
graphqlExpress({schema})
)
app.use(homePath,
graphiqlExpress({
endpointURL: homePath
})
)
app.listen(PORT, () => {
console.log(`Visit ${URL}:${PORT}${homePath}`)
})
} catch (e) {
console.log(e)
}
}
user.js
export default {
Query: {
getAllUsers: async () => {
return (await Users.find({}).toArray()).map(prepare)
}
}
}
What is the best way to pass the MongoDB or the Users collection to the resolver files. Or is there an even better solution for this issue?
First of all, this is NOT a proper solution, because declaring global variables while outsourcing schema is a bad design at all. But it works out and maybe this way someone gets an idea about how to improve this fix.
So to solve the issue all I had to do is changing the variable from local const to global.
So in index.js const Users = db.collection('user') is rewritten by global.Users = db.collection('user').
Same for the user.js. Here return (await Users.find({}).toArray()).map(prepare) is rewritten by return (await global.Users.find({}).toArray()).map(prepare).
I want to use koa-views with Koa and Koa-Router with Next.js. In previous projects, I had no issues with express but in this project, I have to use Koa. Using its router, I want to render a page: /some/page/:id. Following the same Nextjs way:
router.get('/some/page/:id', async (ctx, next) => {
const actualPage = '/some/page/id' // id.js (not actual name 😝)
await ctx.render(actualPage, {/* could pass object */})
});
That would work if I was using express. With Koa:
const Koa = require('koa');
const views = require('koa-views');
// const render = require('koa-views-render'); <-- I what's this?
[..] // Making things short here
const server = new Koa();
const router = new Router();
// My issue, I'm seeing tutorials using other engines: .ejs etc
// I'm not using any, I only have .js files
server.use(views(__dirname + "/pages", { extension: 'js' }));
Using the same router.get... function as above, I get:
Error: Engine not found for the ".js" file extension
When I go to /some/page/123, I'd expect it to render the file /pages/some/page/id.js. How?
It turns out I do not need any extra modules to achieve this 🙀
Create a function called, ie, routes then pass app and router as a param
const routes = (router, app) => {
router.get('/some/page/:id', async (ctx) => {
const { id } = ctx.params
const actualPage = '/some/page/id'
// Render the page
await app.render(ctx.req, ctx.res, actualPage, {foo: 'Bar'})
}
}
module.exports = routes
Inside your server.js file:
// const routes = require('./routes);
// const app = next({ dev }); // import other modules for this section
// app.prepare().then(() => {
// const router = new Router();
// [..]
// routes(router, app)
// })
The commented out section is a slim down version to make a point in where things should be.