I am writing a simple app to generate content for a blog. So, a POST request to the express server sends an object that contains html string. While I was testing the api, I got the error as Maximum call stack size exceeded once. The html also contains base64 encoded images. But, the size of the html content was about 1.5MB when i got the error.
I tested again with contents having size more than 10MB, but worked fine. I don't know why it happened once.
My code to handle the post request looks something like this:
async function create(postData){
if(await Post.findOne({title: postData.title})){
throw 'Post with same title already exists!';
}
// find the actual user(user id is sent in req.body)
const user = await User.findById(postData.userId);
if(user === null || user === undefined) {
throw 'Cannot find user with given user id while creating the post!';
}
// create new post
const newPost = postData;
delete newPost.userId;
const post = new Post(newPost);
post.user = user;
await post.save();
// add post to user
user.posts.push(post);
await user.save();
return post;
}
I thought user.posts.push(post); took more time to execute, thus blocking the call stack but every other tests succeeded with larger object size. I thought of using streams like JSONStream to handle large objects but even after testing with object sizes around 15MB worked fine.
What am i missing here?
Related
The request.files method is great. I can map through multipart files and push them to s3 with the moveToDisk method. However, it seems that when moveToDisk returns an error or anything else it interrupts the loop. I'm pretty sure I have things at least close to set up correctly. I can successfully upload the first file and even store information to my database. But Anything beyond one file doesn't work. Anyone else having trouble with files or moveToDisk
const sprites = request.files("sprites");
for (let sprite of sprites) {
const collection = request.all().collection;
const uuid = uuidv4();
await client
.db("pixel-shop")
.collection("sprites")
.insertOne({
collection: collection,
token: sprite.fileName,
path: `pixel-shop/${collection}/${uuid}`,
created_at: new Date(),
});
await sprite?.moveToDisk(
`pixel-shop/sprites/${request.all().collection}`,
{ name: uuid }
);
}
Some things I've noticed in the logs are a cannot write file error which is weird because the first file writes fine, something about a .getUrl method, as well as some stuff about SSL but all my stuff aside from my localhost are secure.
I'm trying to make a transaction with mongoose.
When I'm running the code, it looks like mongoose document remembers the session even after calling await session.endSession(). And later, when I call .save() on that document, I get error "Use of expired sessions is not permitted".
This is the demo (just a simplified example):
const MessageSchema = new mongoose.Schema({ text: String }, {strict: 'throw'});
const Message = mongoose.model('Message', MessageSchema);
const session = await mongoose.startSession();
let message;
await session.withTransaction(async () => {
message = (await Message.create([{text: 'My message'}], {session}))[0];
});
await session.endSession();
message.text = 'My message 2';
await message.save();
That final .save() is throwing that error. If you would log message.$session().hasEnded you would get true. My expectation was that, if session has ended, .save() would be smart not to use it.
What I want to achieve:
Create some documents with transaction, commit them, so they are in the database.
Later, use that same document, that is already in the database, to make some changes to it.
What am I doing wrong here? How can I prevent .save() from throwing an error and trying to use expired session?
There are no sections on endSession anywhere in the mongoose docs, but as you very well discovered it yourself, when this function is called, it only sets a flag to your session object rather than destroying it. If it was destroyed, you would have another error.
My expectation was that, if session has ended, .save() would be smart not to use it.
It is, in fact, smart enough not to use it, the framework only informs you that you are attempting an illegal instruction.
What I want to achieve: Create some documents with transaction, commit them, so they are in the database.
That's exactly what you do with your withTransaction call. This wrapper helps you create, commit and abort/retry in case something bad happened, so after that call and if all ended well you indeed successfully created a document in the database.
Later, use that same document, that is already in the database, to make some changes to it.
Depends on what you mean by "later". If it's part of the same endpoint (and I don't know why you would immediately modify that document rather than committing it rightfully in the first place), then as I said in the comments, moving the endSession call would likely fix the issue:
const MessageSchema = new mongoose.Schema({ text: String }, {strict: 'throw'});
const Message = mongoose.model('Message', MessageSchema);
const session = await mongoose.startSession();
let message;
await session.withTransaction(async () => {
message = (await Message.create([{text: 'My message'}], {session}))[0];
});
message.text = 'My message 2';
await message.save();
await session.endSession();
If it's part of another endpoint, then just make another transaction like you just did, except you modify the document instead of creating it. Or if you don't need transaction at all, use a method like findOneAndUpdate or findByIdAndUpdate. I reckon you seem to be familiar enough with JS and Mongoose to do that on your own.
What am I doing wrong here?
Basically, not much. You already understood that you can't call .save() after ending a session.
How can I prevent .save() from throwing an error and trying to use expired session?
You can't. It is merely a reminder that you are attempting an illegal operation. You can however try/catch the error and decide to do nothing about it in your catch clause, OR write a if statement checking for message.$session().hasEnded.
I'm trying to overcome Call stack size exceeded error but with no luck,
Goal is to re-run the GET request as long as I get music in type field.
//tech: node.js + mongoose
//import components
const https = require('https');
const options = new URL('https://www.boredapi.com/api/activity');
//obtain data using GET
https.get(options, (response) => {
//console.log('statusCode:', response.statusCode);
//console.log('headers:', response.headers);
response.on('data', (data) => {
//process.stdout.write(data);
apiResult = JSON.parse(data);
apiResultType = apiResult.type;
returnDataOutside(data);
});
})
.on('error', (error) => {
console.error(error);
});
function returnDataOutside(data){
apiResultType;
if (apiResultType == 'music') {
console.log(apiResult);
} else {
returnDataOutside(data);
console.log(apiResult); //Maximum call stack size exceeded
};
};
Your function returnDataOutside() is calling itself recursively. If it doesn't gets an apiResultType of 'music' on the first time, then it just keeps calling itself deeper and deeper until the stack overflows with no chance of ever getting the music type because you're just calling it with the same data over and over.
It appears that you want to rerun the GET request when you don't have music type, but your code is not doing that - it's just calling your response function over and over. So, instead, you need to put the code that makes the GET request into a function and call that new function that actually makes a fresh GET request when the apiResultType isn't what you want.
In addition, you shouldn't code something like this that keeping going forever hammering some server. You should have either a maximum number of times you try or a timer back-off or both.
And, you can't just assume that response.on('data', ...) contains a perfectly formed piece of JSON. If the data is anything but very small, then the data may arrive in any arbitrary sized chunks. It make take multiple data events to get your entire payload. And, this may work on fast networks, but not on slow networks or through some proxies, but not others. Instead, you have to accumulate the data from the entire response (all the data events that occur) concatenated together and then process that final result on the end event.
While, you can code the plain https.get() to collect all the results for you (there's an example of that right in the doc here), it's a lot easier to just use a higher level library that brings support for a bunch of useful things.
My favorite library to use in this regard is got(), but there's a list of alternatives here and you can find the one you like. Not only do these libraries accumulate the entire request for you with you writing any extra code, but they are promise-based which makes the asynchronous coding easier and they also automatically check status code results for you, follow redirects, etc... - many things you would want an http request library to "just handle" for you.
I want to create a basic chat system which allows the user to attach a file to a given message.
This is a problem with web sockets because it is very difficult to send a file through them. unless you want to convert it to binary data. That wont work for me however because I want to be able to interact with the files on the server, and since i have multiple file types (different images, videos and gifs), I cant just decode the binary.
Here is my code:
Where I make the request to upload the message:
// this.state._new_message : {
// _message_type_id:int,
// _message:string || file,
// _user_id:int,
// _conversation_id:int
// } -> this is what a _new_message object looks like.
let fd = new FormData()
if(this.state._new_message._message_type_id !== 1){
fd.append("file", this.state._new_message._message)
}
fd.append("message", JSON.stringify(this.state._new_message))
Axios.post(API_URL+"/api/chat/send_message", fd)
.then(res => res.data)
.then(res => {
this.socket.emit("message", res._message)
})
Where I handle the request:
def send_message(self, request):
data = request.form["message"]
data = json.loads(data)
if data["_message_type_id"] != 1:
data["_message"] = "Loading Message. Please Wait."
query = fdb.prepare(data, "_messages") # Converts the data dict to a query string
fdb.create(query["query_string"], query["query_params"]) # Interacts with Database
_message = fdb.read("""SELECT * FROM _messages WHERE _id = LAST_INSERT_ID()""", (), False)
if data["_message_type_id"] != 1:
file = request.files["file"]
# OpenCV magic... #
path = "path/to/file"
file.save(path)
fdb.update("""UPDATE _messages SET _message = %s WHERE _id = %s""", (url_path, _message["_id"])) # Interacts with Database
url_path = "url/path/to/file"
_message["_message"] = url_path # If message is not text, set it to be the path to the file instead
return jsonify({
"_message" : _message
})
This is where I listen for the event (server):
#socket.on("message")
def handleMessage(msg):
print(msg)
send(msg, broadcast=True)
return None
This is where I listen for the event (client):
this.socket.on("message", data => {
if(data == null || data._message_type_id == undefined) return
if(this.props.chat.messages.messages !== null)
if(this.props.chat.messages.messages.filter(message => parseInt(message._id) === parseInt(data._id)).length > 0) return
this.props.addMessage(data)
})
From this code, you'd expect to send a message through HTTP, that message to be processed and sent back through HTTP, then the response to be sent to the server through the socket, and the socket to then send that message to all the clients which are listening to that socket. But it doens't work that way. Some times it does, other times it doesn't.
One thing I can confirm is that the messages are saved to the database and file system 100% of the time. I also always get a response from the HTTP request, but when I emit the event from the client and print them message in the server, I some times get None or the previous message logged to the console.
So when that message is sent to the client, my "error catching block" will see that its either undefined or already exists, so it will return without adding it (because it shouldnt add it of course).
Has anyone experienced this issue before? Do I have a wrong idea about something? Or is there wrong with my code?
I figured it out.
After a bit of time analyzing my code, I noticed that I do not execute the following two queries in the same transaction:
fdb.create(query["query_string"], query["query_params"]) # Interacts with Database
_message = fdb.read("""SELECT * FROM _messages WHERE _id = LAST_INSERT_ID()""", (), False)
The fdb extension I developed is basically a high level abstraction of sqlalchemy which has a very loose transaction management system (auto open/close). So I had to implicitly tell it to not close the transaction until it executes both queries.
As a result, I always get the latest insert, and not the insert which occurred beforehand (as was the case before I updated my code).
Goes to show you that abstraction of code isn't always a good idea if you don't think of all the use cases in advanced.
I'm trying to figure out how to create calls from my Twilio number after I've dialed into it and entered a number. After reading the docs I see that this is done with the gather feature, which can then be redirected to another Twiml document to handle the response. However, I can't quite get it to work. I'm extremely confused on how to execute Twiml correctly and how to access the request parameters in another Twiml doc. I've also looked into Twimlets but I wasn't able to construct what I needed there correctly either.
I've gone back and tried to just make a simple voice message play when only my number calls. If it's not me calling then it needs to be redirected to a Twiml url which will try to connect to my phone. If that fails it will prompt the caller to leave a message.
//Handle incoming call requests
app.post('/call', function(req, res) {
var twiml = new twilio.TwimlResponse();
res.type('text/xml');
if ( req.body.From === "+1555555555") {
twiml.say('Hello', {voice: alice});
res.send(twiml.toString());
} else {
// Do something here.
}
});
I've found the correct solution for my problem. I wasn't initiating the twilio.TwimlResponse() correctly.
In order to solve this issue, I needed to use == instead of === so that my req.body.from value wouldn't get coerced.