I have a complicated project (not my code...) where a Flask servers launches computations using SCOOP -- that is, in another thread.
I'd like to know how I can send intermediary data from my SCOOP thread to display it on my Flask web page. I am not afraid of a little Javascript.
Python websockets seems like the way to go, but I'm unsure how to use it.
Let's say I will use Javascript this way in my Flask web page to fetch my data (example from Websockets' doc) :
var ws = new WebSocket("ws://127.0.0.1:5678/");
var messages = document.createElement('ul');
ws.onmessage = function (event) {
// Do stuff
}
Now, all my data is encapsulated in an object (Calibration2 -- again, not my code!). So, the following example (still from Websockets' doc) does not suit me:
#asyncio.coroutine
def time(websocket, path):
while True:
now = datetime.datetime.utcnow().isoformat() + 'Z'
yield from websocket.send(now)
yield from asyncio.sleep(random.random() * 3)
start_server = websockets.serve(time, '127.0.0.1', 5678)
Because I want the handler coroutine to be part of my Calibration2 class, and call it whenever Calibration2 wants to. But according to Websockets, the coroutine has to have this prototype, with websocket and path. But how can I access Calibration2's insides from such a function?
I'm pretty sure my issue is mostly a misunderstanding of Python scopes. I'm not a pro (yet!). So, if you can point me in some direction, I'd be glad, thanks!
Related
currently I have a generator function that is called by a charting program in the javascript side of my webapp.
I am using the #stream_with_context decorator but that unfortunately only gives me the context when the generator begins running but I want the context constantly updated as I have a boolean variable that the generator needs to know that is constantly changing and I am updating it in the session dictionary.
I would like to be able to access this updated session in the generator. Or if there is any way I can communicate a boolean variable to the generator that would be fine too.
I don't care how it gets there I just thought using session would be the easiest.
Status Update: I have still not found a way to do this... It seems pretty simple but there is nothing on the internet about requesting the current context.
Flask
#app.route('/plotter', methods=['GET', 'POST'])
def plotter():
#some setup in here
#stream_with_context
def generate_data():
while True:
if(session['bool']):
#do something
#app.route('/elsewhere')
def bool_status():
session['bool'] = #new_val <---- this where the val is updated... in another view called by a button.
Javascript
// this how the generator is called
const source = new EventSource("{{url_for(source)}}");
I've got a web service under development that uses Django and Django Channels to send data across websockets to a remote application. The arrangement is asynchronous and I pass information between the 2 by sending JSON formatted commands across websockets and then receive replies back on the same websocket.
The problem I'm having is figuring out how to get the replies back to a Javascript call from a Django template that invokes a Python function to initiate the JSON websocket question. Since the command question & data reply happen in different Django areas and the originating Javascript/Python functions call does not have a blocking statement, the Q&A are basically disconnected and I can't figure out how to get the results back to the browser.
Right now, my idea is to use Django global variables or store the results in the Django models. I can get either to work, but I beleive the Django global variables would not scale beyond multiple workers from runserver or if the system was eventually spread across multiple servers.
But since the reply data is for different purposes (for example, list of users waiting in a remote lobby, current debugging levels in remote system, etc), the database option seems unworkable because the reply data is varying structure. That, plus the replies are temporal and don't need to be permanently stored in the database.
Here's some code showing the flow. I'm open to different implementation recommendations or a direct answer to the question of how to share information between the 2 Django functions.
In the template, for testing, I just have a button defined like this:
<button id="request_lobby">Request Lobby</button>
With a Javascript function. This function is incomplete as I've yet to do anything about the response (because I can't figure out how to connect it):
$("#request_lobby").click(function(){
$.ajax({
type: "POST",
url: "{% url 'test_panel_function' %}",
data: { csrfmiddlewaretoken: '{{ csrf_token }}', button:"request_lobby" },
success: function(response){
}
});
});
This is the Django/Python function in views.py . The return channel for the remote application is pre-stored in the database as srv.server_channel when the websocket is initially connected (not shown):
#login_required
def test_panel_function(request):
button = request.POST.get('button', '')
if button == "request_lobby" :
srv = Server.objects.get(server_key="1234567890")
json_res = []
json_res.append({"COMMAND": "REQUESTLOBBY"})
message = ({
"text": json.dumps(json_res)
})
Channel(srv.server_channel).send(message)
return HttpResponse(button)
Later, the remote application sends the reply back on the websocket and it's received by a Django Channels demultiplexer in routing.py :
class RemoteDemultiplexer(WebsocketDemultiplexer):
mapping = {
"gLOBBY" : "gLOBBY.receive",
}
http_user = True
slight_ordering = True
channel_routing = [
route_class(RemoteDemultiplexer, path=r"^/server/(?P<server_key>[a-zA-Z0-9]+)$"),
route("gLOBBY.receive" , command_LOBBY),
]
And the consumer.py :
#channel_session
def command_LOBBY(message):
skey = message.channel_session["server_key"]
for x in range(int(message.content['LOBBY'])):
logger.info("USERNAME: " + message.content[str(x)]["USERNAME"])
logger.info("LOBBY_ID: " + message.content[str(x)]["LOBBY_ID"])
logger.info("OWNER_ID: " + message.content[str(x)]["IPADDRESS"])
logger.info("DATETIME: " + message.content[str(x)]["DATETIME"])
So I need to figure out how to get the reply data in command_LOBBY to the Javascript/Python function call in test_panel_function
Current ideas, both of which seem bad and why I think I need to ask this question for SO:
1) Use Django global variables:
Define in globals.py:
global_async_result = {}
And include in all relevant Django modules:
from test.globals import global_async_result
In order to make this work, when I originate the initial command in test_panel_function to send to the remote application (the REQUESTLOBBY), I'll include a randomized key in the JSON message which would be round-tripped back to command_LOBBY and then global_async_result dictionary would be indexed with the randomized key.
In test_panel_function , I would wait in a loop checking a flag for the results to be ready in global_async_result and then retrieve them from the randomized key and delete the entry in global_async_result.
Then the reply can be given back to the Javascript in the Django template.
That all makes sense to me, but uses global variables (bad), and seems that it wouldn't scale as the web service is spread across servers.
2) Store replies in Django mySQL model.py table
I could create a table in models.py to hold the replies temporarily. Since Django doesn't allow for dynamic or temporary table creations on the fly, this would have to be a pre-defined table.
Also, because the websocket replies would be different formats for different questions, I could not know in advance all the fields ever needed and even if so, most fields would not be used for differing replies.
My workable idea here is to create the reply tables using a field for the randomized key (which is still routed back round-trip through the websocket) and another large field to just store the JSON reply entirely.
Then in test_panel_function which is blocking in a loop waiting for the results, pull the JSON from the table, delete the row, and decode. Then the reply can be given back to the Javascript in the Django template.
3) Use Django signals
Django has a signals capability, but the response function doesn't seem to be able to be embedded (like inside test_panel_function) and there seems to be no wait() function available for an arbitrary function to just wait for the signal. If this were available, it would be very helpful
connection = new WebSocket("ws://localhost:1050/join?username=test")
connection.onopen = function(){
alert('Connection open!');
}
connection.onmessage = function(e){
var server_message = e.data;
alert(server_message);
}
connection.onclose = function() {
alert("websocket closing")
}
The connection to the server is established and an alert is displayed for Connection open! However immediately afterwards the connection closes. The server does not call close and there seem to be no other errors in the console. This is happening in both chrome and firefox.
I looked at a bunch of different similar examples on the web but to no avail.
to Keep Websocket Opened prevent handler from returning by return false; in connection.onmessage
like this :
connection.onmessage = function(e){
var server_message = e.data;
alert(server_message);
return false;
}
I believe I've stumbled across the solution that OP found but failed miserably to explain. I don't have enough reputation to comment, otherwise I'd be responding to all of the confused comments begging for clarification on OP's response.
The short version is that I think OP was referring to his server-side connection handler when he said "All I had to do was block the handler from returning before the websocket connection closes".
It turns out my server was closing the webSocket automatically because I didn't understand how a certain webSocket function worked. Specifically, I was using a Python server script with asyncio/websockets and the following code:
async def receiveCommandsLoop(player):
while True:
msg = await player.websocket.recv()
print(command)
async def handleClient(websocket, path):
username = await websocket.recv()
player = players[username]
...
#Start task to listen for commands from player
asyncio.get_event_loop().create_task(receiveCommandsLoop(player))
start_server = websockets.serve(handleClient, '', 8765)
The idea was that websockets.serve would use handleClient to begin the connection and do some setup, then create a new task with receiveCommandsLoop that would take over the job of communication.
But it turns out: when you call websockets.serve, Python expects that when your handler (in this case, handleClient) returns, you must be done with the socket, and it closes it automatically.
Thus, by the time receiveCommandsLoop was run, handleClient had returned, and the webSocket had been automatically closed.
I was able to fix this by simply modifying my handleClient function to directly run the loop originally contained in receiveCommandsLoop. Hope this helps someone out there.
This also could be the case when you're trying to send binary data over a websocket connection, but some side (client or server) is trying to interpret it as a text - many libraries and frameworks do it unless you explicitly specify you do want binary data.
It could also be a login problem. The websocket will automatically close the website required authentication but no authentication information was provided.
Piecing together hints from this post and others, I found a solution that works when using the python websocket server example found everywhere that includes something like:
async def handler(websocket, path):
data = await websocket.recv()
reply = f"Data recieved as: {data}!"
await websocket.send(reply)
To those of us new to websocket, I think the assumption is that the handler function will be called each time the client sends a message, which turns out not to be the case. As others mention, the connection closes as soon as the handler function returns once. The solution I found is to change it to:
async def handler(websocket, path):
async for data in websocket:
reply = f"Data recieved as: {data}!"
print(data)
await websocket.send(reply)
My client-side javascript code is equivalent to the OP's and I didn't have to change anything for this to work.
Unfortunately I can't explain why async for data in websocket: makes it actually wait forever and spontaneously run the inner code block each time a message is received, but it does for me and I get all the expected log messages both on the python server side and the client javascript console.
If anyone more knowledgeable on this topic can comment on whether this is a good-for-general-use solution or if there's a gotcha to look out for here, it would be much appreciated.
Fixed it!
All I had to do was block the handler from returning before the websocket connection closes
I'm trying transmit an image file from the server to the client, but my javascript callback becomes active before the stream closes I doing this because sending it in a traditional render json: times out and takes way to long anyway. The stream takes much less time, but i keep can't get all the data before the callback fires up.
controller code
def mytest
image=ImageList.new(AssistMe.get_url(image_url))
response.stream.write image.export_pixels(0, 0, image.columns, image.rows, 'RGBA').to_s
response.stream.close
end
javascript
var getStream, runTest;
runTest = function() {
return $.post('/dotest', getStream);};
getStream = function(params) {
return document.getElementById('whatsup2').innerHTML =
"stream is here " + params.length;};
the response is an array, I can make it an array of arrays by adding a "[" at the front and a "],['finish'] at the end to be able to detect the end of the data, but I haven't been able to figure out how to get javascript to wait until the end of stream to run. I assume i need to set up some kind of pole to check for the end, but how do I attach it to the callback?
Okay, here's a blog that describes this pretty well
blog
But i decided to forgo a stream and use .to_s. Since you can pipe several actions tougher
render object.method.method.to_s you get all the server side benefits of using a stream without the complexity. If you have a slow process where you need to overlap the client and server actions, then go to the blog and do it. Otherwise to_s covers it pretty well
I am making an AJAX chat room with the guidance of an AJAX book teaching me to use JSON and eval() function.
This chat room has normal chat function and a whiteboard feature.
When a normal text message comes from the php server in JSON format, the javascript in browser does this:
Without Whiteboard Command -------------------------------------------
function importServerNewMessagesSince(msgid) {
//loadText() is going to return me a JSON object from the server
//it is an array of {id, author, message}
var latest = loadText("get_messages_since.php?message=" + msgid);
var msgs = eval(latest);
for (var i = 0; i < msgs.length; i++) {
var msg = msgs[i];
displayMessage(escape(msg.id), escape(msg.author), escape(msg.contents));
} ...
The whiteboard drawing commands are sent by server in JSON format with special user name called "SVR_CMD", now the javascript is changed slightly:
With Whiteboard Command --------------------------------------------------
function importServerNewMessagesSince(msgid) {
//loadText() is going to return me a JSON object from the server
//it is an array of {id, author, message}
var latest = loadText("get_messages_since.php?message=" + msgid);
var msgs = eval(latest);
for (var i = 0; i < msgs.length; i++) {
var msg = msgs[i];
if (msg.author == "SVR_CMD") {
eval(msg.contents); // <-- Problem here ...
//I have a javascript drawLine() function to handle the whiteboard drawing
//server command sends JSON function call like this:
//"drawLine(200,345,222,333)" eval() is going to parse execute it
//It is a hacker invitation to use eval() as someone in chat room can
//insert a piece of javascript code and send it using the name SVR_CMD?
else {
displayMessage(escape(msg.id), escape(msg.author), escape(msg.contents));
}
} ...
Now, if the hacker changes his username to SVR_CMD in the script, then in the message input start typing javascript code, insdead of drawLine(200,345,222,333), he is injecting redirectToMyVirusSite(). eval() will just run it for him in everyone's browser in the chat room.
So, as you can see, to let the eval to execute a command from an other client in the chat room is obviously a hacker invitation. I understand the book I followed is only meant to be an introduction to the functions. How do we do it properly with JSON in a real situation?
e.g. is there a server side php or .net function to javascriptencode/escape to make sure no hacker can send a valid piece of javascript code to other client's browser to be eval() ? Or is it safe to use JSON eval() at all, it seems to be a powerful but evil function?
Thank you,
Tom
What is this book? eval is evil, there is not a single reason to use it, ever.
To transform a JSON string into a javascript object, you can do the following:
var obj = JSON.parse(latest)
Which means you can then use:
[].forEach.call(obj, function( o ) {
// You can use o.message, o.author, etc.
} )
To do the opposite (javascript object -> JSON string), the following works:
var json = JSON.stringify(obj)
It only is unsafe if the executed code is generated by other clients and not by the server. Of course you would need to prevent anybody to use that name, though I don't understand why you would use the "author" field? Just send an object {"whiteboard":"drawLine(x,y,z)"} instead of {"author":"SVR_CMD","contents":"drawLine(x,y,z)"}.
But it is right, eval() is still an invitation for hackers. One can always send invalid data and try to influence the output more or less directly. The only way for escaping is a proper serialisation of the data you want to receive and send - the drawings data. How do you receive the whiteboard commands? There is no serverside "escape" function to make javascript code "clean" - it would always be a security hole.
I would expect a serialisation like
message = {
"author": "...", // carry the information /who/ draws
"whiteboard": {
"drawline": [200, 345, 222, 333]
}
}
so you can sanitize the commands (here: "drawline") easiliy.
The use of eval() might be OK if you have very complex commands and want to reduce the transferred data by building them serverside. Still, you need to parse and escape the received commands from other clients properly. But I'd recommend to find a solution without eval.
Setting eval issue aside, do not use field that can be filled by user - .author in your code - for authentication purposes. Add another field to your JSON message, say .is_server_command that when present, would signify special treating of message. This field is will be not depended on user input and thus wouldn't be hijacked by "hacker".