Flask socketio multithread dump dictionary to queue and send to clients - javascript

Goal is simply to be able to make a thread queue of dictionaries and report them to client.
EDIT
This is different of Flask throwing 'working outside of request context' when starting sub thread because:
It is not done in a route function, it is done in socketio.start_background_task
The only socketio code takes place in context, with the socketio.emit we are sending a dictionary.
Strategy:
There are 2 different taks to perform in server side, for each build a thread, then in another socketio thread collect the results which are in a thread safe queue FIFO of dictionaries.
Then send these dictionaries to client and wait for each acknowledge.
So now the issue is reduced to solve:
RuntimeError: Working outside of request context.
from flask import Flask, flash, request, redirect, render_template, Response, escape, jsonify, url_for, session, copy_current_request_context
#socketio
from flask_socketio import SocketIO, send, emit, join_room, leave_room, close_room, rooms, disconnect
import threading
from threading import Thread, Event, Lock
import queue
import random
def ack(value):
if value != 'pong':
logger.info('unexpected return value')
def fn_i():
global q
while True:
time.sleep(1)
q.put({'key_i':random.random()})
return q
def fn_ii():
global q
while True:
time.sleep(10)
q.put({'key_ii':random.random()})
return q
app = Flask(__name__)
socketio = SocketIO(app, async_mode=async_mode)
thread1=None
thread2=None
collector_thread=None
q = queue.Queue()
thread_lock = Lock()
def background_thread_collector():
global thread1
global thread2
global q
thread1 = threading.Thread(target=fn_i)
thread1.start()
thread2 = threading.Thread(target=fn_ii)
thread2.start()
"""Example of how to send server generated events to clients."""
while True:
time.sleep(0.2)
while not q.empty():
socketio.emit('my_response',
q.get(), #{'data': 'Server generated event', 'count': count},
namespace='/test',
broadcast=True,
callback=ack
)
#app.route('/')
def index():
return render_template('index.html', async_mode=socketio.async_mode)
#socketio.on('connect', namespace='/test')
def test_connect():
global collector_thread
logger.info(' Client connected ' + request.sid)
with thread_lock:
if collector_thread is None:
collector_thread = socketio.start_background_task(background_thread_collector)
emit('my_response', {'data': 'Connected', 'count': 0})
if __name__ == '__main__':
socketio.run(app,
host='localhost',
port=10000,
debug=False) #True sends some exceptions and stops)
Cheers

This should be handled better by Flask-SocketIO, but the problem is that you are trying to use a callback on an emit that is set to broadcast to all clients:
socketio.emit('my_response',
q.get(), #{'data': 'Server generated event', 'count': count},
namespace='/test',
broadcast=True,
callback=ack
)
Remove the callback and the emit should work just fine.

Using threading queues with Flasksocketio is not straightforward because requires to handle the apps context among other things, for purpose of refreshing a server log file in client side found it was easier to simply use javascript in this case and the file can be updated accordingly. Even the former code with suggested alterations was not working because of apps context and nevertheless there are multiple blogs or stackoverflow which approach subject in none have found a complete working solution except implementing like explain because any other answer requires complete code which is working, and hence since able to implement here considering this is the accepted answer, Cheers.

Related

Data posted to flask endpoint from JS not processed in endpoint

I have written a simple todo app with react acting as a frontend and flask handling CRUD from a DB. The app is using axios to handle the requests; GET completes fine however when attempting to POST JSON the flask api returns a 400 error. Here's some condensed sample code.
JS POST function.
function testPost(){
axios.post('http://'+window.location.hostname+':8000/todo/', {
title: "test123",
}).then(res => {
console.log(res)
}).catch(err => {
console.log(err)
})
}
Serverside
class Todo(Resource):
def post(self): # create a new todo
conn = pool.getconn()
cur = conn.cursor()
app.logger.info(request.form['title'])
cur.execute("INSERT INTO todo (task, done) VALUES (%s, %s)", (request.form['title'], False))
conn.commit()
app.logger.error(e)
cur.close()
pool.putconn(conn)
Other methods not shown
Then the rest of the server code attaching the resource to the api and the CORS setup (not shown in file order)
app = Flask(__name__)
CORS(app, methods=['POST','GET','PUT','DELETE'])
api = Api(app)
api.add_resource(Todo, '/todo/')
app.run(debug = True, host='0.0.0.0', port=port)
Tests
Using python to test the api works fine, running this in a seperate python file will add to the DB.
response = requests.post(URL + "todo/", data={"title": f"test{randint(1, 100)}"})
My best guess is that axios is not adding the data to the request in a way that the backend is unable to process. Before using axios I tried to make the request with XMLHttprequest however this presented the same problem. I swapped to axios on the recommendation of someone else, given its alleged improved simplicity.
request.form['key'] and request.get_json()['key'] are completely different fields python requests in the way I used it posts to the former and js posts to the latter. Modifying the function to use whichever is available fixes this.

Flask socketio not receiving any message from javascript even after the connection is established

I am trying to get response from javascript after the connection is established between the flask and javascript.The onconnect() function is working properly but onmessage() is not.
I tried broadcast along with the emit method in javascript too but it's not working.
This is my app.py
app=Flask(__name__)
bootstrap=Bootstrap(app)
socketio=SocketIO(app)
app.config['SECRET_KEY']="MY_KEY"
#app.route('/')
def login():
return render_template('index.html')
#app.route('/home',methods=['GET','POST'])
def home():
if(request.method=='POST'):
data=request.form
name=data['name']
return render_template('message.html',name=name)
else:
return render_template('index.html')
#socketio.on('connect')
def onconnect():
print("connect")
#socketio.on('message')
def onmessage(message):
print('message')
if(__name__=='__main__'):
socketio.run(app)
and this is my javascript file
const ws =io.connect(window.location.href)
ws.on('connect', function() {
console.log("Connection estalished")
ws.emit("adsd",broadcast=true)
});
EDIT:
There is a mistake in the javscript.
const ws =io()
This should be used for establishing the connection , not the previous method.
My Project is completed.
Link for the github project
First of all, only the server can broadcast, clients can only emit to the server.
Second, you are emitting an event named adsd, so you need to add a handler for that event in your server. For example:
#socketio.on('adsd')
def adsd_handler():
print('got adsd!')

socketio: data loss due to socket getting closed due to transport error

I've been trying very hard to figure this out but no luck so far.
So I am sending some data from server to client using socket in a loop however the client is unable to receive all the data and closes the socket in the middle of data transmission for no reason.
As you can see in the image below client successfully receives data till 11th iteration of the loop(refer to the server code below) however after that socket is closed cause of transport error. What possibly could I be doing wrong here?
Client side logs
Server side logs
(Python) server side code
from flask import Flask, render_template, request, flash, redirect, jsonify, make_response
from flask_socketio import SocketIO, emit, disconnect
import time
from gevent import monkey
monkey.patch_all()
app = Flask(__name__)
socketio = SocketIO(app, engineio_logger=True)
#socketio.on('run_tgt')
def run_tg(tg_args):
for x in range(20):
time.sleep(2)
emit('tg_output',x)
if __name__ == "__main__":
socketio.run(app, host='0.0.0.0', debug=True)
(Javascript) client side code
<script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/2.0.3/socket.io.js"></script>
var socket = io.connect('http://' + document.domain + ':' + location.port );
function run_tg() {
socket.emit('run_tgt', { 'data': 'data'});
socket.on('tg_output', function(tg_output) {
console.log(tg_output);
$("#output_div").append(tg_output);
});
}
Fixed the issue by adding monkey patch to the top of the script
from gevent import monkey
monkey.patch_all()
Also set async_handlers to true
socketio = SocketIO(app, async_handlers=True, engineio_logger=True)

Python Eventlet + Javascript Websocket returns Blob

I am experimenting with Python's Eventlet Websocket support, using this simple example :
import eventlet
from eventlet import wsgi
from eventlet import websocket
from eventlet.support import six
# demo app
import os
import json
import time
import random
#websocket.WebSocketWSGI
def handle(ws):
""" This is the websocket handler function. Note that we
can dispatch based on path in here, too."""
if ws.path == '/echo':
while True:
ws.send("hello")
#ws.send(json.dumps({"msg":"hello"}))
time.sleep(1)
elif ws.path == '/data':
for i in six.moves.range(10000):
ws.send("0 %s %s\n" % (i, random.random()))
eventlet.sleep(0.1)
def dispatch(environ, start_response):
""" This resolves to the web page or the websocket depending on
the path."""
if environ['PATH_INFO'] == '/data':
return handle(environ, start_response)
else:
start_response('200 OK', [('content-type', 'text/html')])
return [open(os.path.join(
os.path.dirname(__file__),
'websocket.html')).read()]
if __name__ == "__main__":
# run an example app from the command line
listener = eventlet.listen(('127.0.0.1', 7000))
print("\nVisit http://localhost:7000/ in your websocket-capable browser.\n")
wsgi.server(listener, dispatch)
I'm not going to include the entire websocket handler I have in the Javascript, just the ws.onmessage method:
ws.onmessage = function (evt)
{
console.log(evt.data)
var received_msg = evt.data;
#Do stuff, i.e var obj = JSON.parse(received_msg)
#callback(obj)
};
The console.log(evt.data) indicates a succesful connection with the websocket (and you can assume this is all fine). However, the logging shows Blob {size: 31, type: ""} as the content of evt.data. I assume this is some kind of response object which is interpreted as binary (file) data (though I might be entirely wrong), but I'm not sure what to do with this.
I see that Blob data is often the data type for file-like objects. I suppose I could approach it as such, but I really only want to send json data back and forth. I've tried dumping a dict as a JSON and sending that, but it did the same. Even a string is received in Blob format.
How do I use eventlet for json data transmission?
As of 2017-05 Eventlet websocket API does not support string websocket frames. Patches are welcome, it's easy.
Your options:
read Blob at javascript end FileReader.readAsText
use another websocket library (pure Python implementation will work fine with Eventlet patching)
add string frame support to Eventlet websocket library

Python Flask-Sockets not working within blueprints

I am experimenting with Flask-Sockets. Support for Blueprints was added in the past, something I really do need.
from flask import Flask, request, abort, redirect, url_for, render_template, make_response, Response, jsonify, session
import json
#Importing blueprints
from play import play, play_socket
app = Flask(__name__)
sockets = Sockets(app)
app.register_blueprint(play, url_prefix=r"/play")
sockets.register_blueprint(play_socket, url_prefix=r"/play")
#sockets.route('/echo')
def echo_socket(ws):
while not ws.closed:
message = ws.receive()
response = json.dumps({"Message":message,"Response":"Message received"})
ws.send(response)
So, connecting a websocket in JavaScript and setting the endpoint to 'echo' (i.e. var ws = new WebSocket("ws://HOST:PORT/"+"echo")) works perfectly. I can send messages and they get bounced right back at me.
However, when I want to move this function to a blueprint (in the blueprint play_socket, it doesn't work anymore. Assume I changed the endpoint to '/status_ping' in the javascript:
#play_socket.route('/status_ping')
def ping(ws):
while not ws.closed:
message = ws.receive()
response = json.dumps({"Message":message,"Response":"Message received"})
ws.send(response)
The websocket is connected successfully from the client-side and I can confirm this by inserting a simple print("HERE") or whatever in the def ping(socket):, but it's immediately closed afterwards, rendering it useless.
I found out that if I move the body of the while not ws.closed: loop above the header (copy it), it 'works'. However, I can't use this as I need the socket to push data from the server to the clients. It seems to be going wrong when this while loop is executed. The socket is immediately closed for some reason. changing while not ws.closed: to while True: has no effect.
I tried to isolate the problem as much as I could, but please let me know if you need more info.
EDIT: Codesample for blueprint
from flask import Flask, request, abort, redirect, url_for, render_template, make_response, Response, jsonify, session, current_app
import sys
sys.path.append('../')
from flask_sockets import Sockets
import json
import time
api_blueprint = Blueprint('api_blueprint', __name__)
#sockets.route('/update_status')
def echo_socket(ws):
message = ws.receive()
while not ws.closed:
ws.send(json.dumps({"data":message}))
time.sleep(1)
if ws.closed:
session.clear()
print("session cleared")
sockets = Sockets(current_app)
sockets.register_blueprint(api_blueprint, url_prefix=r"/api")
The main run.py file where the app context is available, is started with this:
if __name__ == "__main__":
from gevent import pywsgi
from geventwebsocket.handler import WebSocketHandler
server = pywsgi.WSGIServer(('0.0.0.0', 15080), app, handler_class=WebSocketHandler)
print(server)
server.serve_forever()
Hi Lennart Kloppenburg,
Try to put the #play_socket.route('/status_ping') blueprint route before sockets = Sockets(app).
The following works for me and I can talk to the ping websocket on /play/status_ping:
from flask import Blueprint, Flask, request, abort, redirect, url_for, render_template, make_response, Response, jsonify, session
from flask_sockets import Sockets
import json
play = Blueprint('simple_page', __name__)
play_socket = Blueprint('simple_page2', __name__)
#play_socket.route('/status_ping')
def ping(ws):
while not ws.closed:
message = ws.receive()
response = json.dumps({"Message":message,"Response":"Message received"})
ws.send(response)
app = Flask(__name__)
sockets = Sockets(app)
app.register_blueprint(play, url_prefix=r"/play")
sockets.register_blueprint(play_socket, url_prefix=r"/play")

Categories

Resources